Apache StormCrawler 3.x - Documentation

Below is a simple docker-compose.yaml configuration to spin up URLFrontier, Zookeeper, Storm Nimbus, Storm Supervisor, and the Storm UI:

Notes:

After setting up your Docker Compose, you should start it up:

Check the logs and see, if every service is up and running:

Next, access the Storm UI via https://localhost:8080 and check, that a Storm Nimbus as well as a Storm Supervisor is available.

Build the generated archetype by running

This will create a uberjar named ${artifactId}-${version}.jar (matches the artifact id and the version specified during the archetype generation) in your target directory.

Now you are ready to insert your first seeds into URLFrontier. To do so, create a file seeds.txt containing your seeds:

After you have saved it, we need to inject the seeds into URLFrontier. This can be done by running URLFrontiers client:

where seeds.txt is the previously created file containing URLs to inject, with one URL per line.

Now it is time to run our first crawl. To do so, we need to start our crawler topology in distributed mode and deploy it on our Storm Cluster.

where ${NETWORK} is the name of the Docker network of the previously started Docker Compose. You can find this name by running

After running the storm jar command, you should carefully monitor the logs via

as well as the Storm UI. It should now list a running topology.

In the default archetype, the fetched content is printed out to the default system out print stream.

Congratulations! You learned how to start your first simple crawl using Apache StormCrawler.

Feel free to explore the rest of our documentation to build more complex crawler topologies.

The crawl begins with the Frontier, which is responsible for scheduling, prioritization, politeness, and retry logic. URLs are emitted by a FrontierSpout and partitioned by the URLPartitioner, typically using the host as a key to enforce politeness constraints.

The Fetcher retrieves web resources and emits both the fetched content and associated metadata such as HTTP status codes, headers, and MIME types. Based on the content type, documents are routed to specialized parsers, including SiteMapParser, JSoupParser for HTML content, and TikaParser for binary formats via Apache Tika.

Parsed content is then sent to the Indexer and persisted by the Storage layer. Throughout the pipeline, fetch and parse outcomes are reported to the StatusUpdater, which feeds URL status information back to the frontier, closing the crawl feedback loop.

URL Filters determine whether a URL should be accepted, rejected, or modified before it is scheduled for fetching. They operate on seed URLs, discovered links, and redirect targets, ensuring that only crawl-worthy URLs enter the frontier.

In Figure 1, URL Filters are conceptually positioned between link discovery and the frontier. Their primary role is to control crawl scope and prevent frontier explosion.

Typical URL Filters include:

By applying these filters early, StormCrawler prevents unnecessary fetches and maintains an efficient, focused crawl.

Parse Filters operate after content has been successfully fetched and parsed. They allow fine-grained control over how extracted data and outgoing links are processed.

Parse Filters are applied within the parsing bolts, following parsing by SiteMapParser, JSoupParser, or TikaParser. They can modify extracted text, metadata, and links before the content is indexed or new URLs are emitted.

Common Parse Filters include:

Parse Filters enable domain-specific logic without coupling it directly to the crawler’s core components.

URL Filters focus on deciding what should be crawled, while Parse Filters focus on deciding what should be kept and how it should be interpreted.

There are actually two different bolts for fetching the content of URLs:

Both declare the same output:

with the StatusStream being used for handling redirections, restrictions by robots directives, or fetch errors, whereas the default stream gets the binary content returned by the server as well as the metadata to the following components (typically a parsing bolt).

Both use the same Protocols implementations and URLFilters to control the redirections.

The FetcherBolt has an internal set of queues where the incoming URLs are placed based on their hostname/domain/IP (see config fetcher.queue.mode) and a number of FetchingThreads (config fetcher.threads.number – 10 by default) which pull the URLs to fetch from the FetchQueues. When doing so, they make sure that a minimal amount of time (set with fetcher.server.delay – default 1 sec) has passed since the previous URL was fetched from the same queue. This mechanism ensures that we can control the rate at which requests are sent to the servers. A FetchQueue can also be used by more than one FetchingThread at a time (in which case fetcher.server.min.delay is used), based on the value of fetcher.threads.per.queue.

Incoming tuples spend very little time in the execute method of the FetcherBolt as they are put in the FetchQueues, which is why you’ll find that the value of Execute latency in the Storm UI is pretty low. They get acked later on, after they’ve been fetched. The metric to watch for in the Storm UI is Process latency.

The SimpleFetcherBolt does not do any of this, hence its name. It just fetches incoming tuples in its execute method and does not do multi-threading. It does enforce politeness by checking when a URL can be fetched and will wait until it is the case. It is up to the user to declare multiple instances of the bolt in the Topology class and to manage how the URLs get distributed across the instances of SimpleFetcherBolt, often with the help of the link:https:/

The JSoupParserBolt can be used to parse HTML documents and extract the outlinks, text, and metadata it contains. If you want to parse non-HTML documents, use the Tika-based ParserBolt from the external modules.

This parser calls the URLFilters and ParseFilters defined in the configuration. Please note that it calls MetadataTransfer prior to calling the ParseFilters. If you create new Outlinks in your , you’ll need to make sure that you use MetadataTransfer there to inherit the Metadata from the parent document.

The JSoupParserBolt automatically identifies the charset of the documents. It uses the status stream to report parsing errors but also for the outlinks it extracts from a page. These would typically be used by an extension of AbstractStatusUpdaterBolt and persisted in some form of storage.

StormCrawler can handle sitemap files thanks to the SiteMapParserBolt. This bolt should be placed before the standard ParserBolt in the topology, as illustrated in CrawlTopology.

The reason for this is that the SiteMapParserBolt acts as a filter: it passes on any incoming tuples to the default stream so that they get processed by the ParserBolt, unless the tuple contains isSitemap=true in its metadata, in which case the SiteMapParserBolt will parse it itself. Any outlinks found in the sitemap files are then emitted on the .

The SiteMapParserBolt applies any configured ParseFilters to the documents it parses and, just like its equivalent for HTML pages, it uses MetadataTransfer to populate the Metadata objects for the Outlinks it finds.

ParseFilters are called from parsing bolts such as JSoupParserBolt and SiteMapParserBolt to extract data from web pages. The extracted data is stored in the Metadata object. ParseFilters can also modify the Outlinks and, in that sense, act as URLFilters.

ParseFilters need to implement the interface ParseFilter, which defines three methods:

Here is the default JSON configuration file for ParseFilters. The configuration allows multiple instances of the same filter class with different parameters and supports complex parameter objects. ParseFilters are executed in the order they appear in the JSON file.

The URL filters can be used to both remove or modify incoming URLs (unlike Nutch where these functionalities are separated between URLFilters and URLNormalizers). This is generally used within a parsing bolt to normalize and filter outgoing URLs, but is also called within the FetcherBolt to handle redirections.

URLFilters need to implement the interface URLFilter which defines a single method:

and inherits a default one from Configurable:

The configuration is done via a JSON file which is loaded by the wrapper class URLFilters. The URLFilter instances can be used directly, but it is easier to use the class URLFilters instead. Some filter implementations can also be configured with the standard configuration mechanism.

Here is an example of a JSON configuration file.

The JSON configuration allows loading several instances of the same filtering class with different parameters and can handle complex configuration objects since it makes no assumptions about the content of the field param. The URLFilters are executed in the order in which they are defined in the JSON file.

The following network protocols are implemented in StormCrawler:

Since #829, the HTTP protocol version used is configurable via http.protocol.versions (see also comments in crawler-default.yaml).

For example, to force that only HTTP/1.1 is used:

The metadata argument to HTTPProtocol.getProtocolOutput() can affect the behavior of the protocol. The following metadata keys are detected by HTTPProtocol implementations and utilized in performing the request:

Example:

Notes:

The politeness of a web crawler is not limited to how frequently it fetches pages from a site, but also in how it identifies itself to sites it crawls. This is done by setting the HTTP header User-Agent, just like your web browser does.

The full user agent string is built from the concatenation of the configuration elements:

Whereas StormCrawler used to provide a default value for these, this is not the case since version 2.11 and you will now be asked to provide a value.

You can specify the user agent verbatim with the config http.agent but you will still need to provide a http.agent.name for parsing robots.txt files.

This is also known as the robots.txt protocol, it is formalised in RFC 9309. Part of what the robots directives do is to define rules to specify which parts of a website (if any) are allowed to be crawled. The rules are organised by User-Agent, with a * to match any agent not otherwise specified explicitly, e.g.:

In the example above the rule allows access to the URLs with the /publications/ path prefix, and it restricts access to the URLs with the /example/ path prefix and to all URLs with a .gif suffix. The "*" character designates any character, including the otherwise-required forward slash.

The value of http.agent.name is what StormCrawler looks for in the robots.txt. It MUST contain only uppercase and lowercase letters ("a-z" and "A-Z"), underscores ("_"), and hyphens ("-").

Unless you are running a well known web crawler, it is unlikely that its agent name will be listed explicitly in the robots.txt (if it is, well, congratulations!). While you want the agent name value to reflect who your crawler is, you might want to follow rules set for better known crawlers. For instance, if you were a responsible AI company crawling the web to build a dataset to train LLMs, you would want to follow the rules set for Google-Extended (see list of Google crawlers) if any were found.

This is what the configuration http.robots.agents allows you to do. It is a comma-separated string but can also take a list of values. By setting it alongside http.agent.name (which should also be the first value it contains), you are able to broaden the match rules based on the identity as well as the purpose of your crawler.

If your Apache StormCrawler topology doesn’t extend org.apache.storm.crawler.ConfigurableTopology, you will need to manually register StormCrawler’s Metadata class for serialization in Storm. For more information on Kryo serialization in Apache Storm, you can refer to the documentation.

To register Metadata for serialization, you’ll need to import backtype.storm.Config and org.apache.storm.crawler.Metadata. Then, in your topology class, you’ll register the class with:

where conf is your Storm configuration for the topology.

Alternatively, you can specify in the configuration file:

The class MetadataTransfer is an important part of the framework and is used in key parts of a pipeline:

An instance (or extension) of MetadataTransfer gets created and configured with the method:

which takes as parameter the standard Storm .

A MetadataTransfer instance has mainly two methods, both returning Metadata objects:

The former is used when creating Outlinks, i.e., in the parsing bolts but also for handling redirections in the [[FetcherBolt(s)]].

The latter is used by extensions of the AbstractStatusUpdaterBolt class to determine which Metadata should be persisted.

The behavior of the default MetadataTransfer class is driven by configuration only. It has the following options:

Note that the method getMetaForOutlink calls filter to determine which key values to keep.

The values below are used by sub-classes of AbstractIndexerBolt.

This refers to persisting the status of a URL (e.g. ERROR, DISCOVERED etc.) along with asomething like a nextFetchDate that is being calculated by a Scheduler.

Configures parsing of fetched text and the handling of discovered URIs

Options on how Storm Crawler should handle metadata tracking as well as minimising metadata clashes