Repository files navigation

This project is unmaintained and active projects relying on it are advised to migrate to alternative solutions.

simplecrawler is designed to provide a basic, flexible and robust API for crawling websites. It was written to archive, analyse, and search some very large websites and has happily chewed through hundreds of thousands of pages and written tens of gigabytes to disk without issue.

• Provides a very simple event driven API usingEventEmitter
• Extremely configurable base for writing your own crawler
• Provides some simple logic for auto-detecting linked resources - which you can replace or augment
• Automatically respects any robots.txt rules
• Has a flexible queue system which can be frozen to disk and defrosted
• Provides basic statistics on network performance
• Uses buffers for fetching and managing data, preserving binary data (except when discovering links)

• Installation
• Getting started
• Events
  • A note about HTTP error conditions
  • Waiting for asynchronous event listeners
• Configuration
• Fetch conditions
• Download conditions
• The queue
  • Manually adding to the queue
  • Queue items
  • Queue statistics and reporting
  • Saving and reloading the queue (freeze/defrost)
• Cookies
  • Cookie events
• Link Discovery
• FAQ/Troubleshooting
• Node Support Policy
• Current Maintainers
• Contributing
• Contributors
• License

npm install --save simplecrawler

Initializing simplecrawler is a simple process. First, you require the module and instantiate it with a single argument. You then configure the properties you like (eg. the request interval), register a few event listeners, and call the start method. Let's walk through the process!

After requiring the crawler, we create a new instance of it. We supply the constructor with a URL that indicates which domain to crawl and which resource to fetch first.

varCrawler=require("simplecrawler");varcrawler=newCrawler("http://www.example.com/");

You can initialize the crawler with or without thenew operator. Being able to skip it comes in handy when you want to chain API calls.

varcrawler=Crawler("http://www.example.com/").on("fetchcomplete",function(){console.log("Fetched a resource!")});

By default, the crawler will only fetch resources on the same domain as that in the URL passed to the constructor. But this can be changed through thecrawler.domainWhitelist property.

Now, let's configure some more things before we start crawling. Of course, you're probably wanting to ensure you don't take down your web server. Decrease the concurrency from five simultaneous requests - and increase the request interval from the default 250 ms like this:

crawler.interval=10000;// Ten secondscrawler.maxConcurrency=3;

You can also define a max depth for links to fetch:

crawler.maxDepth=1;// Only first page is fetched (with linked CSS & images)// Or:crawler.maxDepth=2;// First page and discovered links from it are fetched// Or:crawler.maxDepth=3;// Etc.

For a full list of configurable properties, see theconfiguration section.

You'll also need to set up event listeners for theevents you want to listen to.crawler.fetchcomplete andcrawler.complete are good places to start.

crawler.on("fetchcomplete",function(queueItem,responseBuffer,response){console.log("I just received %s (%d bytes)",queueItem.url,responseBuffer.length);console.log("It was a resource of type %s",response.headers['content-type']);});

Then, when you're satisfied and ready to go, start the crawler! It'll run through its queue finding linked resources on the domain to download, until it can't find any more.

crawler.start();

simplecrawler's API is event driven, and there are plenty of events emitted during the different stages of the crawl.

Fired when the crawl starts. This event gives you the opportunity toadjust the crawler's configuration, since the crawl won't actually startuntil the next processor tick.

Fired when the discovery of linked resources has completed

Fired when a resource wasn't queued because of an invalid domain name

Fired when a resource wasn't queued because it was disallowed by thesite's robots.txt rules

Fired when a fetch condition returns an error

Fired when a fetch condition prevented the queueing of a URL

Fired when a new queue item was rejected because anotherqueue item with the same URL was already in the queue

Fired when an error was encountered while updating a queue item

Fired when an item was added to the crawler's queue

Fired when a request times out

Fired when a request encounters an unknown error

Fired just after a request has been initiated

Fired when an error was encountered while trying to add acookie to the cookie jar

Fired when the headers for a request have been received

Fired when a download condition returns an error

Fired when the downloading of a resource was preventedby a download condition

Fired when the crawler's cache was enabled and the server responded with a 304 Not Modified status for the request

Fired when the server returned a redirect HTTP status for the request

Fired when the server returned a 404 Not Found status for the request

Fired when the server returned a 410 Gone status for the request

Fired when the server returned a status code above 400 that isn't 404 or 410

Fired when the request has completed

Fired when an error was encountered while unzipping the response data

Fired when a resource couldn't be downloaded because it exceeded the maximum allowed size

Fired when an error was encountered while retrieving a robots.txt file

Fired when the crawl has completed - all resources in the queue have been dealt with

By default, simplecrawler does not download the response body when it encounters an HTTP error status in the response. If you need this information, you can listen to simplecrawler's error events, and through node's nativedata event (response.on("data",function(chunk) {...})) you can save the information yourself.

Sometimes, you might want to wait for simplecrawler to wait for you while you perform some asynchronous tasks in an event listener, instead of having it racing off and firing thecomplete event, halting your crawl. For example, if you're doing your own link discovery using an asynchronous library method.

simplecrawler provides await method you can call at any time. It is available viathis from inside listeners, and on the crawler object itself. It returns a callback function.

Once you've called this method, simplecrawler will not fire thecomplete event until either you execute the callback it returns, or a timeout is reached (configured incrawler.listenerTTL, by default 10000 ms.)

crawler.on("fetchcomplete",function(queueItem,data,res){varcontinue=this.wait();doSomeDiscovery(data,function(foundURLs){foundURLs.forEach(function(url){crawler.queueURL(url,queueItem);});continue();});});

simplecrawler is highly configurable and there's a long list of settings you can change to adapt it to your specific needs.

Controls which URL to request first

Determines what hostname the crawler should limit requests to (so long asfilterByDomain is true)

Determines the interval at which new requests are spawned by the crawler,as long as the number of open requests is under themaxConcurrency cap.

Maximum request concurrency. If necessary, simplecrawler will increasenode's http agent maxSockets value to match this setting.

Maximum time we'll wait for headers

Maximum time we'll wait for async listeners

Crawler's user agent string

Default:"Node/simplecrawler <version> (https://github.com/simplecrawler/simplecrawler)"

Queue for requests. The crawler can use any implementation so long as ituses the same interface. The default queue is simply backed by an array.

Controls whether the crawler respects the robots.txt rules of any domain.This is done both with regards to the robots.txt file, and<meta> tagsthat specify anofollow value for robots. The latter only applies ifthe defaultdiscoverResources method is used, though.

Controls whether the crawler is allowed to change thehost setting if the first response is a redirect toanother domain.

Controls whether HTTP responses are automatically decompressed based ontheir Content-Encoding header. If true, it will also assign theappropriate Accept-Encoding header to requests.

Controls whether HTTP responses are automatically character converted tostandard JavaScript strings using theiconv-litemodule before emitted in thefetchcomplete event.The character encoding is interpreted from the Content-Type headerfirstly, and secondly from any<meta charset="xxx" /> tags.

Controls whether the crawler fetches only URL's where the hostnamematcheshost. Unless you want to be crawling the entireinternet, I would recommend leaving this on!

Controls whether URL's that points to a subdomain ofhostshould also be fetched.

Controls whether to treat the www subdomain as the same domain ashost. So ifhttp://example.com/example hasalready been fetched,http://www.example.com/example won't befetched also.

Controls whether to strip the www subdomain entirely from URL's at queueitem construction time.

Internal cache store. Must implementSimpleCache interface. You cansave the site to disk using the built in file system cache like this:

crawler.cache=newCrawler.cache('pathToCacheDirectory');

Controls whether an HTTP proxy should be used for requests

IfuseProxy is true, this setting controls what hostnameto use for the proxy

IfuseProxy is true, this setting controls what port touse for the proxy

IfuseProxy is true, this setting controls what usernameto use for the proxy

IfuseProxy is true, this setting controls what passwordto use for the proxy

Controls whether to use HTTP Basic Auth

IfneedsAuth is true, this setting controls what usernameto send with HTTP Basic Auth

IfneedsAuth is true, this setting controls what passwordto send with HTTP Basic Auth

Controls whether to save and send cookies or not

The module used to store cookies

Controls what headers (besides the default ones) to include with everyrequest.

Controls what domains the crawler is allowed to fetch from, regardless ofhost orfilterByDomain settings.

Controls what protocols the crawler is allowed to fetch from

Controls the maximum allowed size in bytes of resources to be fetched

Default:16777216

Controls what mimetypes the crawler will scan for new resources. IfdownloadUnsupported is false, this setting will alsorestrict what resources are downloaded.

Controls whether to download resources with unsupported mimetypes (asspecified bysupportedMimeTypes)

Controls what URL encoding to use. Can be either "unicode" or "iso8859"

Controls whether to strip query string parameters from URL's at queueitem construction time.

Controls whether to sort query string parameters from URL's at queueitem construction time.

Collection of regular expressions and functions that are applied in thedefaultdiscoverResources method.

Controls whether the defaultdiscoverResources shouldscan for new resources inside of HTML comments.

Controls whether the defaultdiscoverResources shouldscan for new resources inside of<script> tags.

Controls the max depth of resources that the crawler fetches. 0 meansthat the crawler won't restrict requests based on depth. The initialresource, as well as manually queued resources, are at depth 1. Fromthere, every discovered resource adds 1 to its referrer's depth.

Controls whether to proceed anyway when the crawler encounters an invalidSSL certificate.

Controls what HTTP agent to use. This is useful if you want to configureeg. a SOCKS client.

Controls what HTTPS agent to use. This is useful if you want to configureeg. a SOCKS client.

simplecrawler has an concept called fetch conditions that offers a flexible API for filtering discovered resources before they're put in the queue. A fetch condition is a function that takes a queue item candidate and evaluates (synchronously or asynchronously) whether it should be added to the queue or not.Please note: with the next major release, all fetch conditions will be asynchronous.

You may add as many fetch conditions as you like, and remove them at runtime. simplecrawler will evaluate every fetch condition in parallel until one is encountered that returns a falsy value. If that happens, the resource in question will not be fetched.

This API is complemented bydownload conditions that determine whether a resource's body data should be downloaded.

Adds a callback to the fetch conditions array. simplecrawler will evaluateall fetch conditions for every discovered URL, and if any of the fetchconditions returns a falsy value, the URL won't be queued.

Returns:Number - The index of the fetch condition in the fetch conditions array. This can later be used to remove the fetch condition.

Evaluated for every discovered URL to determine whether to put it in thequeue.

Removes a fetch condition from the fetch conditions array.

Returns:Boolean - If the removal was successful, the method will return true. Otherwise, it will throw an error.

While fetch conditions let you determine which resources to put in the queue, download conditions offer the same kind of flexible API for determining which resources' data to download. Download conditions support both a synchronous and an asynchronous API, butwith the next major release, all download conditions will be asynchronous.

Download conditions are evaluated after the headers of a resource have been downloaded, if that resource returned an HTTP status between 200 and 299. This lets you inspect the content-type and content-length headers, along with all other properties on the queue item, before deciding if you want this resource's data or not.

Adds a callback to the download conditions array. simplecrawler will evaluateall download conditions for every fetched resource after the headers of thatresource have been received. If any of the download conditions returns afalsy value, the resource data won't be downloaded.

Returns:Number - The index of the download condition in the download conditions array. This can later be used to remove the download condition.

Evaluated for every fetched resource after its header have been received todetermine whether to fetch the resource body.

Removes a download condition from the download conditions array.

Returns:Boolean - If the removal was successful, the method will return true. Otherwise, it will throw an error.

Like any other web crawler, simplecrawler has a queue. It can be directly accessed throughcrawler.queue and implements an asynchronous interface for accessing queue items and statistics. There are several methods for interacting with the queue, the simplest beingcrawler.queue.get, which lets you get a queue item at a specific index in the queue.

Get a queue item by index

All queue method are in reality synchronous by default, but simplecrawler is built to be able to use different queues that implement the same interface, and those implementations can be asynchronous - which means they could eg. be backed by a database.

To add items to the queue, usecrawler.queueURL.

Queues a URL for fetching after cleaning, validating and constructing a queueitem from it. If you're queueing a URL manually, use this method rather thanCrawler#queue#add

Returns:Boolean - The return value used to indicate whether the URL passed all fetch conditions and robots.txt rules. With the advent of async fetch conditions, the return value will no longer take fetch conditions into account.
Emits:invaliddomain,fetchdisallowed,fetchconditionerror,fetchprevented,queueduplicate,queueerror,queueadd

Because when working with simplecrawler, you'll constantly be handed queue items, it helps to know what's inside them. Here's the formal documentation of the properties that they contain.

QueueItems represent resources in the queue that have been fetched, or will be eventually.

Properties

First of all, the queue can provide some basic statistics about the network performance of your crawl so far. This is done live, so don't check it 30 times a second. You can test the following properties:

• requestTime
• requestLatency
• downloadTime
• contentLength
• actualDataSize

You can get the maximum, minimum, and average values for each with thecrawler.queue.max,crawler.queue.min, andcrawler.queue.avg functions respectively.

Gets the maximum value of a stateData property from all the items in thequeue. This means you can eg. get the maximum request time, download sizeetc.

Gets the minimum value of a stateData property from all the items in thequeue. This means you can eg. get the minimum request time, download sizeetc.

Gets the average value of a stateData property from all the items in thequeue. This means you can eg. get the average request time, download sizeetc.

For general filtering or counting of queue items, there are two methods:crawler.queue.filterItems andcrawler.queue.countItems. Both take an object comparator and a callback.

Filters and returns the items in the queue that match a selector

Counts the items in the queue that match a selector

The object comparator can also contain other objects, so you may filter queue items based on properties in theirstateData object as well.

crawler.queue.filterItems({stateData:{code:301}},function(error,items){console.log("These items returned a 301 HTTP status",items);});

It can be convenient to be able to save the crawl progress and later be able to reload it if your application fails or you need to abort the crawl for some reason. Thecrawler.queue.freeze andcrawler.queue.defrost methods will let you do this.

A word of warning - they are not CPU friendly as they rely onJSON.parse andJSON.stringify. Use them only when you need to save the queue - don't call them after every request or your application's performance will be incredibly poor - they block likecrazy. That said, using them when your crawler commences and stops is perfectly reasonable.

Note that the methods themselves are asynchronous, so if you are going to exit the process after you do the freezing, make sure you wait for callback - otherwise you'll get an empty file.

Writes the queue to disk in a JSON file. This file can later be importedusingdefrost

Import the queue from a frozen JSON file on disk.

simplecrawler has an internal cookie jar, which collects and resends cookies automatically and by default. If you want to turn this off, set thecrawler.acceptCookies option tofalse. The cookie jar is accessible viacrawler.cookies, and is an event emitter itself.

Fired when a cookie has been added to the jar

Fired when one or multiple cookie have been removed from the jar

simplecrawler's discovery function is made to be replaceable — you can easily write your own that discovers only the links you're interested in.

The method must accept a buffer and aqueueItem, and return the resources that are to be added to the queue.

It is quite common to pair simplecrawler with a module likecheerio that can correctly parse HTML and provide a DOM like API for querying — or even a whole headless browser, like phantomJS.

The example below demonstrates how one might achieve basic HTML-correct discovery of only link tags using cheerio.

crawler.discoverResources=function(buffer,queueItem){var$=cheerio.load(buffer.toString("utf8"));return$("a[href]").map(function(){return$(this).attr("href");}).get();};

There are a couple of questions that pop up more often than others in the issue tracker. If you're having trouble with simplecrawler, please have a look at the list below before submitting an issue.

• Q: Why does simplecrawler discover so many invalid URLs?

  A: simplecrawler's built-in discovery method is purposefully naive - it's a brute force approach intended to find everything: URLs in comments, binary files, scripts, image EXIF data, inside CSS documents, and more — useful for archiving and use cases where it's better to have false positives than fail to discover a resource.

  It's definitely not a solution for every case, though — if you're writing a link checker or validator, you don't want erroneous 404s throwing errors. Therefore, simplecrawler allows you to tune discovery in a few key ways:

  • You can either add to (or remove from) thecrawler.discoverRegex array, tweaking the search patterns to meet your requirements; or
  • Swap out thediscoverResources method. Parsing HTML pages is beyond the scope of simplecrawler, but it is very common to combine simplecrawler with a module likecheerio for more sophisticated resource discovery.

  Further documentation is available in thelink discovery section.

• Q: Why did simplecrawler complete without fetching any resources?

  A: When this happens, it is usually because the initial request was redirected to a different domain that wasn't in thecrawler.domainWhitelist.

• Q: How do I crawl a site that requires a login?

  A: Logging in to a site is usually fairly simple and most login procedures look alike. We've included an example that covers a lot of situations, but sadly, there isn't a one true solution for how to deal with logins, so there's no guarantee that this code works right off the bat.

  What we do here is:

  1. fetch the login page,
  2. store the session cookie assigned to us by the server,
  3. extract any CSRF tokens or similar parameters required when logging in,
  4. submit the login credentials.

  varCrawler=require("simplecrawler"),url=require("url"),cheerio=require("cheerio"),request=require("request");varinitialURL="https://example.com/";varcrawler=newCrawler(initialURL);request("https://example.com/login",{// The jar option isn't necessary for simplecrawler integration, but it's// the easiest way to have request remember the session cookie between this// request and the nextjar:true},function(error,response,body){// Start by saving the cookies. We'll likely be assigned a session cookie// straight off the bat, and then the server will remember the fact that// this session is logged in as user "iamauser" after we've successfully// logged incrawler.cookies.addFromHeaders(response.headers["set-cookie"]);// We want to get the names and values of all relevant inputs on the page,// so that any CSRF tokens or similar things are included in the POST// requestvar$=cheerio.load(body),formDefaults={},// You should adapt these selectors so that they target the// appropriate form and inputsformAction=$("#login").attr("action"),loginInputs=$("input");// We loop over the input elements and extract their names and values so// that we can include them in the login POST requestloginInputs.each(function(i,input){varinputName=$(input).attr("name"),inputValue=$(input).val();formDefaults[inputName]=inputValue;});// Time for the login request!request.post(url.resolve(initialURL,formAction),{// We can't be sure that all of the input fields have a correct default// value. Maybe the user has to tick a checkbox or something similar in// order to log in. This is something you have to find this out manually// by logging in to the site in your browser and inspecting in the// network panel of your favorite dev tools what parameters are included// in the request.form:Object.assign(formDefaults,{username:"iamauser",password:"supersecretpw"}),// We want to include the saved cookies from the last request in this// one as welljar:true},function(error,response,body){// That should do it! We're now ready to start the crawlercrawler.start();});});crawler.on("fetchcomplete",function(queueItem,responseBuffer,response){console.log("Fetched",queueItem.url,responseBuffer.toString());});

• Q: What does it mean that events are asynchronous?

  A: One of the core concepts of node.js is its asynchronous nature. I/O operations (like network requests) take place outside of the main thread (which is where your code is executed). This is what makes node fast, the fact that it can continue executing code while there are multiple HTTP requests in flight, for example. But to be able to get back the result of the HTTP request, we need to register a function that will be called when the result is ready. This is whatasynchronous means in node - the fact that code can continue executing while I/O operations are in progress - and it's the same concept as with AJAX requests in the browser.

• Q: Promises are nice, can I use them with simplecrawler?

  A: No, not really. Promises are meant as a replacement for callbacks, but simplecrawler is event driven, not callback driven. Using callbacks to any greater extent in simplecrawler wouldn't make much sense, since you normally need to react more than once to what happens in simplecrawler.

• Q: Something's happening and I don't see the output I'm expecting!

  Before filing an issue, check to see that you're not just missing something by loggingall crawler events with the code below:

  varoriginalEmit=crawler.emit;crawler.emit=function(evtName,queueItem){crawler.queue.countItems({fetched:true},function(err,completeCount){if(err){throwerr;}crawler.queue.getLength(function(err,length){if(err){throwerr;}console.log("fetched %d of %d — %d open requests, %d open listeners",completeCount,length,crawler._openRequests.length,crawler._openListeners);});});console.log(evtName,queueItem ?queueItem.url ?queueItem.url :queueItem :null);originalEmit.apply(crawler,arguments);};

  If you don't see what you need after inserting that code block, and you still need help, please attach the output of all the events fired with your email/issue.

Simplecrawler will officially support stable and LTS versions of Node which are currently supported by the Node Foundation.

Currently supported versions:

• 8.x
• 10.x
• 12.x

• Christopher Giffard
• Fredrik Ekelund
• Konstantin Bläsi
• XhmikosR

Please see thecontributor guidelines before submitting a pull request to ensure that your contribution is able to be accepted quickly and easily!

simplecrawler has benefited from the kind efforts of dozens of contributors, to whom we are incredibly grateful. We originally listed their individual contributions but it became pretty unwieldy - thefull list can be found here.

Copyright (c) 2017, Christopher Giffard.

All rights reserved.

Redistribution and use in source and binary forms, with or without modification,are permitted provided that the following conditions are met:

• Redistributions of source code must retain the above copyright notice, thislist of conditions and the following disclaimer.
• Redistributions in binary form must reproduce the above copyright notice, thislist of conditions and the following disclaimer in the documentation and/orother materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" ANDANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIEDWARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE AREDISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FORANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ONANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THISSOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.