Repository files navigation

Easily create XML sitemaps for your website.

Generates a sitemap by crawling your site. Uses streams to efficiently write the sitemap to your drive and runs asynchronously to avoid blocking the thread. Is cappable of creating multiple sitemaps if threshold is reached. Respects robots.txt and meta tags.

This package is not meant to be used in a production code base directly, but rather on the deployed product. This means you develop your app/website as usual, deploy it and create the sitemap with this toolafterwards. The simplest way is to use theCLI (this is a different package!) to create the sitemap on the command line. If you have a more advanced usecase or want to adjust the crawler behavior you should use the programmtic version (this package). Create the crawler as needed and simply run it vianode mycrawler.js.

• Install
• Usage
• API
• Options
• Events
• FAQ
• License

This module is available onnpm.

$ npm install -S sitemap-generator

This module is running only with Node.js and is not meant to be used in the browser.

constSitemapGenerator=require('sitemap-generator');// create generatorconstgenerator=SitemapGenerator('http://example.com',{stripQuerystring:false});// register event listenersgenerator.on('done',()=>{// sitemaps created});// start the crawlergenerator.start();

The crawler will fetch all folder URL pages and file typesparsed by Google. If present therobots.txt will be taken into account and possible rules are applied for each URL to consider if it should be added to the sitemap. Also the crawler will not fetch URL's from a page if the robots meta tag with the valuenofollow is present and ignore them completely ifnoindex rule is present. The crawler is able to apply thebase value to found links.

The generator offers straightforward methods to start and stop it. You can also add URL's manually.

Starts crawler asynchronously and writes sitemap to disk.

Stops the running crawler and halts the sitemap generation.

Returns the crawler instance. For more information about the crawler check thesimplecrawler docs.

This can be useful to ignore certain sites and don't add them to the sitemap.

constcrawler=generator.getCrawler();crawler.addFetchCondition((queueItem,referrerQueueItem,callback)=>{callback(null,!queueItem.path.match(/myregex/));});

Returns the sitemap instance (SitemapRotator).

This can be useful to add static URLs to the sitemap:

constcrawler=generator.getCrawler()constsitemap=generator.getSitemap()// Add static URL on crawl init.crawler.on('crawlstart',()=>{sitemap.addURL('/my/static/url')})

Add a URL to crawler's queue. Useful to help crawler fetch pages it can't find itself.

There are a couple of options to adjust the sitemap output. In addition to the options beneath the options of the used crawler can be changed. For a complete list please check it'sofficial documentation.

vargenerator=SitemapGenerator('http://example.com',{maxDepth:0,filepath:'./sitemap.xml',maxEntriesPerFile:50000,stripQuerystring:true});

Type:string
Default:undefined

If defined, adds a<changefreq> line to each URL in the sitemap. Possible values arealways,hourly,daily,weekly,monthly,yearly,never. All other values are ignored.

Type:string
Default:./sitemap.xml

Filepath for the new sitemap. If multiple sitemaps are created "part_$index" is appended to each filename. If you don't want to write a file at all you can passnull as filepath.

Type:HTTPAgent
Default:http.globalAgent

Controls what HTTP agent to use. This is useful if you want configure HTTP connection through a HTTP/HTTPS proxy (seehttp-proxy-agent).

Type:HTTPAgent
Default:https.globalAgent

Controls what HTTPS agent to use. This is useful if you want configure HTTPS connection through a HTTP/HTTPS proxy (seehttps-proxy-agent).

Example:

// don't forget to:// npm i http-proxy-agent https-proxy-agentconstHttpProxyAgent=require("http-proxy-agent");constHttpsProxyAgent=require("https-proxy-agent");constproxyAddress='http://localhost:1234';consthttpProxyAgent=newHttpProxyAgent(proxyAddress);consthttpsProxyAgent=newHttpsProxyAgent(proxyAddress);options.httpAgent=httpProxyAgent;options.httpsAgent=httpsProxyAgent;

Apply a test condition to a URL before it's added to the sitemap.

Type:function
Default:null

Example:

constgenerator=SitemapGenerator(url,{ignore:url=>{// Prevent URLs from being added that contain `<pattern>`.return/<pattern>/g.test(url)}})

Type:boolean
Default:true

Indicates whetherGoogle AMP pages should be ignored and not be added to the sitemap.

Type:boolean
Default:false

Whether to add a<lastmod> line to each URL in the sitemap. If present the responsesLast-Modified header will be used. Otherwise todays date is added.

Type:number
Default:50000

Google limits the maximum number of URLs in one sitemap to 50000. If this limit is reached the sitemap-generator creates another sitemap. A sitemap index file will be created as well.

Type:array
Default:[]

If provided, adds a<priority> line to each URL in the sitemap. Each value in priorityMap array corresponds with the depth of the URL being added. For example, the priority value given to a URL equalspriorityMap[depth - 1]. If a URL's depth is greater than the length of the priorityMap array, the last value in the array will be used. Valid values are between1.0 and0.0.

Example:

[1.0,0.8,0.6,0.4,0.2,0]

Type:string
Default:Node/SitemapGenerator

Change the default crawler user agent.

The Sitemap Generator emits several events which can be listened to.

Triggered when the crawler successfully added a resource to the sitemap. Passes the url as argument.

generator.on('add',(url)=>{// log url});

Triggered when the crawler finished and the sitemap is created.

generator.on('done',()=>{// sitemaps created});

Thrown if there was an error while fetching an URL. Passes an object with the http status code, a message and the url as argument.

generator.on('error',(error)=>{console.log(error);// => { code: 404, message: 'Not found.', url: 'http://example.com/foo' }});

If an URL matches a disallow rule in therobots.txt file or meta robots noindex is present this event is triggered. The URL will not be added to the sitemap. Passes the ignored url as argument.

generator.on('ignore',(url)=>{// log ignored url});

MIT ©Lars Graubner