You signed in with another tab or window.Reload to refresh your session.You signed out in another tab or window.Reload to refresh your session.You switched accounts on another tab or window.Reload to refresh your session.Dismiss alert
Walker walkes aka as crawls through websites and collects performance and SEO relevant data. The results can be browsed through a very simple web interface. Apart from that they are exposed as prometheus metrics (not implemented yet).
Be careful when crawling your website with walker with aggressive settings, it might take your site down
Configuration
---# target of your scrapetarget:http://www.bestbytes.de# number of concurrent go routinesconcurrency:2# where to run the webinterfaceaddr:":3001"# if you want to ignore <meta name="robots" content="noindex,nofollow"/>ignorerobots:true# in some cases using cookies is friendlier to the serverusecookies:true# ignoring urls## based on query parameters in this example all links, that contain a queryparameter fooignorequerieswith: -foo## skip everything that has a queryignoreallqueries:true# what paths (that would be a prefixes)ignore: -/foomo...
error detection
everything greater than 400 will be tracked as an error