- Notifications
You must be signed in to change notification settings - Fork347
The next web scraper. See through the <html> noise.
License
matthewmueller/x-ray
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
varXray=require('x-ray')varx=Xray()x('https://blog.ycombinator.com/','.post',[{title:'h1 a',link:'.article-title@href'}]).paginate('.nav-previous a@href').limit(3).write('results.json')
npm install x-rayFlexible schema: Supports strings, arrays, arrays of objects, and nested object structures. The schema is not tied to the structure of the page you're scraping, allowing you to pull the data in the structure of your choosing.
Composable: The API is entirely composable, giving you great flexibility in how you scrape each page.
Pagination support: Paginate through websites, scraping each page. X-ray also supports a request
delayand a paginationlimit. Scraped pages can be streamed to a file, so if there's an error on one page, you won't lose what you've already scraped.Crawler support: Start on one page and move to the next easily. The flow is predictable, followinga breadth-first crawl through each of the pages.
Responsible: X-ray has support for concurrency, throttles, delays, timeouts and limits to help you scrape any page responsibly.
Pluggable drivers: Swap in different scrapers depending on your needs. Currently supports HTTP andPhantomJS driver drivers. In the future, I'd like to see a Tor driver for requesting pages through the Tor network.
Scrape theurl for the followingselector, returning an object in the callbackfn.Theselector takes an enhanced jQuery-like string that is also able to select on attributes. The syntax for selecting on attributes isselector@attribute. If you do not supply an attribute, the default is selecting theinnerText.
Here are a few examples:
- Scrape a single tag
xray('http://google.com','title')(function(err,title){console.log(title)// Google})
- Scrape a single class
xray('http://reddit.com','.content')(fn)
- Scrape an attribute
xray('http://techcrunch.com','img.logo@src')(fn)
- Scrape
innerHTML
xray('http://news.ycombinator.com','body@html')(fn)
You can also supply ascope to eachselector. In jQuery, this would look something like this:$(scope).find(selector).
Instead of a url, you can also supply raw HTML and all the same semantics apply.
varhtml='<body><h2>Pear</h2></body>'x(html,'body','h2')(function(err,header){header// => Pear})
Specify adriver to make requests through. Available drivers include:
- request - A simple driver built around request. Use this to set headers, cookies or http methods.
- phantom - A high-level browser automation library. Use this to render pages or when elements need to be interacted with, or when elements are created dynamically using javascript (e.g.: Ajax-calls).
Returns Readable Stream of the data. This makes it easy to build APIs around x-ray. Here's an example with Express:
varapp=require('express')()varx=require('x-ray')()app.get('/',function(req,res){varstream=x('http://google.com','title').stream()stream.pipe(res)})
Stream the results to apath.
If no path is provided, then the behavior is the same as.stream().
Constructs aPromise object and invoke itsthen function with a callbackcb. Be sure to invokethen() at the last step of xray method chaining, since the other methods are not promisified.
x('https://dribbble.com','li.group',[{title:'.dribbble-img strong',image:'.dribbble-img [data-src]@data-src'}]).paginate('.next_page@href').limit(3).then(function(res){console.log(res[0])// prints first result}).catch(function(err){console.log(err)// handle error in promise})
Select aurl from aselector and visit that page.
Limit the amount of pagination ton requests.
Abort pagination ifvalidator function returnstrue.Thevalidator function receives two arguments:
result: The scrape result object for the current page.nextUrl: The URL of the next page to scrape.
Delay the next request betweenfrom andto milliseconds.If onlyfrom is specified, delay exactlyfrom milliseconds.
varx=Xray().delay('1s','10s')
Set the request concurrency ton. Defaults toInfinity.
varx=Xray().concurrency(2)
Throttle the requests ton requests perms milliseconds.
varx=Xray().throttle(2,'1s')
Specify a timeout ofms milliseconds for each request.
varx=Xray().timeout(30)
X-ray also has support for selecting collections of tags. Whilex('ul', 'li') will only select the first list item in an unordered list,x('ul', ['li']) will select all of them.
Additionally, X-ray supports "collections of collections" allowing you to smartly select all list items in all lists with a command like this:x(['ul'], ['li']).
X-ray becomes more powerful when you start composing instances together. Here are a few possibilities:
varXray=require('x-ray')varx=Xray()x('http://google.com',{main:'title',image:x('#gbar a@href','title')// follow link to google images})(function(err,obj){/* { main: 'Google', image: 'Google Images' }*/})
varXray=require('x-ray')varx=Xray()x('http://mat.io',{title:'title',items:x('.item',[{title:'.item-content h2',description:'.item-content section'}])})(function(err,obj){/* { title: 'mat.io', items: [ { title: 'The 100 Best Children\'s Books of All Time', description: 'Relive your childhood with TIME\'s list...' } ] }*/})
Filters can specified when creating a new Xray instance. To apply filters to a value, append them to the selector using|.
varXray=require('x-ray')varx=Xray({filters:{trim:function(value){returntypeofvalue==='string' ?value.trim() :value},reverse:function(value){returntypeofvalue==='string' ?value.split('').reverse().join('') :value},slice:function(value,start,end){returntypeofvalue==='string' ?value.slice(start,end) :value}}})x('http://mat.io',{title:'title | trim | reverse | slice:2,3'})(function(err,obj){/* { title: 'oi' }*/})
- selector: simple string selector
- collections: selects an object
- arrays: selects an array
- collections of collections: selects an array of objects
- array of arrays: selects an array of arrays
- Levered Returns: Uses x-ray to pull together financial data from various unstructured sources around the web.
Support us with a monthly donation and help us continue our activities. [Become a backer]
Become a sponsor and get your logo on our website and on our README on Github with a link to your site. [Become a sponsor]
MIT
About
The next web scraper. See through the <html> noise.