- Notifications
You must be signed in to change notification settings - Fork52
Metadata scraper with support for oEmbed, Twitter Cards and Open Graph Protocol for Node.js ⚡
License
jacktuck/unfurl
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
A metadata scraper with support for oEmbed, Twitter Cards and Open Graph Protocol for Node.js (>=v8.0.0).
Note: Will not work in the Browser
Unfurl(spread out from a furled state) will take aurl and someoptions, fetch theurl, extract the metadata we care about and format the result in a sane way. It supports all major metadata providers and expanding it to work for any others should be trivial.
So you know when you link to something on Slack, or Facebook, or Twitter - they typically show a preview of the link. To do so they have crawled the linked website for metadata and enriched the link by providing more context about it. Which usually entails grabbing its title, description and image/player embed.
npm install unfurl.js
oembed?: boolean- support retrieving oembed metadatatimeout? number- req/res timeout in ms, it resets on redirect. 0 to disable (OS limit applies)follow?: number- maximum redirect count. 0 to not follow redirectcompress?: boolean- support gzip/deflate content encodingsize?: number- maximum response body size in bytes. 0 to disableheaders?: Headers | Record<string, string> | Iterable<readonly [string, string]> | Iterable<Iterable<string>>- map of request headers, overrides the defaults
Default headers:
{ 'Accept': 'text/html, application/xhtml+xml', 'User-Agent': 'facebookexternalhit'}import{unfurl}from'unfurl.js'constresult=unfurl('https://github.com/trending')
typeMetadata={title?:stringdescription?:stringkeywords?:string[]favicon?:stringauthor?:stringtheme_color?:stringcanonical_url?:stringoEmbed?:OEmbedPhoto|OEmbedVideo|OEmbedLink|OEmbedRichtwitter_card:{card:stringsite?:stringcreator?:stringcreator_id?:stringtitle?:stringdescription?:stringplayers?:{url:stringstream?:stringheight?:numberwidth?:number}[]apps:{iphone:{id:stringname:stringurl:string}ipad:{id:stringname:stringurl:string}googleplay:{id:stringname:stringurl:string}}images:{url:stringalt:string}[]}open_graph:{title:stringtype:stringimages?:{url:stringsecure_url?:stringtype:stringwidth:numberheight:numberalt?:string}[]url?:stringaudio?:{url:stringsecure_url?:stringtype:string}[]description?:stringdeterminer?:stringsite_name?:stringlocale:stringlocale_alt:stringvideos:{url:stringstream?:stringheight?:numberwidth?:numbertags?:string[]}[]article:{published_time?:stringmodified_time?:stringexpiration_time?:stringauthor?:stringsection?:stringtags?:string[]}}}typeOEmbedBase={type:"photo"|"video"|"link"|"rich"version:stringtitle?:stringauthor_name?:stringauthor_url?:stringprovider_name?:stringprovider_url?:stringcache_age?:numberthumbnails?:[{url?:stringwidth?:numberheight?:number}]}typeOEmbedPhoto=OEmbedBase&{type:"photo"url:stringwidth:numberheight:number}typeOEmbedVideo=OEmbedBase&{type:"video"html:stringwidth:numberheight:number}typeOEmbedLink=OEmbedBase&{type:"link"}typeOEmbedRich=OEmbedBase&{type:"rich"html:stringwidth:numberheight:number}
(If you use unfurl.js too feel free toadd your project)
- vapid/vapid - A template-driven content management system
- beeman/micro-unfurl - small microservice that unfurls a URL and returns the OpenGraph meta data.
- probot/unfurl - a GitHub App built with probot that unfurls links on Issues and Pull Request discussions
About
Metadata scraper with support for oEmbed, Twitter Cards and Open Graph Protocol for Node.js ⚡