- Notifications
You must be signed in to change notification settings - Fork0
High-performance metadata scraper
License
NotificationsYou must be signed in to change notification settings
fabon-f/fetch-site-metadata
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
High-performance metadata scraper for Node.js
- Don't download whole contents to get site metadata.
- Fetch and parse the content of the
headelement only. Interrupt HTTP request when the<body>element starts. - Download only first few kilobytes to determine image size (by
image-dimensionspackage)
- Fetch and parse the content of the
npm install fetch-site-metadata
This package requires Node.js v18 or higher. ESM only.
importfetchSiteMetadatafrom'fetch-site-metadata'awaitfetchSiteMetadata('https://github.com/')// result:{title:'GitHub: Let’s build from here',description:'GitHub is where over 94 million developers shape the future of software, together. Contribute to the open source community, manage your Git repositories, review code like a pro, track bugs and feat...',icon:'https://github.githubassets.com/favicons/favicon.svg',image:{src:'https://github.githubassets.com/images/modules/site/social-cards/campaign-social.png',width:'1200',height:'630',alt:'GitHub is where over 94 million developers shape the future of software, together. Contribute to the open source community, manage your Git repositories, review code like a pro, track bugs and feat...'}}
Return value:Promise<Metadata>
Fetch target site and scrape metadata. This function send multiple requests to determine OG image size and other informations by default.
suppressAdditionalRequest option suppress this behavior. WithsuppressAdditionalRequestoption, this function fetches only the specified URL and reduces processing time, but provides only information which can be extracted from the specified page.
You can also pass the options forfetch function.
typeOptions={suppressAdditionalRequest?:boolean}&RequestInittypeImageInfo={src:stringwidth:string|undefinedheight:string|undefinedalt:string|undefined}typeMetadata={title:string|undefineddescription:string|undefinedicon:string|undefinedimage:ImageInfo|undefined}
// example using fetch optionsfetchSiteMetadata('https://example.test',{headers:{'User-Agent':'bot'}})
About
High-performance metadata scraper
Resources
License
Stars
Watchers
Forks
Packages0
No packages published