web-data-extraction

The this.url class is designed to fetch and parse URL data, returning an object with structured information that can then be used for machine learning algorithms in a database or other storage.

web-scraping url-parsing metadata-extraction web-data-extraction neurons-me-ecosystem structured-url-data machine-learning-urls data-driven-web-analysis intelligent-link-processing ai-ready-url-processing

UpdatedAug 26, 2025
JavaScript

lightfeed /extractor

Star56

Using LLMs and AI browser automation to robustly extract web data

nlp markdown crawler etl html-to-markdown data-engineering html-parser openai rss-feed webscraping data-pipeline web-data-extraction ai-agents article-extractor web-extraction rag llm google-gemini llm-scraper llm-extraction

UpdatedSep 30, 2025
TypeScript

crawlcore /qcrawl

Star42

qcrawl - fast async web crawling & scraping framework for Python.

python crawler framework async scraping crawling web-scraping asyncio web-data-extraction web-scraping-python camoufox

UpdatedDec 7, 2025
Python

luminati-io /java-web-scraping

Star24

Quick guide with code example how to use Java for web scraping

java maven scraping-websites web-data-extraction

UpdatedDec 18, 2024

GNewsScraper is a TypeScript package that scrapes article data from Google News based on a keyword or phrase. It returns the results as an array of JSON objects, making it convenient to access and use the scraped information

typescript web-scraping json-parsing web-crawling google-news data-scraping google-news-scraper web-data-extraction web-automation keyword-search gnews news-scraping gnews-api article-extraction gnews-scraper

UpdatedAug 19, 2023
TypeScript

jjonescz /awe

Sponsor

Star12

AI-based web extractor

deep-learning information-extraction web-scraping web-data-extraction structured-web-data

UpdatedFeb 25, 2023
Python

DemonMartin /scrappey-wrapper

Star12

An API wrapper for Scrappey.com written in Node.js (cloudflare bypass & solver)

web-scraping data-extraction web-data-extraction scraping-framework scraping-tool cloudflare-bypass web-scraping-solution cloudflare-solver api-scraping scraping-solution website-data-extraction scraping-library cloudflare-anti-bot scraping-service data-scraping-tool website-scraping-tool turnstile-solver

UpdatedJan 10, 2024
JavaScript

Boomslet /Web_Crawler

Star9

Open-source web crawler

python url html open-source website opensource links web-crawler urls free data-extraction webcrawler web-crawling web-data-extraction urllib web-crawler-python

UpdatedJul 21, 2018
Python

wbsg-uni-mannheim /WDCFramework

Star8

Java Framework which is used by the Web Data Commons project to extract Microdata, Microformats and RDFa data, Web graphs, and HTML tables from the web crawls provided by the Common Crawl Foundation.

schema-org json-ld microdata web-data-extraction

UpdatedDec 13, 2022
Java

kaizenplatform /FacebookInsightsConnector

Star8

The Tableau Web Data Connector for Facebook Insights API

facebook tableau facebook-insights web-data-extraction

UpdatedJun 26, 2017
JavaScript

SaurabhSSB /BookMiner

Star8

A pipeline to scrape, extract, and analyze book data from web pages to insights.

python books jupyter-notebook eda data-visualization web-scraping data-analysis html-parsing beautifulsoup csv-export data-pipeline web-data-extraction data-science-project project-portfolio book-dataset

UpdatedSep 30, 2025
HTML

lekhmanrus /real-shot-pdf

Star7

RealShotPDF is a Chrome extension designed to simplify the process of creating PDF documents from web content. The extension allows users to navigate through selected webpages, parse and display links in a tree view, and generate PDFs for the chosen pages. It operates locally without sending any data to external servers.