webscraper

Star

Here are 1,390 public repositories matching this topic...

Language:All

Filter by language

All1,390 Python807 JavaScript144 Jupyter Notebook102 HTML55 Go46 Java39 TypeScript39 C#38 R17 PHP16

Sort:Most stars

Sort options

Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

anaskhan96 /soup

Star2.2k

Web Scraper in Go, similar to BeautifulSoup

go golang webscraper web-scraper beautifulsoup webscraping html-node

UpdatedNov 2, 2023
Go

jaypyles /Scraperr

Star1.4k

Self-hosted webscraper.

opensource webscraper self-hosted

UpdatedNov 26, 2024
TypeScript

Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.

html cli http json scraper web rest command-line curl xml webscraper wget css-selector xpath xquery data-processing httpie webscraping datascraping xmlstarlet

UpdatedFeb 22, 2025
Pascal

scrapfly /scrapfly-scrapers

Star446

Scalable Python web scraping scripts for +40 popular domains

python crawler scraper automation spider web-crawler scraping crawling webscraper proxies web-scraping webscraping twitter-scraper datascraping python-scraper captcha-bypass antibot scraping-python crawling-python web-scraping-python

UpdatedMar 20, 2025
Python

rootVIII /proxy_requests

Star387

a class that uses scraped proxies to make http GET/POST requests (Python requests)

python http proxy proxy-requests webscraper proxy-server http-proxy python3 recursion requests proxy-list webscraping python-requests http-getter recursion-problem http-proxy-middleware http-get requests-module webscraper-api

UpdatedDec 3, 2020
Python

salimk /Rcrawler

Star355

An R web crawler and scraper

crawler scraper r webscraper crawlers webcrawler webscraping webscrapping rpackage

UpdatedMar 27, 2022
R

onepointAI /onepoint

Star311

An AI assistant tool that integrates coding, writing, and reading functions. For better alternatives seehttps://monica.im/desktop

electron react macos ai toolkit webscraper reading coding all-in-one xiaoai-tts chatgpt gpt-35-turbo

UpdatedMay 12, 2023
TypeScript

toby-p /rightmove_webscraper.py

Star263

Python class to scrape data from rightmove.co.uk and return listings in a pandas DataFrame object

python data-science data-mining csv pandas-dataframe webscraper pandas python3 data-analysis rightmove

UpdatedDec 27, 2023
Python

serpapi /lego-ai-parser

Star232

Lego AI Parser is an open-source application that uses OpenAI to parse visible text of HTML elements.

python html parser machine-learning scraper tools ai parser-library parser-generator webscraper artificial-intelligence datascience webapp openai classification webscraping gpt-3

UpdatedJun 10, 2024
Python

mehmetozkaya /DotnetCrawler

Star175

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link :https://medium.com/@mehmetozkaya/creating-custom-web-crawler-w…

crawler csharp dotnetcore scraping crawling webscraper scrapy entity-framework-core webcrawler webscraping scrapy-crawler ddd-architecture htmlagilitypack webcrawling webcrawler-htmlagilitypack

UpdatedDec 20, 2022
C#

intergalacticalvariable /reader

Star166

📚 This is an adapted version of Jina AI's Reader for local deployment using Docker. Convert any URL to an LLM-friendly input with a simple prefixhttp://127.0.0.1:3000/https://website-to-scrape.com/

docker scraper proxy webscraper self-hosted webscraping website-screenshot website-screenshot-capturer rag webscraping-data llm