robots-parser
Here are 28 public repositories matching this topic...
Language:All
Sort:Most stars
NodeJS robots.txt parser with support for wildcard (*) matching.
- Updated
Nov 22, 2025 - JavaScript
A pure-Python robots.txt parser with support for modern conventions.
- Updated
Nov 8, 2025 - DIGITAL Command Language
An extensible robots.txt parser and client library, with full support for every directive and specification.
- Updated
Apr 10, 2021 - PHP
Alternative robots parser module for Python
- Updated
Nov 23, 2025 - Python
🤖 robots.txt as a service. Crawls robots.txt files, downloads and parses them to check rules through an API
- Updated
Dec 2, 2020 - Java
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
- Updated
Jun 23, 2023 - JavaScript
Go robots.txt parser
- Updated
Nov 27, 2017 - Go
Robots.txt parser and fetcher for Elixir
- Updated
Mar 30, 2023 - Elixir
A lightweight and simple robots.txt parser in node
- Updated
Feb 27, 2023 - TypeScript
🤖 Ruby gem wrapper around Google Robotstxt Parser C++ library
- Updated
Dec 6, 2019 - Ruby
A parser for robots.txt with support for wildcards. See also RFC 9309.
- Updated
Nov 21, 2023 - Erlang
RFC 9309 spec compliant robots.txt builder and parser. 🦾 No dependencies, fully typed.
- Updated
Sep 14, 2024 - TypeScript
Robots.txt parser and generator - Work in progress
- Updated
Apr 26, 2018 - Go
Visual App for Testing URLs and User-agents blocked by robots.txt Files
- Updated
Dec 8, 2022 - Python
💧 Test your robots.txt with this testing tool. Check if a URL is blocked, which statement is blocking it and for which user agent. You can also check if the resources for the page (CSS and JavaScript) are disallowed!. Robots.txt files help you guide how search engines crawl your site, and can be an integral part of your SEO strategy.
- Updated
Oct 5, 2017 - Python
Parse robots.txt and traverse sitemaps.
- Updated
Feb 7, 2025 - C#
Robots.txt parser / generator
- Updated
Sep 18, 2018 - TypeScript
A small, tested, no-frills parser of robots.txt files in Swift.
- Updated
Oct 11, 2024 - Swift
Provides python access to Googles parser for robot.txt files as used by their GoogleBot webscraper.
- Updated
Jul 22, 2024 - Python
Improve this page
Add a description, image, and links to therobots-parser topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with therobots-parser topic, visit your repo's landing page and select "manage topics."