robots-txt
Here are 226 public repositories matching this topic...
Language:All
Sort:Most stars
advertools - online marketing productivity and analysis tools
- Updated
Mar 30, 2025 - Python
A simple and flexible web crawler that follows the robots.txt policies and crawl delays.
- Updated
May 19, 2021 - Go
Tame the robots crawling and indexing your Nuxt site.
- Updated
Apr 29, 2025 - TypeScript
The robots.txt exclusion protocol implementation for Go language
- Updated
Nov 9, 2022 - Go
A simple but powerful web crawler library for .NET
- Updated
Dec 15, 2023 - C#
A set of reusable Java components that implement functionality common to any web crawler
- Updated
Apr 25, 2025 - Java
Determine if a page may be crawled from robots.txt, robots meta tags and robot headers
- Updated
Feb 3, 2025 - PHP
🤖 The largest directory for AI-ready documentation and tools implementing the proposed llms.txt standard
- Updated
Apr 28, 2025 - TypeScript
Ultimate Website Sitemap Parser
- Updated
Apr 28, 2025 - Python
Opt-Out tool to check Copyright reservations in a way that even machines can understand.
- Updated
Jan 8, 2024 - Python
Open-Source Python Based SEO Web Crawler
- Updated
Jul 7, 2023 - Python
NodeJS robots.txt parser with support for wildcard (*) matching.
- Updated
Oct 28, 2024 - JavaScript
Known tags and settings suggested to opt out of having your content used for AI training.
- Updated
Jun 21, 2024 - HTML
Makes it easy to add robots.txt, sitemap and web app manifest during build to your Astro app.
- Updated
Dec 15, 2023 - TypeScript
grobotstxt is a native Go port of Google's robots.txt parser and matcher library.
- Updated
Mar 16, 2022 - Go
Gatsby plugin that automatically creates robots.txt for your site
- Updated
Jan 29, 2024 - JavaScript
Collection of SEO utilities like sitemap, robots.txt, etc. for a Remix application. Forked fromhttps://github.com/balavishnuvj/remix-seo
- Updated
Apr 14, 2025 - TypeScript
🤖 A curated list of websites that restrict access to AI Agents, AI crawlers and GPTs
- Updated
Apr 2, 2025 - Python
Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.
- Updated
Feb 16, 2025
Improve this page
Add a description, image, and links to therobots-txt topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with therobots-txt topic, visit your repo's landing page and select "manage topics."