Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

A python script to check if URLs are allowed or disallowed by a robots.txt file.

NotificationsYou must be signed in to change notification settings

p0dalirius/RobotsValidator

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

12 Commits
 
 
 
 
 
 
 
 

Repository files navigation

The robotsvalidator script allows you to check if URLs are allowed or disallowed by a robots.txt file.
GitHub release (latest by date)YouTube Channel Subscribers

Features

  • Getting robots.txt file from local file
  • Getting robots.txt file from an URL
  • Verbose mode, showing all the rules with their results.

Verbose mode

There is a verbose mode using--debug option, which prints every rule with its result:

Contributing

Pull requests are welcome. Feel free to open an issue if you want to add other features.

About

A python script to check if URLs are allowed or disallowed by a robots.txt file.

Topics

Resources

Stars

Watchers

Forks

Sponsor this project

  •  

Languages

[8]ページ先頭
©2009-2025 Movatter.jp