Features

Usage

$ ./crawlersuseragents.py -h[~] Access web pages as web crawlers User-Agents, v1.1usage: crawlersuseragents.py [-h] [-v] [-t THREADS] [-x PROXY] [-k] [-L] [-j JSONFILE] urlThis Python script can be used to check if there is any differences in responses of an applicationwhen the request comes from a search engine's crawler.positional arguments:  url                   e.g. https://example.com:port/pathoptional arguments:  -h, --help            show this help message and exit  -v, --verbose         arg1 help message  -t THREADS, --threads THREADS                        Number of threads (default: 5)  -x PROXY, --proxy PROXY                        Specify a proxy to use for requests (e.g., http://localhost:8080)  -k, --insecure        Allow insecure server connections when using SSL (default: False)  -L, --location        Follow redirects (default: False)  -j JSONFILE, --jsonfile JSONFILE                        Save results to specified JSON file.

Auto-detecting responses that stands out

Contributing

References