Repository files navigation

Send a little karma down the way and support women empowerment in Zanzibar byhelping tofund the local production of reusable female hygieneproducts. A verydear friend of mine runs the project. They were already able to buy hundreds ofeducational books. Sometimes, it takes so little to make a huge impact. Ifyou'd like to thank me or support this work, donate. Additionally, any currentand future sponsoring of my work via GitHub or other channels will flow onehundred percent to the NGO.

This is a constantly updated collection of user agents I encountered whilerunning web servers on the internet. It's not an exhaustive list. It insteadfocuses on bots, crawlers, certain malware, automated software, scripts anduncommon ones. Lists of regular browser user agents are available elsewhereand too numerous to sanely and cleanly manage.

There are lots of use cases for user agent information, especially when parsingweb server logs. Below are some examples that illustrate how to quickly getfiltered information out of this data set using the excellentjqcommand-line tool.

cat data/*.json | jq -r 'select(.category==7) | .user_agents[]'

cat data/*.json | jq -r 'select(.country=="CN") | select(.type==2)  | .user_agents[]'

cat data/*.json | jq -r 'select(.type==99) | .known_cidrs[]'

To get a list of all encountered user agents you can run a command like

cat /var/log/nginx/* | awk -F\" '{print $6}' | sort -u > uas.txt

• Create a single file JSON entry per entity. Usetemplate.json to start. Thenew.sh helper script is great for this.
• Index codes are listed in folderindexes.
• Fill out as much information as possible, use existing entries for reference.Be especially thorough regarding country, website and description.
• Format with Prettier. The default style is sufficient. You can do so byinstalling it (npm install -g prettier) and runningprettier --write entry.json.
• If there are multiple mostly identical user agent strings for an entry,restrict to one example per majorsemantic version.
• All array entries are sorted, alphabetically and numerically.
• Ifcountry does not apply or is international, use"ZZ" andnull whennot applicable.
• null is to be interpreted as "not applicable" or "unknown", depending oncontext.

This work is licensed under aCreative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

The data is completely free for personal, non-commercial usage, including FOSSprojects. If you plan to include it in a product you earn money on or use forinfrastructure you earn money with, I welcome your decision. However, you willneed to license it by becoming a permanenttop-tier GitHub sponsor.If this is too steep for you,let me know and we'll talk.