Repository files navigation

BlackIP is a project that collects and unifies public blocklists of IP addresses, to make them compatible withSquid andIPSET (IptablesNetfilter).

git clone --depth=1 https://github.com/maravento/blackip.git

blackip.txt is already optimized. Download it and unzip it in the path of your preference.

wget -q -N https://raw.githubusercontent.com/maravento/blackip/master/blackip.tar.gz&& cat blackip.tar.gz*| tar xzf -

wget -q -N https://raw.githubusercontent.com/maravento/blackip/master/checksum.md5md5sum blackip.txt| awk'{print $1}'&& cat checksum.md5| awk'{print $1}'

• Should not be usedblackip.txt inIPSET and inSquid at the same time (double filtrate).
• blackip.txt is a list IPv4. Does not include CIDR.

Edit your Iptables bash script and add the following lines (run with root privileges):

#!/bin/bash# https://linux.die.net/man/8/ipset# variablesipset=/sbin/ipsetiptables=/sbin/iptables# Replace with your path to blackip.txtips=/path_to_lst/blackip.txt# ipset rules$ipset -L blackip>/dev/null2>&1if [$?-ne 0 ];thenecho"set blackip does not exist. create set..."$ipset -! create blackip hash:net family inet hashsize 1024 maxelem 10000000elseecho"set blackip exist. flush set..."$ipset -! flush blackipfi$ipset -! save> /tmp/ipset_blackip.txt# read file and sort (v8.32 or later)cat$ips| sort -V -u|whileread line;do# optional: if there are commented linesif ["${line:0:1}"="#" ];thencontinuefi# adding IPv4 addresses to the tmp listecho"add blackip$line">> /tmp/ipset_blackip.txtdone# adding the tmp list of IPv4 addresses to the blackip set of ipset$ipset -! restore< /tmp/ipset_blackip.txt# iptables rules$iptables -t mangle -I PREROUTING -mset --match-set blackip src,dst -j DROP$iptables -I INPUT -mset --match-set blackip src,dst -j DROP$iptables -I FORWARD -mset --match-set blackip src,dst -j DROPecho"done"

You can add the following lines to the bash above to include full country IP ranges withIPDeny adding the countries of your choice.

# Put these lines at the end of the "variables" section# Replace with your path to zones folderzones=/path_to_folder/zones# download zonesif [!-d$zones ];then mkdir -p$zones;fiwget -q -N http://www.ipdeny.com/ipblocks/data/countries/all-zones.tar.gztar -C$zones -zxvf all-zones.tar.gz>/dev/null2>&1rm -f all-zones.tar.gz>/dev/null2>&1# replace the line:cat$ips| sort -V -u|whileread line;do# with (e.g: Russia and China):cat$zones/{cn,ru}.zone$ips| sort -V -u|whileread line;do

• Ipset allows mass filtering, at a much higher processing speed than other solutions (checkbenchmark).
• Blackip is a list containing millions of IPv4 lines and to be supported by Ipset, we had to arbitrarily increase the parametermaxelem (for more information, checkipset's hashsize and maxelem parameters).
• Ipset/iptables limitation: "When entries added by the SET target of iptables/ip6tables, then the hash size is fixed and the set won't be duplicated, even if the new entry cannot be added to the set" (for more information, checkMan Ipset).
• Heavy use of these rules can slow down your PC to the point of crashing. Use them at your own risk.
• Tested on iptables v1.8.7, ipset v7.15, protocol version: 7.

Edit:

/etc/squid/squid.conf

And add the following lines:

# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS# Block Rule for BlackIPacl blackip dst"/path_to/blackip.txt"http_access deny blackip

• blackip.txt has been tested in Squid v3.5.x and later.

BlackIP contains millions of IP addresses, therefore it is recommended:

• Useblackcidr.txt to add IP/CIDR that are not included inblackip.txt (By default it contains some Block CIDR).
• Useallowip.txt (a whitelist of IPv4 IP addresses such as Hotmail, Gmail, Yahoo. etc.).
• Useaipextra.txt to add whitelists of IP/CIDRs that are not included inallowip.txt.
• By default,blackip.txt excludes some private or reserved rangesRFC1918. Use IANA (iana.txt) to exclude them all.
• By default,blackip.txt excludes some DNS servers included indns.txt. You can use this list and expand it to deny or allow DNS servers.
• To increase security, close Squid to any other request to IP addresses with ZTR.

### INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS #### Allow Rule for IPacl allowip dst"/path_to/allowip.txt"http_access allow allowip# Allow Rule for IP/CIDR ACL (not included in allowip.txt)acl aipextra dst"/path_to/aipextra.txt"http_access allow aipextra# Allow Rule for IANA ACL (not included in allowip.txt)acl iana dst"/path_to/iana.txt"http_access allow iana# Allow Rule for DNS ACL (excluded from blackip.txt)acl dnslst dst"/path_to/dns.txt"http_access allow dnslst# or deny dnlst# Block Rule for IP/CIDR ACL (not included in blackip.txt)acl blackcidr dst"/path_to/blackcidr.txt"http_access deny blackcidr## Block Rule for BlackIPacl blackip dst"/path_to/blackip.txt"http_access deny blackip## Zero Trust Rule (ZTR)acl no_ip url_regex -i [0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}http_access deny no_ip

This section is only to explain how update and optimization process works. It is not necessary for user to run it. This process can take time and consume a lot of hardware and bandwidth resources, therefore it is recommended to use test equipment.

The update process ofblackip.txt is executed in sequence by the scriptbipupdate.sh. The script will request privileges when required.

wget -q -N https://raw.githubusercontent.com/maravento/blackip/master/bipupdate/bipupdate.sh&& chmod +x bipupdate.sh&& ./bipupdate.sh

Update requires python 3x and bash 5x.

pkgs='wget git curl idn2 perl tar rar unrar unzip zip python-is-python3 ipset'if! dpkg -s$pkgs>/dev/null2>&1;then  apt -y install$pkgsfi

Make sure your Squid is installed correctly. If you have any problems, run the following script: (sudo ./squid_install.sh):

#!/bin/bash# kill old versionwhile pgrep squid> /dev/null;doecho"Waiting for Squid to stop..."    killall -s SIGTERM squid&>/dev/null    sleep 5done# squid remove (if exist)apt purge -y squid-&>/dev/nullrm -rf /var/spool/squid- /var/log/squid- /etc/squid- /dev/shm/-&>/dev/null# squid install (you can use 'squid-openssl' or 'squid')apt install -y squid-openssl squid-langpack squid-common squidclient squid-purge# create logif [!-d /var/log/squid ];then    mkdir -p /var/log/squidfi&>/dev/nullif [[!-f /var/log/squid/{access,cache,store,deny}.log ]];then    touch /var/log/squid/{access,cache,store,deny}.logfi&>/dev/null# permissionschown -R proxy:proxy /var/log/squid# enable servicesystemctlenable squid.servicesystemctl start squid.serviceecho"Done"

Capture IPv4 from downloaded public blocklists (seeSOURCES) and unifies them in a single file.

Most of theSOURCES contain millions of invalid and nonexistent IP. Then, a double check of each IP is done (in 2 steps) via DNS and invalid and nonexistent are excluded from Blackip. This process may take. By default it processes in parallel ≈ 6k to 12k x min, depending on the hardware and bandwidth.

HIT 8.8.8.8Host 8.8.8.8.in-addr.arpa domain name pointer dns.googleFAULT 0.0.9.1Host 1.9.0.0.in-addr.arpa. not found: 3(NXDOMAIN)

Run Squid-Cache with BlackIP and any error sends it toSquidError.txt on your desktop.

BlackIP: Done 02/02/2024 15:47:14

• tw.txt containing IPs of teamviewer servers. By default they are commented. To block or authorize them, activate them inbipupdate.sh. To update it usetw.sh.
• You must activate the rules inSquid before usingbipupdate.sh.
• Some lists have download restrictions, so do not runbipupdate.sh more than once a day.
• During the execution ofbipupdate.sh it will request privileges when needed.
• If you useaufs, temporarily change it toufs during the upgrade, to avoid:ERROR: Can't change type of existing cache_dir aufs /var/spool/squid to ufs. Restart required.
• If someone considers that a domain should not be on Blackip, they can create anIssue and notify it to remove it.

allowip.txt is already updated and optimized. The update process ofallowip.txt is executed in sequence by the scriptaipupdate.sh.

wget -q -N https://raw.githubusercontent.com/maravento/blackip/master/bipupdate/wlst/aipupdate.sh&& chmod +x aipupdate.sh&& ./aipupdate.sh

• abuse.ch - Feodo Tracker
• abuse.ch - Zeustracker blocklist
• alienvault - reputation
• BBcan177 - minerchk
• BBcan177 - pfBlockerNG Malicious Threats
• binarydefense - Artillery Threat Intelligence Feed and Banlist Feed
• blocklist.de - export-ips_all
• blocklist.de - IPs all
• Cinsscore - badguys
• CriticalPathSecurity - Public-Intelligence-Feeds
• dan.me.uk - TOR Node List
• darklist - raw
• dshield.org - block
• ellio.tech - Threat List
• Emerging Threats - compromised ips
• Emerging Threats Block
• Firehold - Forus Spam
• Firehold - level1
• Greensnow - blocklist
• IPDeny - ipblocks
• Myip - full BL
• MyIP - latest BL
• Nick Galbreath client9 - datacenters
• OpenBL - base
• opsxcq - proxy-list
• Project Honeypot - list_of_ips
• Rulez - BruteForceBlocker
• rulez.sk - bruteforceblocker
• SecOps-Institute - TOR Exit Node List
• Spamhaus - drop-lasso
• stamparm - ipsum
• StopForumSpam - 180
• StopForumSpam - Toxic CIDR
• torproject - TOR BulkExitList
• Uceprotect - backscatterer Level 1
• Uceprotect - backscatterer Level 2
• Uceprotect - backscatterer Level 3
• Ultimate Hosts IPs Blocklist - ips
• yoyo - adservers

• abuse.ch - Ransomwaretracker
• Malc0de - IP Blocklist
• Malwaredomain - IP List
• Maxmind - high-risk-ip-sample-list
• unsubscore - blacklist

• Allow IP/CIDR extra
• Allow IPs
• Allow URLs
• Amazon AWS
• Block IP/CIDR Extra
• DNS
• IANA
• Microsoft Azure Datacenter

• cidr2ip
• Debug IPs

Changes must be proposed via Issues. We do not accept Pull Requests.

We thank all those who contributed to this project. Those interested may contribute sending us new "Blocklist" links to be included in this project.

Special thanks to:Jhonatan Sneider

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Due to recent arbitrary changes in computer terminology, it is necessary to clarify the meaning and connotation of the termblacklist, associated with this project:

In computing, a blacklist, denylist or blocklist is a basic access control mechanism that allows through all elements (email addresses, users, passwords, URLs, IP addresses, domain names, file hashes, etc.), except those explicitly mentioned. Those items on the list are denied access. The opposite is a whitelist, which means only items on the list are let through whatever gate is being used. SourceWikipedia

Therefore,blacklist,blocklist,blackweb,blackip,whitelist and similar, are terms that have nothing to do with racial discrimination.