Repository files navigation

Thespf_fetch project is a collection of utilities to make managinggreylisting easier.

Greylisting is a technique for defeating spam by temporarily rejectingemail unknown senders. Well-behaved mail servers will try again aftera short interval, per theRFC. Spammers will usually move on and nottry again. Once the rejected server tries again, it's ip address isadded to a list of known-good mail servers for a period time, often amonth or longer.

One of the problem with greylisting, however, is that large senderslike Google, Microsoft, Yahoo and others have multiple outbound mailservers and do not often retry from the same host, ip address or blockof ip addresses. This results in the greylisting software rejectingthe mailer numerous times until the email is redelivered by one of thealready greylisted ip addresses.

One naive technique for pre-greylisting large mailers like Google isto add theirMX records to the approved senders list.Unfortunately, large companies rarely use the same servers and ipaddresses for both sending and receiving.

Thespf_fetch project uses a simple technique for pre-determiningwhich ip addresses and blocks of addresses to add to the approvedlist:SPF records.Sender Policy Framework record are DNS entries that list ip addressesand blocks of ip addresses that the company will send email from. Notall companies haveSPF records, but many do. Nearly all the largecompanies like Google and Yahoo do, in part because they helped definethe SPF standard.

Looking upSPF records is not a simple matter of typingdig gmail.com SPF.SPF records are not a record type. They're storedasTXT fields and have to be parsed. They caninclude therecords from another domain orredirect to another domain. To getthe full list of ip addresses authorized to send on behalf of a domainrequires multiple recursive lookups.

This is wherespf_fetch comes in.

spf_fetch is a utility to recursively look-upSPF records from alist of domains. The output is a list of IPv4 and IPv6 addresses andblocks. This list can be used to create a whitelist of outboundMTA addresses for email greylisting utilities like OpenBSDspamd(8).

• Fully-recursive look ups
• Specify just IPv4 or IPv6 records (defaults to both)
• Specify DNS server (defaults to system defined)
• Lookup from file, command-line orstdin
• Filters to process, transform or filter addresses after recursivelookup

See the man page for further details.

spf_fetch can now run user-supplied filters to process, transform orremove addresses after lookup but before returning them tostdout

Ships withfilters/filter_sipcalc to validate IPv4 and IPv6addresses.

Note: The filterrequiressipcalc so is not runwithout explicitly adding with command-line parameter-t filters/filter_sipcalc.

Given a list of domains like in the examples/example_domains:

gmail.comgmx.comgooglemail.comgoogle.com

spf_fetch will return:

#IPs for gmail.com...#IPs for gmx.com...213.165.64.0/2374.208.5.64/2674.208.122.0/26212.227.126.128/25212.227.15.0/24212.227.17.0/2774.208.4.192/2682.165.159.0/24217.72.207.0/27#IPs for googlemail.com...#IPs for google.com...#IPs for _spf.google.com...#IPs for _netblocks.google.com...64.18.0.0/2064.233.160.0/1966.102.0.0/2066.249.80.0/2072.14.192.0/1874.125.0.0/16108.177.8.0/21173.194.0.0/16207.126.144.0/20209.85.128.0/17216.58.192.0/19216.239.32.0/19#IPs for _netblocks2.google.com...#IPs for _netblocks3.google.com...172.217.0.0/19

Each domain from the list and all domains discovered asinclude orredirect domains in the SPF records will be recursively looked up toget their relevant IPs addresses.

To get a good list of domains, see the github mailcheck(https://github.com/mailcheck/mailcheck/wiki/List-of-Popular-Domains)

See filecommon_domains for a properly formatted version of thefile.

spf_update_pf is a utility that will callspf_fetch with a file ofdomains that should be whitelisted and add them to a table in OpenBSDpf(4).

spf_update_pf defaults to look for a file calledcommon_domains in/etc/mail. It will callspf_fetch with the file as the input andredirect the output to a file calledcommon_domains_ips in/etc/mail.

spf_update_pf defaults to add the ip addresses to a table calledcommon_white defined inpf.conf(5).

• Specify file andpf(5) table
• Pass additional arguments tospf_fetch

See the man page for further details.

The output fromspf_update_pf is now directed tocommon_domains_ips rather than the previouscommon_domains_white.If you're using apf(5) table you'll want to rewrite line inpf.conf(5) to something like:

table <common_white> persist file "/etc/mail/common_domains_ips"

spf_mta_capture is anexperimental utility to watch a log fileto capture the domains of all outbound mail. It will then retrieve theSPF records for the domain by callingspf_fetch and then add them toa table inpf(5).

At the moment,spf_mta_capture only parses log files created byOpenSMTPD.

spf_mta_capture is meant to run as apipe program bysyslogd(8).

• Writes send-to domains to file with timestamp in comment on sameline
• Specify output file andpf(5) table
• Pass additional arguments tospf_fetch
• Truncate entries older thannow() - n seconds
• Specify truncation seconds

See the man page for further details.

Copyright (c) 2016-2017 Aaron Poffenbergerakp@hypernote.com

Permission to use, copy, modify, and distribute this software for anypurpose with or without fee is hereby granted, provided that the abovecopyright notice and this permission notice appear in all copies.

THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIESWITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OFMERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FORANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGESWHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN ANACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OFOR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

Let me know byIssue or email if you find bugs or other problems.