Currently all mail services run ontools-mail. A few pointers:

• Exim is hard. This documentation is not:http://bradthemad.org/tech/notes/exim_cheatsheet.php
• Most relevantly:exim -q -v to manually run over the queue once
• Log files are in/var/log/exim/*

Edit/etc/exim4/deny_senders.list and add the envelope from address that you want to block on a new line.

$echo'wiktcapt@tools.wmflabs.org'|sudo-itee-a/etc/exim4/deny_senders.listwiktcapt@tools.wmflabs.org$sudo-icat/etc/exim4/deny_senders.list#AddMAILFROMaddresstoblock.Oneperlinewiktcapt@tools.wmflabs.org

If an incident seems to be occurring, be safe and stop exim before you spend a lot of time digging into the root cause. Having our MX address put on a blacklist for spamming is worse than some downtime.

$sudo-ipuppetagent--disable"Investigating exim incident --$USER"$sudo-ipuppetagent-tv# Check to see that Puppet is actually disabled$sudo-iserviceexim4stop$psaxuww|grepexim# Did it really stop?

Now you can proceed to investigate the queue without more messages going in or out. SMTP is robust to network segmentation so even leaving things down for a few hours is not a huge problem. Messages will be delivered eventually.

Each exec node (and most other hosts actually) run a local copy of exim that queues messages for outbound delivery via thetools-mail smarthost. Even if you have purged the queue on the smarthost before restarting exim following an incident, there may be messages queued across the grid ready to flood in as soon as the service is back online.

1. Provision new node
2. Disable Puppet on new node, stop Exim service
3. Update Hiera keyprofile::toolforge::active_mail_relay to point to new instance
4. Run Puppet everywhere
5. Stop Puppet and Exim on the old node
6. Swap floating IP to new node
7. Enable Puppet and Exim service on new node
8. Decomission old node

• Help:Toolforge/Email - user docs
• General VPS Exim docs

• Toolforge admin