Repository files navigation

IntelMQ command line tool to process events.

Callintelmqcbmail --help to see the current usage.

The concept’s documentation can be found here:https://intevation.github.io/intelmq-mailgen/

These libraries and programs are required:

• The python librarypsycopg2 (python3-psycopg2) for PostgreSQL communication.
• The python librarygpg (python3-gpg), part of the library gpgme. Dueto issues with Ubuntu 20.04, this dependency is not installed wheninstalled withpip orsetup.py Other means of distributions (debpackages) are not affected by this bug.
• GnuPG (v>=2.2) forpython3-gpg.

As a Python3 application, see theinstall_requires section insetup.py for its dependencies.

If you install the deb-packages, the package management handles alldependencies.

For an installation from source use this command:

pip3 install -v -e .

In order to use IntelMQ Mailgen, you require a workingcertbund-contact-expert in IntelMQ, as Mailgen makes use of informationand data which is not available in the IntelMQs default fields.

For Mailgen to work, the following IntelMQ bots will need to beconfigured first:

1. Expert: CERT-bund Contact Database
2. Expert: CERT-bund Contact Rules
3. Output: PostgreSQL

Youmust follow the setup instructions for these bots before settingup Mailgen.

Theintelmq-events database and theintelmq database-user shouldalready have been set up by the configuration of a PostgreSQL output bot(SQL output bot with engine postgresql).For use with Mailgen this setup has to be extended:

As database-superuser (usually via system user postgres):

1. Create a new database-user:

   createuser --encrypted --pwprompt intelmq_mailgen

2. Extend the database:psql -f sql/notifications.sql intelmq-events

3. Grantintelmq the right to insert new events via a trigger:psql -c "GRANT eventdb_insert TO intelmq" intelmq-events

4. Grant the new user the right to send out notifications:psql -c "GRANT eventdb_send_notifications TO intelmq_mailgen" intelmq-events

The events written into the events database have been processed by therules bot which adds notification directives to the events. Thedirectives tell mailgen which notifications to generate based on thatevent. The statements insql/notifications.sql add triggers andtables to the event database that process these directives as they comein and prepare them for use by mailgen. In particular:

• Thedirectives table contains all the directives. The mainattributes of a directive are
  • ID of the event
  • recipient address
  • data format
  • template name (see “Templates” below)
  • how to aggregate
  • whether and when it was sent. this is the ID of the correspondingrow in thesent table (see below)
• When a new event is inserted into theevents table, a triggerprocedure extracts the directives and inserts them intodirectives.
• Thesent table records which notifications have actually beensent. Its main attributes are
  • the ticket number generated for the notification
  • a time stamp indicating when it was sent

When mailgen processes the directives, it reads the still unsentdirectives from the database, aggregates directives that aresufficiently similar that they could be sent in the same mail and callsa series of scripts for each of the aggregated directives. These scriptsinspect the directive and if they can process the directive generatemails from it. mailgen then sends these mails and records it in thesent table.

For every email sent by Mailgen a ticket number is generated. If a mailwas successfully sent, this number is stored in the tablesent,together with a timestamp when the mail was sent.

intelmq-mailgen currently searches for configuration files in twoplaces:

1. $HOME/.intelmq/intelmq-mailgen.conf (user configuration file) and
2. /etc/intelmq/intelmq-mailgen.conf (system configuration file).

Settings are read from both files with the one in the user’s homedirectory taking precedence.

The system configuration file path can be overridden with the--config command line parameter.

Both files must be in JSON format. A complete example can be found inintelmq-mailgen.conf.example.

gnupg_home has to point to the GnuPG home directory for emailsignatures. It must:

• contains the private and public key parts for the OpenPGP signaturewithout password protection.
• is read/writable for the user running intelmq-mailgen.

For example, the following steps will create such a directory and importa test signing key.

GNUPGHOME=/tmp/gnupghome mkdir $GNUPGHOMEchmod og-rwx $GNUPGHOMEGNUPGHOME=/tmp/gnupghome gpg2 --list-secret-keysGNUPGHOME=/tmp/gnupghome gpg2 --import src/intelmq-mailgen/tests/keys/test1.sec

Depending on your GnuPG version you may want to set additional optionsby editing$GNUPGHOME/gpg.conf.

For example, the following settings will set the default digestalgorithm, suppress emitting the GnuPG version, and add a comment linefor signatures:

personal-digest-preferences SHA256no-emit-versioncomment Key verification <https://example.org/hints-about-verification>

(See the GnuPG documentation for details.)

Now, you should be able to sign using this key without being promptedfor a passphrase. Try, for example:

echo Moin moin. | GNUPGHOME=/tmp/gnupghome gpg2 --clearsign --local-user "5F503EFAC8C89323D54C252591B8CD7E15925678"

mailgen comes with a templating mechanism that the scripts that processthe directives can use. This mechanism assumes that all templates arefiles in the directory from thetemplate_dir setting in theconfiguration file.

The scripts that come with mailgen simply take the template name fromthe directive they are processing. This means that the name is set bythe rules used by the rules bot, so see its documentation andconfiguration for which templates you need.

The first line of a template file is used as the subject line for mails.The remaining lines will become the mail body. The body may optionallybe separated from the subject line by one or more empty lines.

Both subject and body text will be interpreted asPython3 Templatestringsand may allow some substitutions depending on the format. Subject andbody allow the same substitutions.

Typically supported substitutions:

• All formats:

  • ${ticket_number}

• Additional substitutions for CSV-based formats:

  • ${events_as_csv} for the CSV-formatted event data. This is onlyuseful in the body.

• When aggregating by event fields the event fields can also be used.E.g. if a directive aggregates bysource.asn you can use${source.asn}

  Like the template name, aggregation is determined by the rules bot, sosee there for details.

The database section in the configuration may look like:

"database": {    "event": {        "name": "intelmq-events",        "username": "intelmq_mailgen",        "password": "your DB password",        "host": "localhost",        "port": 5432    },    "additional_directive_where": ""},

additional_directive_where parameter is optional and can contain SQLcode appended to theWHERE clause of theSELECT operation on thetabledirectives. TheAND is appended automatically. The columnsof tabledirectives are available asd3 and the columns of tableevents asevents. Normally the tableevents is not queriedand only joined for the where statement ifadditional_directive_where containsevents.. Examples:

"additional_directive_where":"\"template_name\" = 'qakbot_provider'""additional_directive_where":"events.\"feed.code\" = 'oneshot'"

Mind the correct quoting. If access to the table events is required, theused postgres user needsUPDATE permissions access to the table.This is by default not the case for mailgen-installations! Thisimperfection is a result of the update-locking on the tabledirectives and the join ofevents in the same sub-statement.

The logfile shall be monitored for errors to detect unwanted conditions.Especially grep for:

* 'ERROR'* 'Error:'

Each error condition should be handled by an administrator or servicetechnician soon. It is recommended to use a monitor system to notifyadministrators as soon as such a string occurs in the log.

There should be noTraceback or other ERROR information in the logof mailgen. Please read the lines in question, often they have goodhints about cause of the failure. Some problem may be solved bycorrecting the configuration.

INFO lines appear during normal operations. One condition to get an INFOmessage is if Mailgen detects that it is already running to that asecond instance does not start. If this is the case, the running Mailgenprocess may still have problems and during the nature of log file, themessages of the Mailgen that tries to start up, may appear interwovenwith the error conditions.

During a run, if mailgen is started a second time, it will fail to lockthe necessary rows in the database. Thepostgres.log file willrecord the failed locks, e.g. like

2020-12-15 09:00:02 UTC ERROR:  could not obtain lock on row in relation "directives"

which can be ignored.

Mailgen will try to continue processing directives and sending mails,even if some batch of mails could not be send for several reasons.

If it can’t find templates, for instance, it will continue with the nextdirective and log an error message and the stacktrace. The error messagecontains information about the directives that could not be processed.Thedirective_ids part in the output is a list with the IDs of therows in thedirectives table andevent_ids a list with ids forevents in theevents table.

This information can be used by an administrator to see which events andemails may not have gone out in detail, to deal with them later,possibly with a small script depending on the problem cause.

Generated using pgadmin4'sERD tool:

The source file is atintelmq-certbund-contact/sql/contactdb.erd.

• It is assumed that we need to protect against malicious external datacoming to us via the database.
• We do not need (or can) protect against local attacks withadministration rights.
• As our command will be able to run with and without user interaction,we assume that only users with administration rights have access tothe machine and are allowed to start the interactive variant.
• The private key material for signing will have no extra protection bypassphrase, thus the system itself needs to be secured adequately.(This can include separating the setup of intelmq itself on adifferent machine with only access to fill the database.)
• We should pay attention to preventing that the complete system becomesan effective signature (or encryption) oracle. To explain: Consider anattacker who will receive an automatic notification from our system.If this attacker also can trigger a warning over an used feed, she maypartly control which plaintext is to be signed (or somewhereencrypted) and gets the automated result. There is a small potentialunder some circumstances that this can be used for anadaptive-plaintext attack.

It is possible to define names for the CSV-columns in code. For instanceinexample_scripts/10shadowservercsv.py, the dictionarystandard_column_titles maps event field names to column titles.These are used by most of the CSV formats later defined intable_formats. The formats specified there can still use specialcolumn titles if necessary.

Currently, data is not transformed when it is being added to the CSVoutput.

Mailgen always removes the “UTC” notations from time stamps intime.source. It ensures that time stamps will always be UTC.

An easy way to test the actual sending of emails is to use Python’ssmtpd module running theDebuggingServer:

python3 -m smtpd -d -n -c DebuggingServer localhost:8025

(Don’t forget to configure the corresponding SMTP host and port in yourconfig.)

If you want to capture emails in Maildir format you can usehttps://pypi.org/project/dsmtpd/, e.g.

git clone git://github.com/matrixise/dsmtpd.gitcd dsmtpdpython3 -m dsmtpd -i localhost -p 8025 -d /path/to/Maildir

/path/to/Maildir has to be either an existingMaildir or non-existing, inwhich case it will be created by dsmtpd.

You can access the Maildir with mutt, for example:

mutt -f  /path/to/Maildir

Hint: By defaultEsc P will trigger mutt’s<check-traditional-pgp>function, in case youwant to check a no-MIME signature.

The test suite is split into two parts because some tests may faildepending on hardware specs (execution time) and their failure would notindicate errors per se.

The regular unit tests which must succeed can be started withmake check; to run the complete test suite, usemake check_all.

The intelmq-mailgen file was initially copied fromhttps://github.com/certat/intelmq/blob/a29da5d798bd114535326ffdd2f5000c4b6a21e7/intelmq/bin/intelmqcli(revision from 2016-03-08).