OpenSuSE mail filtering with rspamd
Introduction
This document describes how to set up spam filtering on OpenSuSE with rspamd. It assumes you already have a working mail system running with postfix and dovecot. Unless specified otherwise, all commands must be executed as root.
This document is largely based on the excellent rspamd guide for Debian that can be found on workaround.org
Installing the software
Rspamd is not part of the standard distribution repositories. So first we need to add the repository:
zypper addrepo https://download.opensuse.org/repositories/server:mail/openSUSE_Leap_15.1/server:mail.repo zypper refresh
We'll be using redis as back-end for Bayes so we'll need to install that as well:
zypper in redis rspamd
Configuring redis
copy /etc/redis/default.conf.example to /etc/redis/rspamd.conf
cp /etc/redis/default.conf.example /etc/redis/rspamd.conf
In /etc/redis/rspamd.conf set the following options. Options not mentioned here keep their default value
unixsocket /var/run/redis/rspamd.sock unixsocketperm 770 pidfile /var/run/redis/rspamd.pid logfile /var/log/redis/rspamd.log dir /var/lib/redis/rspamd/
Create the database dir:
install -d -o redis -g redis -m 0750 /var/lib/redis/rspamd/
Add limits (ulimit) to each service
install -d -m 0755 /etc/systemd/system/redis@rspamd.service.d echo "[Service] LimitNOFILE=10240" > /etc/systemd/system/redis@rspamd.service.d/limits.conf
Make sure the ownership is set properly on the configuration file:
chown root:redis /etc/redis/rspamd.conf
In order to read the socket later on, the rspamd user must be part of the redis group. So in /etc/group make sure _rspamd is a member of the redis group:
redis:x:472:_rspamd
And finally we start and enable the rspamd redis service:
systemctl start redis@rspamd systemctl enable redis@rspamd
Configuring postfix and rspamd
Now that the basics are setup we're ready to start filtering out spam.
Set rspamd to use the redis socket
First we need to tell rspamd to use the redis socket for communication. Create the file /etc/rspamd/local.d/redis.conf and in it put the following line:
servers = "/var/run/redis/rspamd.sock";
Edit /etc/rspamd/modules.d/neural.conf and /etc/rspamd/modules.d/history_redis.conf and change the line
servers = 127.0.0.1:6379;
to:
servers = "/var/run/redis/rspamd.sock";
And in /etc/rspamd/options.inc make sure you have:
control_socket = "$DBDIR/rspamd.sock mode=0600";
Now start and enable rspamd
systemctl start rspamd systemctl enable rspamd
Make Postfix use rspamd
To let postfix use rspamd add these lines in /etc/postfix/main.cf:
smtpd_milters = inet:127.0.0.1:11332
non_smtpd_milters = inet:127.0.0.1:11332
milter_protocol = 6
milter_mail_macros = i {mail_addr} {client_addr} {client_name} {auth_authen}
And restart postfix
systemctl restart postfix
Postfix will now connect to rspamd that is listening to TCP port 11332 on localhost (127.0.0.1) and pass the email over that connection. The smtpd_milters setting defines that connection for emails that came into the system from the internet via the SMTP protocol. The non_smtpd_milters setting is optional – it makes Postfix have all emails checked that originate from the system itself. Finally the milter_mail_macros setting defines several variables that rspamd expects for better spam detection. rspamd then runs its checks and tells Postfix whether the email should pass or get rejected.
Testing spam detection
For testing we can use a sample spam email that comes with SpamAssassin. It is called GTUBE (Generic Test for Unsolicited Bulk Email). It contains a certain articial pattern that is recognized as spam by SpamAssassin and rspamd.
Download the file on the for testing:
wget http://spamassassin.apache.org/gtube/gtube.txt
…and send that email to our test user John…
sendmail john@example.org < gtube.txt
Check your postfix log:
journalctl -u postfix
You will find something like this:
Nov 12 12:48:21 mail postfix/cleanup[1705]: 592CC1C2555: message-id=<GTUBE1.1010101@example.net> Nov 12 12:48:21 mail postfix/cleanup[1705]: 592CC1C2555: milter-reject: END-OF-MESSAGE from localhost[127.0.0.1]: 5.7.1 Gtube pattern; from=<root@mail.example.org> to=<john@example.org> Nov 12 12:48:21 mail postfix/cleanup[1705]: 592CC1C2555: to=<john@example.org>, relay=none, delay=0.07, delays=0.07/0/0/0, dsn=5.7.1, status=bounced (Gtube pattern)
The interesting parts are those printed in bold letters. “milter-reject” tells that the milter (rspamd) recommended to Postfix to reject the email. It gave the reason “5.7.1 Gtube pattern”. In mail communication you often find these three digit codes. They are defined in RFC 3463. The first digit is most important: 2 = Success 4 = Transient failure (temporary problem – come back later) 5 = Permanent failure (do not try again in this way)
So 5.7.1 tells us that the result code is a permanent failure in delivery. The 7 stands for an issue regarding the security policy. So it’s not a technical failure but instead a security-relevant component on the system has rejected the email. That’s what rspamd did. It even told us the reason: “Gtube pattern”. So you see that rspamd knows about the Gtube spam test pattern and reacts as expected.
Adding headers
As you may know an email contains of the header and the body. Your users will only see header information like the subject, the sender, the recipient and the date and time the email was sent. But there is way more information like the router the email travelled or extended headers added by the various mail server on the way to the destination. Such extended headers begin with an “X-“. rspamd can add such headers to help you filter out spam. For that purpose create a new configuration override file at /etc/rspamd/override.d/milter_headers.conf with this content:
extended_spam_headers = true;
It will add these headers:
X-Rspamd-Server: mail Authentication-Results: dmarc=fail reason="No valid SPF, No valid DKIM" … X-Rspamd-Queue-Id: C22E55A005B3 X-Spamd-Result: default: False [11.55 / 15.00] R_PARTS_DIFFER(0.27)[63.4%] FROM_NO_DN(0.00)[] RCVD_COUNT_ZERO(0.00)[0] R_DKIM_NA(0.00)[] FUZZY_DENIED(12.00)[1:19305c7fdd:1.00:bin,1:35699594fd:0.91:txt] RBL_SENDERSCORE(2.00)[55.181.23.94.bl.score.senderscore.com] ARC_NA(0.00)[] RCPT_COUNT_ONE(0.00)[1] RCVD_TLS_ALL(0.00)[] FROM_EQ_ENVFROM(0.00)[] R_SPF_SOFTFAIL(0.00)[~all] BAYES_HAM(-2.71)[98.75%] TO_MATCH_ENVRCPT_ALL(0.00)[] MIME_GOOD(-0.10)[multipart/related,multipart/alternative,text/plain] MID_RHS_MATCH_FROM(0.00)[] ASN(0.00)[asn:16276, ipnet:94.23.0.0/16, country:FR] TO_DN_NONE(0.00)[] DMARC_POLICY_SOFTFAIL(0.10)[Chronopost.fr : No valid SPF, No valid DKIM,none] SUBJECT_ENDS_EXCLAIM(0.00)[] X-Spam: Yes
Each of the uppercase symbols like FROM_HAS_DN means that a certain detection routing of rspamd was triggered. It does not necessarily mean something bad about the email. For example R_SPF_ALLOW has a negative score that lowers the total score because it is something good about the email. There are a several symbols with a 0.00 score. These do not change the score but show you what rspamd has found. But if you consider certain criteria good or bad then you can define your own scores for them.
The last line here is especially interesting because next on our list is…
Sending spam to the Junk folder
Your users will not realize that their spam emails have an added “X-Spam: Yes” header. Their emails just appear like normal in their inbox. So let’s aid them by moving spam to a separate Junk folder beneath their inbox automatically. Dovecot has support for Sieve filters which are basically scripts that run automatically whenever an email coming in.
John could log into Roundcube and configure a new Sieve filter for himself that would save any emails to his “Junk” folder if the header line “X-Spam: Yes” was found. This rule would be useful for all your users though so let’s find a general solution.
Dovecot supports global Sieve filters that apply to all users. Edit the file /etc/dovecot/conf.d/90-sieve.conf. Look for the “sieve_after” lines. They are commented out. So add a new line there:
sieve_after = /etc/dovecot/sieve-after
The “sieve after” filters are executed after the user’s filters. John can define his own filter rules. And after that Dovecot will run any filter rules it finds in files in /etc/dovecot/sieve-after. Create that directory:
mkdir /etc/dovecot/sieve-after
And create a new file /etc/dovecot/sieve-after/spam-to-folder.sieve reading:
require ["fileinto","mailbox"];
if header :contains "X-Spam" "Yes" {
fileinto :create "INBOX.Junk";
stop;
}
The “require” lines include functionality to move emails into certain folders (fileinto) and to create folders if they don’t exist yet (mailbox). Then if rspamd marked a header as spam it is moved into the INBOX.Junk folder which just appears as “Junk” to the user underneath their inbox.
Dovecot cannot deal with such human-readable files though. So we need to compile it:
sievec /etc/dovecot/sieve-after/spam-to-folder.sieve
That generated a machine-readable file /etc/dovecot/sieve-after/spam-to-folder.svbin.
Restart Dovecot:
systemctl restart dovecot
Now all your users will automatically get spam emails moved to their Junk folder.
Learning existing spam
There are three ways to get started with rspamd…
No Bayes training
This is not as bad as it sounds. rspamd has way more functionality to determine if an email is ham or spam. I recommend that you enable auto learning. Just create a new file /etc/rspamd/override.d/classifier-bayes.conf and make it contain:
autolearn = true;
This is a very conservative approach though. It learns emails as spam if they are so bad that they get rejected. And it learns emails as ham if they have a negative (=very good) score. The rspamd documentation has further examples how to fine-tune auto learning.
Using pre-built statistics
rspamd provides a sample database with over 3000 learned emails. They are from 2015 and perhaps are not a good start because the nature of spam changes over time.
Stop rspamd: systemctl stop rspamd
Take their pre-built statistics and put the *sqlite files into your /var/lib/rspamd directory.
wget https://rspamd.com/rspamd_statistics/bayes.ham.sqlite wget https://rspamd.com/rspamd_statistics/bayes.spam.sqlite
Fix the ownership of those files by running:
chown _rspamd._rspamd /var/lib/rspamd/*sqlite
Convert the sqlite3 database into redis:
rspamadm statconvert --spam-db bayes.spam.sqlite --ham-db bayes.ham.sqlite -h 127.0.0.1:6379 --symbol-spam BAYES_SPAM --symbol-ham BAYES_HAM
And finally start rspamd again…
systemctl start rspamd
Verify that the database is now filled by running:
rspamc stat
At the end of the output you will see something like:
Statfile: BAYES_SPAM type: redis; length: 0; free blocks: 0; total blocks: 0; free: 0.00%; learned: 2072; users: 1; languages: 0 Statfile: BAYES_HAM type: redis; length: 0; free blocks: 0; total blocks: 0; free: 0.00%; learned: 1603; users: 1; languages: 0 Total learns: 3675
Re-learn your existing ham and spam
You have been running your mail server for a while? And you have a good amount of ham and spam emails? Then let’s use those to train rspamd. It is important to train both ham and spam emails. The rspamc command will allow you to feed entire directories/folders of emails to the learning process. Fortunately we are using the Maildir format to store emails which rspamd can understand. An example to train spam:
rspamc learn_spam /var/vmail/example.org/john/Maildir/.INBOX.Junk/cur
And this would be an example to train ham:
rspamc learn_ham /var/vmail/example.org/john/Maildir/cur.
