SpamAssassin+DCC+Razor howto , for Qmail
SpamAssassin + DCC + Razor
SpamAssassin:
If you have not installed spamassassin already, now is the time to do it. If you already have it, you should do an attempt to bring it to the latest release.
yum -y install spamassassin
or
yum -y update spamassassin
You can start / restart spamassassin service right away:
service spamassassin restart
chkconfig --level 35 spamassassin on
Create a user and group for spamassassin:
groupadd -g 711 spamd
useradd -u 711 -g 711 -s /home/spamd spamd
Edit /etc/sysconfig/spamassassin:
vi /etc/sysconfig/spamassassin
and replace it’s contents with the following line. If the file does not exist, create it and add the following line:
SPAMDOPTIONS="-x -u spamd -H /home/spamd -d"
Here is a little explanation of spamd options:
-c, –create-prefs Create user preferences files (we don’t need it)
-x, –nouser-config Disable user config files
-d, –daemonize Daemonize
-m num, –max-children=num Allow maximum num children
-u username, –username=username Run as username
-g groupname, –groupname=groupname Run as groupname (should we use this too? )
-v, –vpopmail Enable vpopmail config (we “should” need it, but not using at the moment)
-x, –nouser-config Disable user config files
-H [dir], –helper-home-dir[=dir] Specify a different HOME directory
vi /etc/mail/spamassassin/local.cf
Add the following line…
required_hits 5
Save and exit from the file.
You will need the following Perl modules for spamassassin:
yum -y install perl-Digest-SHA1
perl-Digest-HMAC
perl-Net-DNS
perl-HTML-Tagset
perl-HTML-Parser
perl-Time-HiRes
perl-suidperl
perl-DateManip
perl-TimeDate
QMR package refers to install perl-Mail-SpamAssassin , but that is not available through yum. And the provided RPM is too old to be compatible with current spamassassin-3.1.9 . So I installed it through CPAN too.
perl -MCPAN -e “install Time::HiRes”
perl -MCPAN -e “install Parse::Syslog”
perl -MCPAN -e “install Statistics::Distributions”
perl -MCPAN -e “install Mail::SPF::Query”
perl -MCPAN -e “install IP::Country::Fast”
perl -MCPAN -e “install MIME::Base64”
perl -MCPAN -e “install Getopt::Long”
perl -MCPAN -e “install URI::Escape”
perl -MCPAN -e “install Mail::SPF”
perl -MCPAN -e “install Mail::SpamAssassin”
perl -MCPAN -e “install Bundle::CPAN”
perl -MCPAN -e “install IO::Zlib”
perl -MCPAN -e “install Test::Harness”
perl -MCPAN -e “install Test::Simple”
perl -MCPAN -e “install Mail::DKIM”
perl -MCPAN -e “install Mail::DomainKeys”
perl -MCPAN -e “install Crypt::OpenSSL::Bignum”
perl -MCPAN -e “install Archive::Tar”
perl -MCPAN -e “install IO::Socket::INET6”
perl -MCPAN -e “install IO::Socket::SSL”
perl -MCPAN -e “install Net::Ident”
perl -MCPAN -e “install Encode::Detect”
perl -MCPAN -e “install Razor2::Client::Agent” # will be installed with Razor software
perl -MCPAN -e “install File::Copy” # problem
The following perl modules are used by various .pre files, in spamassassin. They are already installed as a result of installing “Mail::SpamAssassin”, earlier . You may want to make sure that they are also installed.
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::RelayCountry”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::URIDNSBL”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::Hashcash”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::SPF”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::DCC”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::Pyzor”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::Razor2”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::SpamCop”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::AntiVirus”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::AWL”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::AutoLearnThreshold”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::WhiteListSubject”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::DomainKeys”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::MIMEHeader”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::ReplaceTags”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::DKIM”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::Check”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::HTTPSMismatch”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::URIDetail”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::Shortcircuit”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::Bayes”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::BodyEval”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::DNSEval”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::HTMLEval”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::HeaderEval”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::MIMEEval”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::RelayEval”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::URIEval”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::WLBLEval”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::VBounce”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::Rule2XSBody”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::ASN”
perl -MCPAN -e “install Mail::SpamAssassin::Plugin::ImageInfo”
📝Note: Modules installation will fail on some (sometimes many) of the modules. You have to find a way to install them manually. Normally that involves downloading the related tarball from CPAN site and compiling it manually.
Use the link http://www.yrex.com/spam/spamconfig.php to create a new local.cf file for spamassassin. You will need to adjust a few things in the new file:
…
…
# ok_languages should be disabled.
# ok_languages en
# SpamAssassin 3.1 Note: Language checking has been moved to a plugin in version 3.1.
# This setting will not work unless your administrator has enabled the TextCat plugin
# in /etc/mail/spamassassin/v310.pre.
Time to setup / enable / disable various SpamAssassin checks in your config files:
vi /etc/mail/spamassassin/init.pre
…
loadplugin Mail::SpamAssassin::Plugin::RelayCountry
# Because of the change of how spammers act nowadays,
# and how hosts are setup on the internet, DNS Black listing is not so effective for me.
# loadplugin Mail::SpamAssassin::Plugin::URIDNSBL
# SPF is disabled here, because I am performing SPF checks at SMTP level.
#loadplugin Mail::SpamAssassin::Plugin::SPF
…
📝Note that I have disabled a few plug-ins. The network checks, especially will be dealt on network level / smtpd level, instead of letting them reach spamassassin. Also AntiVirus check would also be performed with ClamAV, when called from Qmail-Scanner. So I do not see a need to check a mail twice for certain tests, such as RBL, AntiVirus, SPF, etc.
vi /etc/mail/spamassassin/v310.pre
…
loadplugin Mail::SpamAssassin::Plugin::DCC
# I am not using Pyzor
# loadplugin Mail::SpamAssassin::Plugin::Pyzor
loadplugin Mail::SpamAssassin::Plugin::Razor2
# loadplugin Mail::SpamAssassin::Plugin::SpamCop
# I am using Qmail-Scanner runs ClamAV on a mail ,
# before it hands it over to spamassassin. This test is redundant for me.
# loadplugin Mail::SpamAssassin::Plugin::AntiVirus
# loadplugin Mail::SpamAssassin::Plugin::AWL
loadplugin Mail::SpamAssassin::Plugin::AutoLearnThreshold
# loadplugin Mail::SpamAssassin::Plugin::WhiteListSubject
# This may be redundant if you also plan to use the DKIM plugin:
# loadplugin Mail::SpamAssassin::Plugin::DomainKeys
loadplugin Mail::SpamAssassin::Plugin::MIMEHeader
loadplugin Mail::SpamAssassin::Plugin::ReplaceTags
…
vi /etc/mail/spamassassin/v312.pre
…
loadplugin Mail::SpamAssassin::Plugin::DKIM
vi /etc/mail/spamassassin/v320.pre
…
loadplugin Mail::SpamAssassin::Plugin::Shortcircuit
loadplugin Mail::SpamAssassin::Plugin::ASN
loadplugin Mail::SpamAssassin::Plugin::Rule2XSBody
Compiling and Updating SpamAssassin rules:
It is better to compile SpamAssassin rules with “sa-compile”. This will result in faster processing. However sa-compile needs re2c .
Dowload re2c from:http://re2c.org/
or
ftp://fr.rpmfind.net/linux/dag/redhat/el5/en/x86\_64/dag/RPMS/re2c-0.13.1-1.el5.rf.x86\_64.rpm
📝Note: Download an i386 version if you are on a 32bit machine.
cd /downloads/
Use command line utility “links” to download this:
cd /downloads/
linksftp://fr.rpmfind.net/linux/dag/redhat/el5/en/x86_64/dag/RPMS/re2c-0.13.1-1.el5.rf.x86_64.rpm
Run sa-update to update any new rules:
sa-update -D
Now Run sa-compile:
sa-compile
Now you can restart spamassassin:
service spamassassin restart
You can test your SpamAssassin installation for errors by running the “–lint” tests :
spamassassin --lint -D
SpamAssassin is installed!
Set up DCC:
Introduction:
The idea of Distributed Checksum Clearinghouses, or DCC in short, is that if mail recipients could compare the mail they receive, they could recognize unsolicited bulk mail. A DCC server totals reports of checksums of messages from clients and answers queries about the total counts for checksums of mail messages. A DCC client reports the checksums for a mail message to a server and is told the total number of recipients of mail with each checksum. If one of the totals is higher than a threshold set by the client and according to local whitelists the message is unsolicited, the DCC client can log, discard, or reject the message.
Download and installation:
cd /downloads/
wget http://www.rhyolite.com/anti-spam/dcc/source/dcc.tar.Z
tar xzf dcc.tar.Z
cd dcc-1.3.111/
./configure
make
make install
Update SpamAssassin configuration file:
vi /etc/mail/spamassassin/local.cf
...
...
dcc_home /var/dcc
dcc_path /usr/local/bin/dccproc
# dcc_dccifd_path, should be the path to dccifd socket, which will become available when dcc service is started. dcc_dccifd_path /var/dcc/dccifd
vi /var/dcc/dcc_conf
. . .
DCCIFD_ENABLE=on
. . .
cp /var/dcc/libexec/rcDCC /etc/rc.d/init.d/dcc
chkconfig --level 35 dcc on
service dcc start
Check if the DCC processes are running:
[root@www dcc-1.3.111]# ps aux | grep -i dcc
root 30623 0.0 0.0 2492 268 ? Ss 16:44 0:00 /var/dcc/libexec/dccifd -tREP,20 -tCMN,5, -llog -wwhiteclnt -Uuserdirs -SHELO -Smail_host -SSender -SList-ID
root 30624 0.1 0.6 28992 6844 ? Sl 16:44 0:00 /var/dcc/libexec/dccifd -tREP,20 -tCMN,5, -llog -wwhiteclnt -Uuserdirs -SHELO -Smail_host -SSender -SList-ID
root 30629 0.0 0.0 4100 580 pts/0 R+ 16:44 0:00 grep -i dcc
Enable the DCC plugin in spamassassin .pre files. Also enable other plugins. Disable pyzor and remove it’s line from local.cf as well. Restart SpamAssassin.
Note for ports on Firewall:
DCC traffic is like DNS traffic. You should treat port 6277 like port 53. Allow outgoing packets to distant UDP port 6277 and incoming packets from distant UDP port 6277. If the command `cdcc info` says no DCC servers are answering, you may need to adjust your firewall.
Set up Razor:
Vipul’s Razor is a distributed, collaborative, spam detection and filtering network. Through user contribution, Razor establishes a distributed and constantly updating catalogue of spam in propagation that is consulted by email clients to filter out known spam. Detection is done with statistical and randomized signatures that efficiently spot mutating spam content. User input is validated through reputation assignments based on consensus on report and revoke assertions which in turn is used for computing confidence values associated with individual signatures.
Download and install Razor:
From: http://razor.sourceforge.net/
cd /downloads/
wget http://internap.dl.sourceforge.net/sourceforge/razor/razor-agents-2.85.tar.bz2
tar xjf razor-agents-2.85.tar.bz2
cd razor-agents-2.85
perl Makefile.PL && make && make install
Added the following to spamassassin’s local.cf file:
vi /etc/mail/spamassassin/local.cf
...
...
razor\_config /etc/razor/razor-agent.conf
OR
echo "razor\_config /etc/razor/razor-agent.conf" >> /etc/mail/spamassassin/local.cf
Then execute the following two commands:
razor-admin -d -home=/etc/razor -create
razor-admin -d -home=/etc/razor -register
Firewall ports for Razor:
Ensure outbound TCP port 2703 is open from your server.
Restart spamassassin service to load the new modules:
[root@www qmailnew\]# service spamassassin restart
Stopping spamd: [ OK ]
Starting spamd: [ OK ]
That’s all. You have quite good configuration of SpamAssassin.
Some Tips:
Remember, putting all the work to spamassassin on your mail server will result in high server load if there are a lot of mails coming in. Avoiding spam requires many moving parts in the system, including some protection at firewall, some at DNS (DNSBL tests), some protection at SMTP level (RBL, SPF, etc), some protection from AntiVirus, and some protection from SpamAssassin. Try keeping the number of SpamAssassin rules to minimum required. It will take you some time to have a correct balance between number of checks and the speed needed for processing mails. You can use the following command to check the average amount of time SpamAssassin spends on each incoming email. the lower the number means the faster your emails are processed through SpamAssassin checks. Having some figure around 10 seconds on average is kind of ok. However that entirely depends on site. If you have 10 email users on a server and you have an average of 20 emails coming in per hour and each SpamAssassin is taking 25 seconds on each message, there is something wrong. Most of the time there is problem of DNS resolution, and RBL lists. Check network connectivity and your firewall. DCC and Razor also needs ports to be open on firewall.
Here are a couple of commands you can execute to find various stats about your spamassassin:
This command will give you the average time (in seconds) which spamassassin is spending on your incoming messages:
# grep seconds /var/log/maillog\* | awk '{sum+=$13} END { print "Average scan time = " sum/NR " Total mails = " NR}'
The output of this command will look like:
Average scan time = 2.59446 Total mails = 22764
The following command you can use to find the number of emails grouped according to scan times:
# grep seconds /var/log/maillog\* | awk '{print $13}' | sort -n | uniq -c | sort -n -k 2
The output of this command will look something like shown below.The left column is number of emails and the right column is the number of seconds. From the output below, you can see the My best scan time is 0.6 seconds, but only three emails got scanned in that short time. The worst case is 106.7 seconds, but fortunately only one email took that long. I believe there must be something pretty wrong at that time. Average case seem to be hanging around 1.6 and 1.7 seconds, or you can say under 2 seconds!
3 0.6
9 0.7
4 0.8
3 0.9
1 1.0
15 1.2
39 1.3
104 1.4
2470 1.5
9551 1.6
5446 1.7
1111 1.8
278 1.9
152 2.0
99 2.1
…
…
1 51.1
1 52.6
1 65.5
1 67.7
1 104.2
1 106.7
Testing:
You can use the “spamassassin -D –lint” to check various rules an syntax of the configuration files. The output is huge, but worth checking.
# spamassassin -D –lint
[17042] dbg: logger: adding facilities: all
[17042] dbg: logger: logging level is DBG
[17042] dbg: generic: SpamAssassin version 3.2.5
[17042] dbg: config: score set 0 chosen.
[17042] dbg: util: running in taint mode? yes
[17042] dbg: util: taint mode: deleting unsafe environment variables, resetting PATH
[17042] dbg: util: PATH included ‘/usr/kerberos/sbin’, keeping
[17042] dbg: util: PATH included ‘/usr/kerberos/bin’, keeping
. . .
. . .
[17042] dbg: rules: compiled meta tests
[17042] dbg: check: is spam? score=4.205 required=5
[17042] dbg: check: tests=MISSING_DATE,MISSING_HEADERS,MISSING_SUBJECT,NO_RECEIVED,NO_RELAYS
[17042] dbg: check: subtests=__HAS_MSGID,__MISSING_REF,__MSGID_OK_DIGITS,__MSGID_OK_HOST,__MSOE_MID_WRONG_CASE,__NONEMPTY_BODY,__SANE_MSGID,__TVD_BODY,__UNUSABLE_MSGID
The other test is to test a sample spam mail and see if it gets marked as spam:
# spamassassin < /usr/share/doc/spamassassin-3.2.5/sample-spam.txt
Received: from localhost by www.example.com
with SpamAssassin (version 3.2.5);
Wed, 26 Aug 2009 11:43:59 +0300
From: Sender sender@example.net
To: Recipient recipient@example.net
Subject: *****SPAM***** Test spam mail (GTUBE)
Date: Wed, 23 Jul 2003 23:30:00 +0200
Message-Id: GTUBE1.1010101@example.net
X-Spam-ASN:
X-Spam-Flag: YES
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on www.example.com
X-Spam-Level: **************************************************
X-Spam-Status: Yes, score=1003.9 required=5.0 tests=DCC_CHECK,DIGEST_MULTIPLE,
GTUBE,NO_RECEIVED,NO_RELAYS,RAZOR2_CF_RANGE_51_100,RAZOR2_CF_RANGE_E4_51_100,
RAZOR2_CHECK shortcircuit=no autolearn=disabled version=3.2.5
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary=”———-=_4A94F5CF.F7EA9A22”
This is a multi-part message in MIME format.
-———–=_4A94F5CF.F7EA9A22
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
Spam detection software, running on the system “www.example.com”, has identified this incoming email as possible spam. The original message has been attached to this so you can view it (if it isn’t spam) or label similar future email. If you have any questions, see the administrator of that system for details.
. . .
. . .
Note about Subject Header rewriting:
There is a known problem of spam emails not being marked/tagged as SPAM (header remains unchanged), using latest versions of Qmail-Scanner with spamassassin. Basically in newer versions of Qmail-Scanner such as 2.06, the scanner uses “fast_spamassassin” by default. When that is used on a spam email, it just tells the scanner that email is spam and moves on. This way the subject header remains unchanged. The problem is fixed by either using “verbose_spamassassin” with qmail-scanner, or using fast_spamasssin with an additional string such as “fast_spamassassin=*****SPAM*****” . This is covered in the Qmail-Scanner article.