WBITT

We Bring In Tomorrows Technology

  • Full Screen
  • Wide Screen
  • Narrow Screen
  • Increase font size
  • Default font size
  • Decrease font size

SpamAssassin+DCC+Razor howto , for Qmail

E-mail Print PDF
User Rating: / 7
PoorBest 

SpamAssassin + DCC + Razor

 

SpamAssassin:

If you have not installed spamassassin already, now is the time to do it. If you already have it, you should do an attempt to bring it to the latest release.

 

yum -y install spamassassin

 

or

 

yum -y update spamassassin

 

 

You can start / restart spamassassin service right away:

 

service spamassassin restart

chkconfig --level 35 spamassassin on

 

 

Create a user and group for spamassassin:

 

groupadd -g 711 spamd

useradd -u 711 -g 711 -s /home/spamd spamd

 

 

 

Edit /etc/sysconfig/spamassassin:

 

vi /etc/sysconfig/spamassassin

 

, and replace it's contents with the following line. If the file does not exist, create it and add the following line:

 

SPAMDOPTIONS="-x -u spamd -H /home/spamd -d"

 

 

Here is a little explanation of spamd options:

 

-c, --create-prefs Create user preferences files (we don't need it)

-x, --nouser-config Disable user config files

-d, --daemonize Daemonize

-m num, --max-children=num Allow maximum num children

-u username, --username=username Run as username

-g groupname, --groupname=groupname Run as groupname (should we use this too? )

-v, --vpopmail Enable vpopmail config (we "should" need it, but not using at the moment)

-x, --nouser-config Disable user config files

-H [dir], --helper-home-dir[=dir] Specify a different HOME directory

 

 

vi /etc/mail/spamassassin/local.cf

 

Add the following line...

required_hits 5

 

Save and exit from the file.

 

 

You will need the following Perl modules for spamassassin:

 

yum -y install perl-Digest-SHA1 \

perl-Digest-HMAC \

perl-Net-DNS \

perl-HTML-Tagset \

perl-HTML-Parser \

perl-Time-HiRes \

perl-suidperl \

perl-DateManip \

perl-TimeDate

 

 

QMR package refers to install perl-Mail-SpamAssassin , but that is not available through yum. And the provided RPM is too old to be compatible with current spamassassin-3.1.9 . So I installed it through CPAN too.

 

perl -MCPAN -e "install Time::HiRes"

perl -MCPAN -e "install Parse::Syslog"

perl -MCPAN -e "install Statistics::Distributions"

perl -MCPAN -e "install Mail::SPF::Query"

perl -MCPAN -e "install IP::Country::Fast"

perl -MCPAN -e "install MIME::Base64"

perl -MCPAN -e "install Getopt::Long"

perl -MCPAN -e "install URI::Escape"

perl -MCPAN -e "install Mail::SPF"

perl -MCPAN -e "install Mail::SpamAssassin"

perl -MCPAN -e "install Bundle::CPAN"

perl -MCPAN -e "install IO::Zlib"

perl -MCPAN -e "install Test::Harness"

perl -MCPAN -e "install Test::Simple"

perl -MCPAN -e "install Mail::DKIM"

perl -MCPAN -e "install Mail::DomainKeys"

perl -MCPAN -e "install Crypt::OpenSSL::Bignum"

perl -MCPAN -e "install Archive::Tar"

perl -MCPAN -e "install IO::Socket::INET6"

perl -MCPAN -e "install IO::Socket::SSL"

perl -MCPAN -e "install Net::Ident"

perl -MCPAN -e "install Encode::Detect"

perl -MCPAN -e "install Razor2::Client::Agent" # will be installed with Razor software

perl -MCPAN -e "install File::Copy" # problem

 

The following perl modules are used by various .pre files, in spamassassin. They are already installed as a result of installing "Mail::SpamAssassin", earlier . You may want to make sure that they are also installed.

 

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::RelayCountry"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::URIDNSBL"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::Hashcash"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::SPF"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::DCC"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::Pyzor"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::Razor2"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::SpamCop"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::AntiVirus"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::AWL"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::AutoLearnThreshold"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::WhiteListSubject"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::DomainKeys"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::MIMEHeader"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::ReplaceTags"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::DKIM"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::Check"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::HTTPSMismatch"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::URIDetail"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::Shortcircuit"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::Bayes"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::BodyEval"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::DNSEval"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::HTMLEval"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::HeaderEval"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::MIMEEval"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::RelayEval"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::URIEval"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::WLBLEval"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::VBounce"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::Rule2XSBody"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::ASN"

perl -MCPAN -e "install Mail::SpamAssassin::Plugin::ImageInfo"

 

Note: Modules installation will fail on some (sometimes many) of the modules. You have to find a way to install them manually. Normally that involves downloading the related tarball from CPAN site and compiling it manually.

 

 

Use the link ( http://www.yrex.com/spam/spamconfig.php ) to create a new local.cf file for spamassassin. You will need to adjust a few things in the new file:

 

...

...

# ok_languages should be disabled.

# ok_languages en

# SpamAssassin 3.1 Note: Language checking has been moved to a plugin in version 3.1.

# This setting will not work unless your administrator has enabled the TextCat plugin

# in /etc/mail/spamassassin/v310.pre.

 

 

Time to setup / enable / disable various SpamAssassin checks in your config files:

 

vi /etc/mail/spamassassin/init.pre

...

loadplugin Mail::SpamAssassin::Plugin::RelayCountry

# Because of the change of how spammers act nowadays,

# and how hosts are setup on the internet, DNS Black listing is not so effective for me.

# loadplugin Mail::SpamAssassin::Plugin::URIDNSBL

# SPF is disabled here, because I am performing SPF checks at SMTP level.

#loadplugin Mail::SpamAssassin::Plugin::SPF

...

 

 

Note that I have disabled a few plug-ins. The network checks, especially will be dealt on network level / smtpd level, instead of letting them reach spamassassin. Also AntiVirus check would also be performed with ClamAV, when called from Qmail-Scanner. So I do not see a need to check a mail twice for certain tests, such as RBL, AntiVirus, SPF, etc.

 

vi /etc/mail/spamassassin/v310.pre

...

loadplugin Mail::SpamAssassin::Plugin::DCC

# I am not using Pyzor

# loadplugin Mail::SpamAssassin::Plugin::Pyzor

loadplugin Mail::SpamAssassin::Plugin::Razor2

# loadplugin Mail::SpamAssassin::Plugin::SpamCop

# I am using Qmail-Scanner runs ClamAV on a mail ,

# before it hands it over to spamassassin. This test is redundant for me.

# loadplugin Mail::SpamAssassin::Plugin::AntiVirus

# loadplugin Mail::SpamAssassin::Plugin::AWL

loadplugin Mail::SpamAssassin::Plugin::AutoLearnThreshold

# loadplugin Mail::SpamAssassin::Plugin::WhiteListSubject

# This may be redundant if you also plan to use the DKIM plugin:

# loadplugin Mail::SpamAssassin::Plugin::DomainKeys

loadplugin Mail::SpamAssassin::Plugin::MIMEHeader

loadplugin Mail::SpamAssassin::Plugin::ReplaceTags

...

 

 

 

vi /etc/mail/spamassassin/v312.pre

...

loadplugin Mail::SpamAssassin::Plugin::DKIM

 

 

 

vi /etc/mail/spamassassin/v320.pre

...

loadplugin Mail::SpamAssassin::Plugin::Shortcircuit

loadplugin Mail::SpamAssassin::Plugin::ASN

loadplugin Mail::SpamAssassin::Plugin::Rule2XSBody

 

Compiling and Updating SpamAssassin rules:

It is better to compile SpamAssassin rules with “sa-compile”. This will result in faster processing. However sa-compile needs re2c .

 

Dowload re2c from: http://re2c.org/

or

ftp://fr.rpmfind.net/linux/dag/redhat/el5/en/x86_64/dag/RPMS/re2c-0.13.1-1.el5.rf.x86_64.rpm

Note: Download an i386 version if you are on a 32bit machine.

 

cd /downloads/

Use command line utility “links” to download this:

 

cd /downloads/

links ftp://fr.rpmfind.net/linux/dag/redhat/el5/en/x86_64/dag/RPMS/re2c-0.13.1-1.el5.rf.x86_64.rpm

 

 

 

Run sa-update to update any new rules:

sa-update -D

 

 

Now Run sa-compile:

sa-compile

 

 

Now you can restart spamassassin:

 

service spamassassin restart

 

 

You can test your SpamAssassin installation for errors by running the “--lint” tests :

 

spamassassin --lint -D

 

SpamAssassin is installed!

 


 

Set up DCC :

 

Introduction:

The idea of Distributed Checksum Clearinghouses, or DCC in short, is that if mail recipients could compare the mail they receive, they could recognize unsolicited bulk mail. A DCC server totals reports of checksums of messages from clients and answers queries about the total counts for checksums of mail messages. A DCC client reports the checksums for a mail message to a server and is told the total number of recipients of mail with each checksum. If one of the totals is higher than a threshold set by the client and according to local whitelists the message is unsolicited, the DCC client can log, discard, or reject the message.

 

 

Download and installation:

 

cd /downloads/

wget http://www.rhyolite.com/anti-spam/dcc/source/dcc.tar.Z

tar xzf dcc.tar.Z

cd dcc-1.3.111/

 

./configure

make

make install

 

 

Update SpamAssassin configuration file:

 

vi /etc/mail/spamassassin/local.cf

...

...

dcc_home /var/dcc

dcc_path /usr/local/bin/dccproc


# dcc_dccifd_path, should be the path to dccifd socket, which will become available when dcc service is started.

dcc_dccifd_path /var/dcc/dccifd

 

 

 

vi /var/dcc/dcc_conf

. . .

DCCIFD_ENABLE=on

. . .

 

 

cp /var/dcc/libexec/rcDCC /etc/rc.d/init.d/dcc


chkconfig --level 35 dcc on

service dcc start

 

 

Check if the DCC processes are running:

 

[root@www dcc-1.3.111]# ps aux | grep -i dcc

root 30623 0.0 0.0 2492 268 ? Ss 16:44 0:00 /var/dcc/libexec/dccifd -tREP,20 -tCMN,5, -llog -wwhiteclnt -Uuserdirs -SHELO -Smail_host -SSender -SList-ID

root 30624 0.1 0.6 28992 6844 ? Sl 16:44 0:00 /var/dcc/libexec/dccifd -tREP,20 -tCMN,5, -llog -wwhiteclnt -Uuserdirs -SHELO -Smail_host -SSender -SList-ID

root 30629 0.0 0.0 4100 580 pts/0 R+ 16:44 0:00 grep -i dcc

 

Enable the DCC plugin in spamassassin .pre files. Also enable other plugins. Disable pyzor and remove it's line from local.cf as well. Restart SpamAssassin.

 

Note for ports on Firewall:

DCC traffic is like DNS traffic. You should treat port 6277 like port 53. Allow outgoing packets to distant UDP port 6277 and incoming packets from distant UDP port 6277.

If the command `cdcc info` says no DCC servers are answering, you may need to adjust your firewall.



 

 

Set up Razor:

Vipul's Razor is a distributed, collaborative, spam detection and filtering network. Through user contribution, Razor establishes a distributed and constantly updating catalogue of spam in propagation that is consulted by email clients to filter out known spam. Detection is done with statistical and randomized signatures that efficiently spot mutating spam content. User input is validated through reputation assignments based on consensus on report and revoke assertions which in turn is used for computing confidence values associated with individual signatures.

 

Download and install Razor:

From: http://razor.sourceforge.net/

 

cd /downloads/

wget http://internap.dl.sourceforge.net/sourceforge/razor/razor-agents-2.85.tar.bz2

 

tar xjf razor-agents-2.85.tar.bz2

cd razor-agents-2.85

perl Makefile.PL && make && make install

 

 

Added the following to spamassassin's local.cf file:

 

vi /etc/mail/spamassassin/local.cf

...

...

razor_config /etc/razor/razor-agent.conf

 

OR

 

echo "razor_config /etc/razor/razor-agent.conf" >> /etc/mail/spamassassin/local.cf

 

Then execute the following two commands:

 

razor-admin -d -home=/etc/razor -create

razor-admin -d -home=/etc/razor -register

 

 

Firewall ports for Razor:

Ensure outbound TCP port 2703 is open from your server.

 

 

Restart spamassassin service to load the new modules:

 

[root@www qmailnew]# service spamassassin restart

Stopping spamd: [ OK ]

Starting spamd: [ OK ]

 

That's all. You have quite good configuration of SpamAssassin.

 

Some Tips:

Remember, putting all the work to spamassassin on your mail server will result in high server load if there are a lot of mails coming in.  Avoiding spam requires many moving parts in the system, including some protection at firewall, some at DNS (DNSBL tests), some protection at SMTP level (RBL, SPF, etc), some protection from AntiVirus, and some protection from SpamAssassin. Try keeping the number of SpamAssassin rules to minimum required. It will take you some time to have a correct balance between number of checks and the speed needed for processing mails. You can use the following command to check the average amount of time SpamAssassin spends on each incoming email. the lower the number means the faster your emails are processed through SpamAssassin checks. Having some figure around 10 seconds on average is kind of ok. However that entirely depends on site. If you have 10 email users on a server and you have an average of 20 emails coming in per hour and each SpamAssassin is taking 25 seconds on each message, there is something wrong. Most of the time there is problem of DNS resolution, and RBL lists. Check network connectivity and your firewall. DCC and Razor also needs ports to be open on firewall.

Here are a couple of commands you can execute to find various stats about your spamassassin:

 

This command will give you the average time (in seconds) which spamassassin is spending on your incoming messages:

# grep seconds /var/log/maillog* | awk '{sum+=$13} END { print "Average scan time = " sum/NR " Total mails = " NR}'

 

The output of this command will look like:

Average scan time = 2.59446 Total mails = 22764

 

The following command you can use to  find the number of emails grouped according to scan times:

# grep seconds /var/log/maillog* | awk '{print $13}' | sort -n | uniq -c | sort -n -k 2

 

The  output of this command will look something like shown below.The left column is number of emails and the right column is the number of seconds. From the output below, you can see the My best scan time is 0.6 seconds, but only three emails got scanned in that short time. The worst case is 106.7 seconds, but fortunately only one email took that long. I believe there must be something pretty wrong at that time. Average case seem to be hanging around 1.6 and 1.7 seconds, or you can say under 2 seconds!

 

3 0.6                                                                                         
9 0.7                                                                                         
4 0.8                                                                                         
3 0.9                                                                                         
1 1.0                                                                                         
15 1.2                                                                                         
39 1.3                                                                                         
104 1.4                                                                                         
2470 1.5                                                                                         
9551 1.6                                                                                         
5446 1.7                                                                                         
1111 1.8                                                                                         
278 1.9                                                                                         
152 2.0                                                                                         
99 2.1                                                                                         

...

... 

1 51.1
1 52.6
1 65.5
1 67.7
1 104.2
1 106.7

 

Testing:

You can use the "spamassassin -D --lint" to check various rules an syntax of the  configuration files. The output is huge, but worth checking.

 

# spamassassin -D --lint

 

[17042] dbg: logger: adding facilities: all         
[17042] dbg: logger: logging level is DBG           
[17042] dbg: generic: SpamAssassin version 3.2.5    
[17042] dbg: config: score set 0 chosen.            
[17042] dbg: util: running in taint mode? yes       
[17042] dbg: util: taint mode: deleting unsafe environment variables, resetting PATH
[17042] dbg: util: PATH included '/usr/kerberos/sbin', keeping                     
[17042] dbg: util: PATH included '/usr/kerberos/bin', keeping

. . .

. . .

[17042] dbg: rules: compiled meta tests

[17042] dbg: check: is spam? score=4.205 required=5

[17042] dbg: check: tests=MISSING_DATE,MISSING_HEADERS,MISSING_SUBJECT,NO_RECEIVED,NO_RELAYS

[17042] dbg: check: subtests=__HAS_MSGID,__MISSING_REF,__MSGID_OK_DIGITS,__MSGID_OK_HOST,__MSOE_MID_WRONG_CASE,__NONEMPTY_BODY,__SANE_MSGID,__TVD_BODY,__UNUSABLE_MSGID

 

The other test is to test a sample spam mail and see if it gets marked as spam:

 

# spamassassin < /usr/share/doc/spamassassin-3.2.5/sample-spam.txt


 

Received: from localhost by www.example.com

with SpamAssassin (version 3.2.5);

Wed, 26 Aug 2009 11:43:59 +0300

From: Sender < This e-mail address is being protected from spambots. You need JavaScript enabled to view it >

To: Recipient < This e-mail address is being protected from spambots. You need JavaScript enabled to view it >

Subject: *****SPAM***** Test spam mail (GTUBE)

Date: Wed, 23 Jul 2003 23:30:00 +0200

Message-Id: < This e-mail address is being protected from spambots. You need JavaScript enabled to view it >

X-Spam-ASN:

X-Spam-Flag: YES

X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on www.example.com

X-Spam-Level: **************************************************

X-Spam-Status: Yes, score=1003.9 required=5.0 tests=DCC_CHECK,DIGEST_MULTIPLE,

GTUBE,NO_RECEIVED,NO_RELAYS,RAZOR2_CF_RANGE_51_100,RAZOR2_CF_RANGE_E4_51_100,

RAZOR2_CHECK shortcircuit=no autolearn=disabled version=3.2.5

MIME-Version: 1.0

Content-Type: multipart/mixed; boundary="----------=_4A94F5CF.F7EA9A22"


This is a multi-part message in MIME format.


------------=_4A94F5CF.F7EA9A22

Content-Type: text/plain; charset=iso-8859-1

Content-Disposition: inline

Content-Transfer-Encoding: 8bit


Spam detection software, running on the system "www.example.com", has

identified this incoming email as possible spam. The original message

has been attached to this so you can view it (if it isn't spam) or label

similar future email. If you have any questions, see

the administrator of that system for details.

 

. . .

. . .

Note about Subject Header rewriting:

There is a known problem of spam emails not being marked/tagged as SPAM (header remains unchanged), using latest versions of Qmail-Scanner with spamassassin. Basically in newer versions of Qmail-Scanner such as 2.06, the scanner uses "fast_spamassassin" by default. When that is used on a spam email, it just tells the scanner that email is spam and moves on. This way the subject header remains unchanged. The problem is fixed by either using "verbose_spamassassin" with qmail-scanner, or using fast_spamasssin with an additional string such as "fast_spamassassin=*****SPAM*****" . This is covered in the Qmail-Scanner article.

You are here How To / Tutorials SpamAssassin+DCC+Razor howto , for Qmail