nospam server

Over the years I have built a number of spam filtering servers, and for some reason I have always called them nospam. At one time I was contributing to the gentoo howto on spam filtering, until it crashed. I use gentoo as the OS for the nospam boxes, although I am often advised not to by developers (due to the fact that if you miss too many updates you will eventually be in a bind). This spam filter has been discussed at length on the internet, but I repeat it here for my own personal reference. Another nice thing about pointing your MX to a mail filter such as this is that if you must down your internal mail server, this nospam box will spool mail for you while your internal mailer is down. Gentoo + Amavisd-new + Spamassassin + Postfix + ClamAV + unofficial plugins and scripts can make a powerful spam filtering machine.


Before we begin it is always useful to reflect on the fact that we are always standing on the shoulders of giants. None of this would be possible without a lot of others time, effort and energy, that so many have given freely onto others (Postfix runs a huge portion of the worlds email for example). Many people helped in creating this filter that works so well…

Overview of how this works:

Lets define what we are going first, then some details. This machine we are building will be the mail server “out front”. It will be the one your MX record points to. This will the box in between the internet and your internal mail server. It will forward all mail to you that you want and hopefully not forward mail you do not want. But just to be clear, this mail server will forward mail to you after checking it… unless you are down, then it will spool it for up to five days.

So, we are talking Internet -> This nospam box -> Internal mail server

This is basically a mail server that gets mail on port 25 just like any other mail server, does some very basic tests (the MTA, Postfix does the tests, like basic RBL, domain name checks and user checks), and then it hands that mail off to a “glue” framework called amavisd-new. Amavisd-new calls ClamAv to virus scan the mail. ClamAv is also using some extra dbs from SaneSecurity to double scan for phishing emails. After the mail comes back from ClamAv (note that at this point mail is simply discarded into the quarantine dir if it is found to contain a virus) it is then sent to SpamAssassin. SpamAssassin will check the mail against its bayseian filters, its own blacklists, any blacklist rules you have set, the extra SaneSecurity scores and then call Pyzor, Razor, and DCC to check for spam. At this point, the mail has a grand total spam “score” associated with it and amavisd-new will decide if the mail needs to be discarded or passed. If it is discarded a mail is sent to the spam email address you have setup (optional, but nice to keep it for a while) and the mail is discarded from the system. If the mail passes (the score is not high enough), the mail will get injected back into postfix on a different port on the local host for final delivery. Postfix will consult the transport tables for final delivery of the message.

Basic Hardware

Nospam is a gentoo linux server running on commodity hardware.  It is x86_64 (amd64 to gentoo users) and the system that I built contains the following hardware. Note you do not need this much ram or disk as of this writing.

  •  16GB of ram
  •   Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 8 core
  • Two 3TB SAS drives in RAID 1
  • LSI/Megasas MR9260-4i controller card.

OS Software

Gentoo was installed on the system, basically following the gentoo handbook.

If you do not have a copy of gentoo handy, the CD is less than 200MB and can be downloaded from the gentoo website. note that you can grab the minimal install CD, and if you are replacing this hardware with similar hardware, you want the amd64 version (which is what the gentoo people call x86_64, or more specifically, is what x86_64 actually is, since Intel licensed it from AMD.

Note that you do not need to compile the megaraid drivers provided by LSI. After installing gentoo on the hardware raid, all you need to do is:

emerge megacli

to install the megaraid command line interface/manager. A few scripts were written to check the megaraid status and email us if there is an issue. Lets install the script (megacheck.sh, simple terrible script) in /usr/local/bin. You can find it in the tarball at the end of this page. We need to actually do the check, so lets make a cron job for that:

vi /etc/cron.d/megaraid

In that file put:

 

MAILTO=””
0,10,20,30,40,50 * * * * root /usr/local/bin/megacheck.sh 2>&1 > /dev/null

A small loose end… we will be installing logwatch later, and also keeping the mail logs separate so that we can analyze them. Lets do this with syslog-ng now instead of later on.

 

vi /etc/syslog-ng/syslog-ng.conf
Leave the top the same up to the end of the source src{ block, delete the rest and add the following:

 

destination mail { file(“/var/log/mail.log”); };
destination messages { file(“/var/log/messages”); };
destination console_all { file(“/dev/tty12”); };

filter f_mail { facility(mail) };
filter f_messages { level(info..warn) and not facility(mail); };

log { source(src); filter(f_mail); destination(mail); };
log { source(src); filter(f_messages); destination(messages); };
log { source(src); destination(console_all); };

Basic Tools

At this point, you may want to do yourself a favour and get some basic tools on the box other than the base OS… some of these could be considered optional, some are not. The Mandatory items are marked with an *:

  • vim the basic text editor that is not annoying like nano, or the size of an OS like emacs
  • bind-tools * yes, we hate bind with a passion, but the bind-tools contains htdig
  • tcpdump
  • mailx *
  • telnet * you need this for testing email properly
  • screen so that when working remotely you don’t have to worry about getting disconnected
  • ntp and ntpclient * (this is pretty mandatory for a mail server)
  • logwatch to keep track of logs and have them emailed to you
  • logrotate * as these log files will get huge and slow the machine down to a craw

We strongly urge you to put all of them on, but if you really don’t feel like it, just the *s.

So, you can do the following:

 

emerge vim bind-tools tcpdump mailx telnet-bsd screen ntp ntpclient logwatch logrotate

The MTA

Setup without Spam Filtering (amavis, sa, clamav, etc)

We need to first get mail working on this machine in order to then process spam. If you are doing this from scratch, you will want to get postfix working all by itself first, and test it, and then get spam filtering working. If you try to get all of this working at first and it does not work, you will have a lot of trouble finding out where it is failing (or at least until you are used to it, it will be painful)… so here we go, lets get postfix working first.

 

emerge postfix

That should put postfix on the box, and the config files will be in /etc/postfix. First, let us edit main.cf and change things to suit our setup.

 

vi /etc/postfix/main.cf

Note, from now on we will assume that when we say edit a file, that you know to vi (or if you prefer something else, fine… I guess, you freak) the file and save and exit when you are done.

Here are the variables we would change in main:
Assume this spam filter box is called nospam.yourdomain.com

 

debugger_command =
PATH=/bin:/usr/bin:/usr/local/bin; export PATH; (echo cont;
echo where) | gdb $daemon_directory/$process_name $process_id 2>&1 >$config_directory/$process_name.$process_id.log & sleep 5
myorigin = yourdomain.com
myhostname = nospam.yourdomain.com
mydestination =
mynetworks = 127.0.0.0/8 192.168.6.0/24
biff = no
smtpd_banner = mail.yourdomain.com
message_size_limit = 100000000
mailbox_size_limit = 100000000
local_transport = error:no local mail delivery
local_recipient_maps =
transport_maps = hash:/etc/postfix/transport
relay_domains = yourdomain.com <if you have other domains, put them here
separated by commas>
smtpd_helo_required = yes
smtpd_sender_restrictions = reject_non_fqdn_sender, reject_unknown_sender_domain
smtpd_recipient_restrictions =
permit_mynetworks,
reject_unauth_destination,
reject_non_fqdn_recipient,
reject_rbl_client zen.spamhaus.org,
reject_rbl_client relays.ordb.org,
reject_unknown_sender_domain,
reject_rhsbl_sender bogusmx.rfc-ignorant.org

Save and exit that file.

Now, edit the /etc/postfix/transport  file so that it contains only the following: Where 1.2.3.4 is your internal mail server (the final destination for mail) that postfix will deliver to.

 

yourdomain.com relay:[1.2.3.4]
<yourotherdomain.com> relay:[1.2.3.4]

When you are done, turn that into a db file for postfix by doing:

 

postmap /etc/postfix/transport

Startup postfix:

 

/etc/init.d/postfix restart

At this point in time, postfix should actually run and forward all mail to the internal server located at 1.2.3.4, if that box allows it. If this fails you should certainly check the /var/lo /mail.log file. Test telnetting to the box and sending mail through the box to a valid internal email address. If this fails freak out or start reading about postfix setups. The most common error is one of connectivity or a missed/mis-configured configuration variable.

The Spam Filtering Systems

Ok, so now you have mail working through the box, great! Now we just need to add all of that other stuff and get it filtering mail.

FIRST, let us add a few small spam filtering measures to postfix itself, make
sure it works and then move on to amavisd-new and friends.

cd /etc/postfix

edit main.cf and add above smtpd_recipient_restrictions =

smtpd_client_restrictions =
reject_unknown_reverse_client_hostname
smtpd_helo_restrictions =
permit_mynetworks,
check_helo_access hash:/etc/postfix/helo_access

Save and exit that file.

Now, you need to edit /etc/postfix/helo_access and add:

yourdomain.com REJECT You are not me!
<your external IP> REJECT You are not me!
<your mailservers IP> REJECT You are not me!
127.0.0.1 REJECT You are not me!

This does not happen often, but sometimes people try to spoof your IP/domain, and that should catch it.

Save and exit that file. Now you need a hash of it so…

postmap /etc/postfix/helo_access

Restart postfix and test:

/etc/init.d/postfix reload

Telnet to port 25, hand forge some good mail to yourself. Enjoy reading your own words.

Good, now lets add a bit more. You should still be in the directory /etc/postfix
so just edit main.cf again and add the following UNDER smtpd_recipient_restrictions so that you end up with:

smtpd_recipient_restrictions =
permit_mynetworks,
reject_unauth_destination,
reject_unknown_sender_domain,
reject_unknown_recipient_domain,
reject_invalid_hostname,
reject_non_fqdn_hostname,
reject_non_fqdn_recipient,
reject_non_fqdn_sender,
reject_rbl_client zen.spamhaus.org,
reject_rbl_client relays.ordb.org,
reject_rbl_client b.barracudacentral.org,
reject_rbl_client bl.spamcop.net,
reject_rbl_client cbl.abuseat.org,
check_recipient_access pcre:/etc/postfix/recipient_checks.pcre,
check_client_access cidr:/etc/postfix/client_checks.cidr,
reject_rhsbl_sender bogusmx.rfc-ignorant.org,

Save and exit that file.

Still in /etc/postfix we need to edit those files we just added to smtpd_recipient_restrictions

edit /etc/postfix/recipient_checks.pcre and add the following:

 

# Postfix doesn’t relay by default. But it may *appear* to do so
# to some testers. The first two statements below remove all
# doubt.

/^\@/ 550 Invalid address format.
/[!%\@].*\@/ 550 This server disallows weird address syntax.

# Let email to the following destinations bypass all the remaining
# “reject” and “check” tests. We always want to let email for these
# recipients in.

/^postmaster\@/ OK
/^hostmaster\@/ OK
/^abuse\@/ OK

Save and exit that file.

Ok, we need to make sure cidr is included in our postfix build, so do a

 

postconf -m |grep cidr

If you see it, great! If not, you need to remove check_client_access cidr:/etc/postfix/client_checks.cidr, from smtpd_recipient_restrictions = or rebuild postfix with cidr included.

Edit /etc/postfix/client_checks.cidr
and add:

 

# Broadcast? Get lost
0.0.0.0/8 REJECT
10.0.0.0/8 REJECT
100.64.0.0/10 REJECT
127.0.0.0/8 REJECT
169.254.0.0/16 REJECT
172.16.0.0/12 REJECT
192.0.0.0/29 REJECT
192.0.2.0/24 REJECT
# Anycast should not be sending mail…
192.88.99.0/24 REJECT
192.168.0.0/16 REJECT
198.18.0.0/15 REJECT
198.51.100.0/24 REJECT
203.0.113.0/24 REJECT
# Multicast should not be sending mail…
224.0.0.0/4 REJECT
240.0.0.0/4 REJECT

Save and exit that file.

Again, restart postfix and test:

 

/etc/init.d/postfix reload

 

Telnet to port 25, hand forge some good mail to yourself. And again, enjoy reading your own words. Game plan, install the filters, configure them, change postfix to use them.

 

emerge amavisd-new spamassassin clamav razor dcc pyzor dspam

Configuring amavisd and clamd

After that is done, lets do the hard part and configure amavisd-new first. The “new” amavisd config file is very different from the old one. So different, I do not think I am even done going through it in its entirety. I have made it very operable with the way I used to set them up, and our spam went down so much, I am going to leave it for now… suffice it to say the amount of code in the config file is too large to just cut/paste here. So, in that vain, download the config file which is included in the main tarball at the end of this wiki and then go through it comparing it to the one on the new system. If you want the rock bottom minimum, that would most likely be something like:

 

$mydomain = ‘yourdomain.com’; # a convenient default for other settings

$MYHOME = ‘/var/amavis’; # a convenient default for other settings, -H
$TEMPBASE = “$MYHOME/tmp”; # working directory, needs to exist, -T
$ENV{TMPDIR} = $TEMPBASE; # environment variable TMPDIR, used by SA, etc.
$QUARANTINEDIR = “$MYHOME/quarantine”; # -Q

# Note, we are going to do syslog here so that we can have a logwatch add on script
# go through the amavisd logs in mail.log and mail us reports about amavisd…
$log_level = 2; # verbosity 0..5, -d
$log_recip_templ = undef; # disable by-recipient level-0 log entries
$do_syslog = 1; # log via syslogd (preferred)
$syslog_facility = ‘mail’; # Syslog facility as a string
# e.g.: mail, daemon, user, local0, … local7

@local_domains_maps = ( [“.$mydomain”, “yourotherdomain.com”] ); #

@mynetworks = qw( 127.0.0.0/8 1.2.3.0/24 );
$sa_tag_level_deflt = -100.0; # add spam info headers if at, or above that level
$sa_tag2_level_deflt = 4.0; # add ‘spam detected’ headers at that level
$sa_kill_level_deflt = 4.0; # triggers spam evasive actions (e.g. blocks mail)
$sa_dsn_cutoff_level = 9; # spam level beyond which a DSN is not sent

# I check for the big spams…
$sa_mail_body_size_limit = 10000*1024;

$mailfrom_notify_spamadmin = “spam.police\@$mydomain”;

$dspam = ‘dspam’;

$myhostname = ‘nospam.yourdomain.com’; # must be a fully-qualified domain name!

$notify_method = ‘smtp:[127.0.0.1]:10025’;
$forward_method = ‘smtp:[127.0.0.1]:10025’; # set to undef with milter!

$final_virus_destiny = D_DISCARD;
$final_banned_destiny = D_DISCARD;
$final_spam_destiny = D_DISCARD; #!!! D_DISCARD / D_REJECT
$final_bad_header_destiny = D_DISCARD;

$spam_quarantine_to = “spam-quarantine\@$mydomain”;

 

Don’t forget to uncomment the ClamAV portion:

 

[‘ClamAV-clamd’,
\&ask_daemon, [“CONTSCAN {}\n”, “/var/run/clamav/clamd.sock”],
qr/\bOK$/m, qr/\bFOUND$/m,
qr/^.*?: (?!Infected Archive)(.*) FOUND$/m ],

 

And make sure that in /etc/clamd.conf the LocalSocket variable is set to the same one above, which is:

 

LocalSocket /var/run/clamav/clamd.sock

Back in /etc/amavisd.conf make sure you put in the clamav-unofficial portion, which we like to put in right before @av_scanners and it looks like this:

 

#This is for the clamav-unofficial that we are running here.
@virus_name_to_spam_score_maps =
(new_RE( # the order matters!
[ qr’^Structured\.(SSN|CreditCardNumber)\b’ => 0.1 ],
[ qr’^(Heuristics\.)?Phishing\.’ => 0.1 ],
[ qr’^(Email|HTML)\.Phishing\.(?!.*Sanesecurity)’ => 0.1 ],
[ qr’^Sanesecurity\.(Malware|Rogue|Trojan)\.’ => undef ],# keep as infected
[ qr’^Sanesecurity\.’ => 0.1 ],
[ qr’^Sanesecurity_PhishBar_’ => 0 ],
[ qr’^Sanesecurity.TestSig_’ => 0 ],
[ qr’^Email\.Spam\.Bounce(\.[^., ]*)*\.Sanesecurity\.’ => 0 ],
[ qr’^Email\.Spammail\b’ => 0.1 ],
[ qr’^MSRBL-(Images|SPAM)\b’ => 0.1 ],
[ qr’^VX\.Honeypot-SecuriteInfo\.com\.Joke’ => 0.1 ],
[ qr’^VX\.not-virus_(Hoax|Joke)\..*-SecuriteInfo\.com(\.|\z)’ => 0.1 ],
[ qr’^Email\.Spam.*-SecuriteInfo\.com(\.|\z)’ => 0.1 ],
[ qr’^Safebrowsing\.’ => 0.1 ],
[ qr’^winnow\.(phish|spam)\.’ => 0.1 ],
[ qr’^INetMsg\.SpamDomain’ => 0.1 ],
[ qr’^Doppelstern\.(Scam4|Phishing|Junk)’ => 0.1 ],
[ qr’^ScamNailer\.’ => 0.1 ],
[ qr’^HTML/Bankish’ => 0.1 ], # F-Prot
[ qr’-SecuriteInfo\.com(\.|\z)’ => undef ], # keep as infected
[ qr’^MBL_NA\.UNOFFICIAL’ => 0.1 ], # false positives
[ qr’^MBL_’ => undef ], # keep as infected
));

 

We need to explicitly allow local mails, as the local logwatch we are sending to root will eventually get flagged as spam as it has a lot of “spam” type stuff in it (banned domain names, etc). This has happened to me personally on two systems, so I know this has to be put in here. Note, I found this bit of code on the net and tested it, but I commented out the skipping of the virus checks. Put the following before the standard “policy_bank{‘MYNETS’} statement, which should be right after the “$inet_socket_port” statement:

 

# explicitly allow local mail, do not scan at all.
@client_ipaddr_policy = (
[qw( 0.0.0.0/8 127.0.0.1/8 [::] [::1] )] => ‘LOCALHOST’,
\@mynetworks => ‘MYNETS’,
);

$policy_bank{‘LOCALHOST’} = {
originating => 1,
# we will still virus check it
# bypass_virus_checks_maps => [1],
bypass_spam_checks_maps => [1],
bypass_banned_checks_maps => [1],
};

Technically, you used to have to modify the /etc/conf.d/clamd file to get this to work. We think that modern gentoo systems have this set, but you can check. It should look like:

 

START_CLAMD=yes
START_FRESHCLAM=yes
CLAMD_NICELEVEL=3
FRESHCLAM_NICELEVEL=19

Configuring SpamAssassin, Razor, Pyzor

Lets configure razor first. Razor will run as amavisd, so lets give them a shell, set razor-admin and then set the shell back.

 

usermod -s /bin/bash amavis
su – amavis
razor-admin -create
exit
usermod -s /bin/false amavis

Lets go nuts and do this the other way too… still do not know which way is actually working, but after this, it works.

 

razor-admin -create
razor-admin -register

if it complains, register again. Don’t worry about the lack of a config file.

 

vi /root/.razor/razor-agent.conf

and change the line: debuglevel = 3
to: debuglevel = 0

now…

 

cp -r /root/.razor /var/amavis
chown -R amavis:amavis /var/amavis

Whew, done with Razor. Pyzor and DCC have no real setup, only in SA, so lets just check DCC first:

 

/usr/bin/cdcc info

This command should return a number of lines that have anon in them… like dcc1.dcc-servers.net,- RTT+1000 ms anon
I would expect a few (3-6). If you get none or only 127.0.0.1, check your network.

Now, it is time for spamassassin.

First, lets edit the main configuration file.

 

vi /etc/mail/spamassassin/local.cf

Make sure the following exists in the file:

 

use_bayes 1
bayes_auto_learn 1
bayes_ignore_header X-Spam-Status
skip_rbl_checks 0
use_razor2 1
use_dcc 1
dcc_path /usr/bin/dccproc
use_pyzor 1
################################################################################
# SaneSecurity & MSRBL Signatures
################################################################################
header L_AV_Phish X-Amavis-AV-Status =~ m{\bAV:(Email|HTML)\.Phishing\.}
header L_AV_Scam X-Amavis-AV-Status =~ m{\bAV:ScamNailer\.Phish\.}
header L_AV_Spam X-Amavis-AV-Status =~ m{\bAV:Email\.Spammail}
header L_AV_PhishHeur X-Amavis-AV-Status =~ m{\bAV:Phishing\.Heuristics\.}
header L_AV_DS_Phish X-Amavis-AV-Status =~ m{\bAV:Doppelstern\.Phishing\.}
header L_AV_DS_Scam X-Amavis-AV-Status =~ m{\bAV:Doppelstern\.Scam}
header L_AV_DS_Junk X-Amavis-AV-Status =~ m{\bAV:Doppelstern\.Junk\.}
header L_AV_SS_PhishBar X-Amavis-AV-Status =~ m{\bAV:Sanesecurity_PhishBar_}
header L_AV_SS_Phish X-Amavis-AV-Status =~ m{\bAV:Sanesecurity\.Phishing\.}
header L_AV_SS_TestSig X-Amavis-AV-Status =~ m{\bAV:Sanesecurity\.TestSig_}
header L_AV_SS_Spear X-Amavis-AV-Status =~ m{\bAV:Sanesecurity\.Spear\.}
header L_AV_SS_Malware X-Amavis-AV-Status =~ m{\bAV:Sanesecurity\.(Malware|Rogue|Troja
n)\.}
header L_AV_SS_Scam X-Amavis-AV-Status =~ m{\bAV:Sanesecurity\.(Scam[A-Za-z0-9]?)}
header L_AV_SS_Spam X-Amavis-AV-Status =~ m{\bAV:Sanesecurity\.(Bou|Cred|Dipl|Job|L
oan|Lott|Porn|Spam[A-Za-z0-9]?|Stk|Junk)\.}
header L_AV_SS_Hdr X-Amavis-AV-Status =~ m{\bAV:Sanesecurity\.Hdr\.}
header L_AV_SS_Img X-Amavis-AV-Status =~ m{\bAV:Sanesecurity\.(Img|ImgO|SpamImg)\.
}
header L_AV_SS_Bounce X-Amavis-AV-Status =~ m{\.Spam\.Bounce(\.[^., ]*)*\.Sanesecurit
y\b}
header __L_AV_SS X-Amavis-AV-Status =~ m{\bAV:Sanesecurity\.}
meta L_AV_SS_other __L_AV_SS && !(L_AV_SS_PhishBar|| L_AV_SS_Phish || L_AV_SS_TestSig || L_AV_SS_Spear || L_AV_SS_Scam || L_AV_SS_Spam || L_AV_SS_Malware || L_AV_SS_Hdr || L_AV_SS_Img || L_AV_SS_Bounce)
header L_AV_MSRBL_Img X-Amavis-AV-Status =~ m{\bAV:MSRBL-Images\b}
header L_AV_MSRBL_Spam X-Amavis-AV-Status =~ m{\bAV:MSRBL-SPAM\.}
header L_AV_MBL X-Amavis-AV-Status =~ m{\bAV:MBL_(?!NA\.UNOFFICIAL)}
header L_AV_SecInf X-Amavis-AV-Status =~ m{-SecuriteInfo\.com\b}
header L_AV_SpamDomain X-Amavis-AV-Status =~ m{^INetMsg\.SpamDomain\b}
header L_AV_Any X-Amavis-AV-Status =~ m{\bAV:}

score L_AV_Phish 14
score L_AV_Scam 10
score L_AV_Spam 5
score L_AV_PhishHeur 5
score L_AV_DS_Phish 14
score L_AV_DS_Scam 10
score L_AV_DS_Junk 8
score L_AV_SS_Phish 5
score L_AV_SS_PhishBar 0.5
score L_AV_SS_TestSig 0.123
score L_AV_SS_Spear 4
score L_AV_SS_Scam 8
score L_AV_SS_Spam 8
score L_AV_SS_Hdr 6
score L_AV_SS_Img 3.5
score L_AV_SS_Bounce 0.1
score L_AV_SS_other 1
score L_AV_SS_Malware 14
score L_AV_MBL 14
score L_AV_MSRBL_Img 3.5
score L_AV_MSRBL_Spam 6
score L_AV_SecInf 8
score L_AV_SpamDomain 6
score L_AV_Any 2

Don’t forget to put in a cron job to have spamassassin updated once a day:

 

vi /etc/cron.d/saupdate

and in that file put:

 

MAILTO=””
0 0 * * * root /usr/bin/sa-update > /dev/null 2>&1
10 0 * * * root /etc/init.d/spamd restart > /dev/null 2>&

ClamAV-unofficial

You can get the clamav-unofficial package at sourceforge. There are a few parts to this. First, there is the scores that you see above in the SA confguration file. These add to SA as it runs. There are also the listings above in the amavisd.conf file for it. In addition to that, we have to files that need to be added to the system. The configuration files are /etc/clamav-unofficial-sigs.conf and /usr/local/bin/clamav-unofficial-sigs.shl. Once you have installed those on the box, make sure you give them some proper permissions if they do not have them, like so:

 

chmod 755 /usr/local/bin/clamav-unofficial-sigs.sh

Also, you want to add a cronjob for this so that it grabs the new unofficials every day.

 

vi /etc/cron.d/clamavunofficial

and put this in it:
MAILTO=””
# Run every hour.
0 * * * * clamav /usr/local/bin/clamav-unofficial-sigs.sh -c /etc/clamav-unofficial-sigs.conf

Finally, run the above command once to get the dbs loaded onto the spam server. You should see them all download into /var/lib/clamav…

 

/usr/local/bin/clamav-unofficial-sigs.sh -c /etc/clamav-unofficial-sigs.conf

Of course, you did that as root, not clamav which has no shell, so change ownership of all the files you just downloaded:

 

chown -R clamav:clamav /var/lib/clamav/*

Ok, now that we are done with the spam scanning software, we need to go back and confgiure postfix to use it!!

Get spam from users (Exchange) back into the system

This one is somewhat tricky as the mail will be going to exchange. Outlook will re-write headers if you send the mail anywhere out of outlook. Due to this, one solution is to setup a dummy account on exchange and IMAP that users “spam” and “ham” folders (that you have created) onto the nospam server for processing by sa-learn. We do that in the following way:

First, we need to install fetchmail

 

emerge fetchmail

Create a user on exchange, we will call it spam_bucket and create a spam and ham folder under its “Inbox” folder. Lets say the password is “password” for this example.

Back on the linux box, create two files in /etc, one called fetchmailrcspam and one called fetchmailrcham, and in them put the following:

 

vi /etc/fetchmailrcham

#fetchmailrcham

poll <exchange server IP> protocol IMAP user “yourdomain.com/spam_bucket” with password “password” folder “Inbox/ham”

 

And the spam one

 

vi /etc/fetchmailrcspam
# fetchmailrcspam
poll <exchange server IP> protocol IMAP user “yourdomain.com/spam_bucket” with password “password” folder “Inbox/spam”

Finally, we need to create a cron job which will call these and then sa-learn to add them to spamassassin. We will do this with a file:

 

vi /etc/cron.d/fetchmail_spam
MAILTO=””
# Read the spam folder of the spam_bucket user every hour during working
# hours, then a few times after, and stuff that into the spam database
0 7-18,20,22 * * * amavis /usr/bin/fetchmail -f /etc/fetchmailrcspam -a -n -m ‘/usr/bin/sa-learn –dbpath /var/amavis/.spamassassin –spam’
# Read the ham folder of the spam_bucket user at midnight, 6am and 7pm,
# and stuff that into the ham database
0 0,6,19 * * * amavis /usr/bin/fetchmail -f /etc/fetchmailrcham -a -n -m ‘/usr/bin/sa-learn –dbpath /var/amavis/.spamassassin –ham’

Back in the MTA

Recall at the start we were going to do all of this filtering by having postfix do basic filtering on valid users, some basic syntax (RFC SMTP exchanges) and RBLs and then give that mail to amavisd. Well, we will now configure postfix to hand off the mail to amavisd. Amavisd will then hand it back to postfix for final delivery when it is done scanning.

Your /etc/postfix/main.cf file will need to have the following changed or added:
relay_recipient_maps = hash:/etc/postfix/relay_recipients
show_user_unknown_table_name = no
virtual_alias_maps = hash:/etc/postfix/virtual
header_checks = pcre:/etc/postfix/pcre-header.cf
body_checks = pcre:/etc/postfix/pcre-body.cf
content_filter = smtp-amavis:[127.0.0.1]:10024

If the files /etc/postfix/pcre-header.cf and /etc/postfix/pcre-body do not exist, create them. Postfix will not start and work properly without them now that we put them in the config file. So, if needed:

 

touch /etc/postfix/pcre-header.cf /etc/postfix/pcre-body.cf

We also need to change the /etc/postfix/master.cf file… we will add this at the bottom.
smtp-amavis unix – – n – 4 smtp
-o smtp_data_done_timeout=1200
-o smtp_send_xforward_command=yes
-o disable_dns_lookups=yes
-o max_use=20

127.0.0.1:10025 inet n – n – – smtpd
-o content_filter=
-o smtpd_delay_reject=no
-o smtpd_client_restrictions=permit_mynetworks,reject
-o smtpd_helo_restrictions=
-o smtpd_sender_restrictions=
-o smtpd_recipient_restrictions=permit_mynetworks,reject
-o smtpd_data_restrictions=reject_unauth_pipelining
-o smtpd_end_of_data_restrictions=
-o smtpd_restriction_classes=
-o mynetworks=127.0.0.0/8
-o smtpd_error_sleep_time=0
-o smtpd_soft_error_limit=1001
-o smtpd_hard_error_limit=1000
-o smtpd_client_connection_count_limit=0
-o smtpd_client_connection_rate_limit=0
-o receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_milters
-o local_header_rewrite_clients=

Note the “4” in the seventh field of the first line (smtp-amavis)… this tells postfix how many servers to start up. This number must match the number of servers that you startup in amavisd. If you startup fewer in amavisd, you will loose mail. Do not startup more in either file.
Note also, in the main.cf, we have added the relay_recipients file. We are going to have to generate that file from reading LDAP (from Active Directory). This is a kludge, but I will put it here anyway…

 

vi /etc/postfix/getadsmtp.sh

add the following to that file, replacing “password” after -w with spam_buckets password:
#!/bin/bash
# Ok, this will look bod. Consider thie a kludge, but it gets the job
# done. This would have been better in perl or something, but I did
# this quick with built in tools. The main problem is it is ugly and lacks
# security like a good perl + secure ldap would have.
#
# The script reads in all vars from the ldapsearch, strips out the mail
# lines, then strips the mail from the front and the / as there is an
# email four miles long with a few “/”s in it, and then puts that in the
# /etc/postfix/relay_recipients file.
#
# Of course, after that we need to postmap the file.
# Lets assume the exchange IP is 1.2.3.5
/usr/bin/ldapsearch -x -h 1.2.3.5 -b “dc=yourdomain,dc=com” -D
“spam_bucket@yourdomain.com” -w password |grep mail: |sed ‘s/^[mail:\]*//’ |grep -v / | sed ‘s/\(.*\)/\L\1/’ |sed ‘s/$/ OK/’ > /etc/postfix/relay_temp
#
/usr/bin/ldapsearch -x -h 1.2.3.5 -b “dc=yourdomain,dc=com” -D
“spam_bucket@yourdomain.com” -w password |grep proxyAddresse |sed ‘s/^[proxyAddressess: ]*//’ | sed ‘s/^SMTP://’ |sed ‘s/mtp://’ |grep -v 400 |grep -v / | sed ‘s/\(.*\)/\L\1/’ |sed ‘s/$/ OK/’ >> /etc/postfix/relay_temp
#
#
# Now lets get rid of the dupes…
/usr/bin/sort /etc/postfix/relay_temp | /usr/bin/awk ‘!x[$0]++’ > /etc/postfix/relay_recipients
#
# remove the temp file I created.
rm /etc/postfix/relay_temp
#
# postmap it so postfix can read it.
#
/usr/sbin/postmap hash:/etc/postfix/relay_recipients

Finally, we need to setup a cron job to run that file say every hour from 8am till 5pm:

 

vi /etc/cron.d/getadsmtp

In that, put the following:
MAILTO=””
0 8-17 * * * root /etc/postfix/getadsmtp.sh

At this point, we should be able to (re)start the services and get going.

 

/etc/init.d/postfix reload
/etc/init.d/spamd start
/etc/init.d/clamd start
/etc/init.d/amavisd start

You will want to tail the log file (/var/log/mail.log) and make sure it is working. Most problems should be reported there, as postfix and amavisd will be logging to the /var/log/mail.log file. As you should have installed postfix with no spam checks as per this document, the only real problem you may have with postfix is the additional checks or the relay_recipients file. Make sure there are users in that file in the format of:

 

user@domain.com OK

and nothing else. Still not working? Did you remember to create /etc/postfix/pcre-header.cf and /etc/postfix/pcre-body.cf? telnet to port 25 and make sure that postfix at least answers you.

Black and White Listing by hand

First of all, we will point out that you really do not want to do hard black or white listing with this system. You can argue this out in your head until the cows come home, but you do NOT want to do hard back and white listing (e.g. by using /etc/postfix/pcre-*). The real reason you do not want to do this is that if anyone were to actually send you a legit email from thatdamndomainthatkeepssendingspam.com, the filter will have no way to even guess that it could be legit. Consider, the purpose of this system is to block unwanted email, but at the same time the main goal is to let legit email pass. If we can never scan the email to see if it is good or bad, we have zero chance of accepting a legit mail from that domain. That way leads the path to destruction. We can not use that mindset of thinking. You are innocent until proven guilty, not guilty 100% of the time with no chance for innocence.

Ok, so then, how do we do it? Simply! All we need to do is adjust the score of that domain or user at that domain in amavisd, and then let spamassassin add that to the cumulative score that it will put on it after all the filters have been applied. We do this within the variable block @score_sender_maps in amavisd

 

vi /etc/amavisd.conf

search for

 

@score_sender_maps

read the docs there in the config file. Note that a negative score is white listing the user/domain and a positive score is black listing a user/domain. You do not want to go crazy here and say “aha, I can just put a score of 450 on this domain and I will not get that mail”, as that still defeats the purpose of the whole system as we have outlined above. How about this… we drop spam with a score of 4.0. Put the score (to blacklist) at say 3 for a domain and 5 for a particular user. This way, if that user ever actually sends you a legit mail, it has a fighting chance, and if the domain has another user that sends you mail, that domain has a fighting chance.  So, for example, if you have a user fred@example.net who is killing you with spam you would perhaps do the following near the bottom of the @score_sender_maps section:

 

‘fred@example.net’ => 5.0,
‘.example.net’ => 3.0,

To get even more directed, if you are receiving spam from a user, again say fred@exmaple.net, look at the original message and look at the spam headers. What was the spamassassin final score on that message (x-spam-score)? If the score is -1.2, then make the positive scoring for that user in the @score_sender_maps section to be 5.3. In the future, if the system really thinks it is a legit mail from that user, then it will get through, but even better, if you put the user at 5 and the domain at 3, it will not pass as 7-1.2 is 5.8, and the filter will stop it.

You would then save that file and then restart amavisd and postfix, like so:

 

/etc/init.d/amavisd restart
/etc/init.d/postfix reload

Finishing up and reporting

Well, if you finally have it up and working now, you may want to just quit, but wait! There’s more!! Lets just do a few more really small things to get this box to mail us with some basic reporting, and make sure we have the logrotation setup, as the mail and spam logs can get quite large.

logrotate

You should have installed logrotate at the start of this document with an emerge logrotate. If not, just do a:

 

emerge logrotate

Now then, lets make sure that we have the logrotation setup. In /etc/logrotate.conf we would like to have:

 

weekly
# we will keep 6 months of logs here…
rotate 24
create
compress

The other values are ok at the default. The real meat of the matter is /var/log/mail.log. Recall that we are logging that with syslog-ng…

 

vi /etc/logrotate.d/syslog-ng

Add the following

 

/var/log/mail.log {
compress
rotate 96
weekly
missingok
sharedscripts
# run awstats before we rotate the maillog…
prerotate
/etc/cron.d/awstats
endscript
postrotate
/etc/init.d/syslog-ng reload > /dev/null 2>&1 || true
endscript
}

We will install awstats and have it give us some statistics on mail that the boss can see, as it is much nicer than the text file we will get from logwatch.
Note also, I am keeping two years of logs from the mail log. If you are short on space (although they are text and compressed) you may want to consider if you need this or not. Most companies require you to keep logs for a long time when it comes to mail.

That should take care of logrotate. Gentoo should have installed some sensible defaults in the /etc/logrotate.d/ direcotry.

logwatch

Again, if you read the start of this page you will have installed logwatch, but if not you can always do a:

 

emerge logwatch

Now, we will do some basic config, and then add the scripts to do the amavis watching for us. So, first it has to email someone:

 

vi /etc/logwatch/logwatch.conf

Add the following (which will override the defaults in /usr/share/logwatch/default.conf/logwatch.conf)

 

MailTo=”somepoorsap@yourdomain.com”
Detail=Med

Ok, now we want to put some logwatch scripts in place so go and download the amavis-logwatch-1.51.02.tgz. Then unpack it on the nospam box…

 

tar -xvzf amavis-logwatch-version.tgz
cd amavis-logwatch-version
make install-logwatch

This will install the script in /etc/logwatch/scripts/services/amavis and the configuration in /etc/logwatch/conf/services/amavis.conf. You should not technically have to do anything to get these working. It should work when logwatch goes off.

awstats

We will put up a small webpage that gives some stats on the mail server other than logwatch (which is a huge text file that gets emailed to one or more users). This requries apache and php, which we will install first.

 

USE=”php apache2″ emerge apache

After that is installed, you will need to setup the default hosts for the box. Lets just do that initially even though we do not have awstats installed yet.

First, do the very basic setup in apache.

 

vi /etc/apache2/httpd.conf

Just change the following for now:

 

ServerName nospam.yourdomain.com

The majority of the configs for apache2 are now in the vhosts and modules directories.

 

vi /etc/apache2/vhost.d/default_vhost.include

You can have it just include the following:

 

ServerAdmin root@localhost
DocumentRoot “/var/www/localhost/htdocs”
<IfModule alias_module>
Alias /awstats/classes “/var/www/localhost/htdocs/awstats/wwwroot/classes”
Alias /awstats/css “/var/www/localhost/htdocs/awstats/wwwroot/css”
Alias /awstats/icon “/var/www/localhost/htdocs/awstats/wwwroot/icon”
ScriptAlias /awstats/ “/var/www/localhost/htdocs/awstats/wwwroot/cgi-bin/”
ScriptAlias /awstats “/var/www/localhost/htdocs/awstats/wwwroot/cgi-bin/awstats.pl”
</IfModule>
<Directory “/var/www/localhost/cgi-bin”>
AllowOverride None
Options None
Order allow,deny
Allow from all
</Directory>

<Directory “/var/www/localhost/htdocs/awstats/wwwroot”>
AllowOverride None
Options ExecCGI -Multiviews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
</Directory>

Now to install awstats.

 

emerge awstats
cd /etc/awstats
cp awstats.model.conf awstats.yourdomain.conf
vi awstats.yourdomain.conf

We need to change a few vars in that file…

 

LogFile=”/usr/local/bin/prepflog.pl < /var/log/mail.log |perl /usr/bin/awstats_maillogconvert.pl standard |”
LogType=M
LogFormat=”%time2 %email %email_r %host %host_r %method %url %code %bytesd”
SiteDomain=”yourdomain.com”
HostAliases=”localhost 127.0.0.1 REGEX[yourdomain\.com$]”
DNSLookup=1
DirData=”/var/www/localhost/htdocs/awstats/wwwroot/”
LevelForBrowsersDetection=0 # 0 disables Browsers detection.
LevelForOSDetection=0 # 0 disables OS detection.
LevelForRefererAnalyze=0 # 0 disables Origin detection.
LevelForRobotsDetection=0 # 0 disables Robots detection.
LevelForSearchEnginesDetection=0 # 0 disables Search engines detection.
LevelForKeywordsDetection=0 # 0 disables Keyphrases/Keywords detection.
LevelForFileTypesDetection=0 # 0 disables File types detection.
ShowRobotsStats=0
ShowEMailSenders=HBML
ShowEMailReceivers=HBML
ShowPagesStats=0
ShowFileTypesStats=0
ShowFileSizesStats=1
ShowDownloadsStats=0
ShowOSStats=0
ShowBrowsersStats=0
ShowOriginStats=0
ShowKeyphrasesStats=0
ShowKeywordsStats=0
ShowMiscStats=0
ShowHTTPErrorsStats=0
ShowSMTPErrorsStats=1

 

Now…

 

mkdir -p /var/www/localhost/htdocs/awstats/wwwroot/cgi-bin/
cp /usr/share/awstats/wwwroot/cgi-bin/awstats.pl /var/www/localhost/htdocs/awstats/wwwroot/cgi-bin/

Finally…

 

vi /etc/cron.d/awstats

and add:

 

MAILTO=””
# Run the awstats twice a day for mail statistics.
0 7,16 * * * root /usr/bin/awstats.pl -config=yourdomain -update

You should run the command by hand at this time to populate the www directory…

 

/usr/bin/awstats.pl -config=yourdomain -update

Doesn’t work? Don’t forget to install /usr/local/bin/prepflog.pl from sourceforge.

At this point you should probably start apache and see if you have the basic setup correct…

/etc/init.d/apache restart

Goto <serverIP>/awstats/awstats.pl?config=yourdomain

If you have a permissions problem or some other error, check the apache log files and see if you can spot the error.

False Positives and Checking the logs

There are times when someone will send an email, and it will get stopped but it should not (as of this writing, the spam filter I installed, which filters thousands of mails a day, has stopped perhaps four false positives, but in the end, it was a sender error that caused the filter to trip. This filter is highly accurate after a short training). There are a few simple ways to find the mail and get it back.

First off, all mail will be logged to /var/log/mail.log,you can go ahead and use all of your grep-foo powers to search that and find the non-offending email. I would suggesting putting a |grep quarantine on the end of your searching. In that you will see that mail is actually being saved to your quarantine directory. (in my case, you can have it forward to exchange as per this document, or you can have amavisd-new quarantine it). Since you now know its name (in the case of local quarantine, amavisd will give you the name of the email in the grep output, for example: spam-TckJCqCQSD.gz Then you would simply use the find command to find that file (on one of my servers that is not forwarding to exchange, the quarantine dir as per the /etc/amavisd.conf file is /var/amavis/quarantine). From there you can actually use some tool to send it on to your local mailer. Use a file that was submitted to a zimbra forum called send_quarantine.pl by Jay McDonald of ThinkTek Solutions. I have a copy right here: sendquarantine.

Acknowledgements and Files

I would like to note a few FAQs/Pages on the ineternet that made this
possible.

The Debian Anti-Spam Anti-Virus Gateway Email Server has a ton of information, although a lot of it is deb specific.

The now practically defunct Gentoo wiki page on it… don’t use this.

The Gentoo mailfiltering gateway guide.

And tons of other online resources. Note that the pages above are nice references, but do not use them blindly.

This entry was posted in Servers, Software. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *