After having thought about it for at least half a year and having researched the topic thoroughly for a good weekend, I finally got to implementing anti-spam and anti-virus measures on my server. It turned out to be more complex than I had initially thought (as always), but it seems to be working now.

To give a bit more background, I am running Postfix with Courier-IMAP and PostgreSQL as database backend. E-mail accounts reside in a virtual folder and have no corresponding Unix accounts.

I decided to use maildrop (I discussed Postfix and procmail issues here) and followed this tutorial, with the following exceptions:

• I had to backport a few packages to sarge (wrote about it here).
• I found out by trial and error that two packages courier-maildrop and maildrop have the same program working differently (essentially, maildrop from the maildrop package works, the other one doesn’t!)

• I added a custom clamAV source to my sources.list files:

  deb http://ftp2.de.debian.org/debian-volatile sarge/volatile main
  

• I wrote my own /etc/maildroprc

The idea is to have e-mail moved automatically to a folder containing spam if (and only if) such a folder exists. What I came up with is the following

This is the folder into which spam messages are delivered

SPAMFOLDER="$DEFAULT/.caughtspam/"

run the message through SpamAssassin

exception {
    xfilter "/usr/bin/spamc -u $LOGNAME"
}

if the message is marked as spam AND SPAMFOLDER exists - deliver there

I have no idea how to check it other than executing [ -d ] in a shell

SPAMFOLDEROK=[ -d $SPAMFOLDER ]; echo $?
if ( /^X-Spam-Flag:.*YES/ && $SPAMFOLDEROK == 0 )
{
  exception {
          to $SPAMFOLDER
  }
}

What still needs to be done is:

• automatic training on users’ emails (to enable per-user training)
• inclusion of user-specific rules (still need to thnik about it a bit as it has serious security implications).

Useful links:

• http://www.courier-mta.org/maildrop/?maildropfilter.html – Maildrop filtering language
• http://spamassassin.apache.org/gtube/ SpamAssasin GTUBE test
• http://www.webmail.us/testvirus – Sending EICAR messages to test anti-virus

Simple package building

1. First things first, get puilder:

   apt-get install
   

2. Then build an initial image:

   pbuilder create --distribution sarge --mirror http://ftp.ch.debian.org/debian
   

3. Get package sources (.dsc, .diff.gz and .orig.tar.gz files)

4. Build the package:

   pbuilder build <package>.dsc
   

Complex package building (with dependencies)

While playing with packages, I had the following problem: packageA build a few packages, one of them packageA-dev. The next package I was trying to build package-B had a build dependency on packageA-dev. However, as packageA was not in sarge, pbuilder couldn’t satisfy the dependencies and failed.

Trying to solve this problem, I found the following workaround:

1. Create an apt repository with newly build packages

   dpkg-scanpackages . /dev/null | gzip > Packages.gz    (in /var/cache/pbuilder/result)
   touch Release
   

2. “Mount” the directory to be visible in the chrooted builder environment:

   --bindmounts "/var/cache/pbuilder"  (runtime)
     or
   BINDMOUNTS="/var/cache/puilder"   (config file)
   

3. Add the appropriate apt sources:

   --othermirror "deb file:/var/cache/pbuilder/result ./"  (runtime)
     or
   OTHERMIRROR="deb file:/var/cache/pbuilder/result ./" (config file)
   

BTW: If you’re changing the config file after you’ve build the image, you should update base.tgz file using the following command:

pbuilder update --override-config --distribution sarge

BTW: pbuilder manual can be useful.

For some time I’ve been running an ISP-grade e-mail hosting system on our server (it might be a bit of an overkill, I know), using Postfix, and Courier IMAP and Postgres as database backend. This is not the topic of this post, but followed this tutorial while setting it up and changed some MySQL-specific things to Postgres (BTE: postfix wiki article also discusses this topic).

The system works fine, but what’s been on my mind is how to enable server-side e-mail filters (rules, etc.). A simple example could be to deliver spam messages directly to “spam” folder, but other things could also be interesting (e.g. rule based filtering, autoreplying, folder sorting, etc.). All in all, what I was looking for is “procmail for virtual mailboxes”.

The most obvious option mailbox_command = /usr/bin/procmail in main.cf doesn’t work, because it only refers to local delivery (done using local, not virtual command). Looking into this matter a bit further, here are the possible soulutions I found on the web:

Change virtual delivery to local

In this case postfix should just work, however you need to make sure that it delivers mail to correct mailboxes (it’s not trivial). Also, if users have per-user procmail files, they can probably easily access each others mailboxes (essentially no file system-level permissions here).

This post and the followup discussion discuss this topic (in short, setting virtual_transport=local or virtual_transport=postfix is not such an easy solution).

There’s an interesting discussion here (check all the e-mails) and one post suggests virtual=local. Again, I’m not sure how well it works. Essentially, the problem is that procmail is not aware of users stored in the database.

Change virtual delivery to maildrop

Maildrop can connect to the database and check user account there (this option is supposedly not enabled in the debian package).

Links:

• Gentoo howto
• Virtual readme disusses how to setup Postfix to deliver some mail using maildrop.
• Postfix Maildrop Howto even more specific.
• Another howto on Debian

Use Courier’s MTA and procmail

This site briefly mentions how to enable it. Haven’t tried it.

To summarize, I think that the most reliable option is to use maildrop with database support. I will give it a try…

Being new to the Gentoo world, I had to quickly write a few ebuilds. This process is reasonably well documented in Ebuild howto and wiki, however, some issues are not very clear.

Here’s what wasn’t obvious to me:

1. Installing packages using ebuild and a full path to your ebuild doesn’t work (from the man-page: The implementation of emerge /path/to/ebuild is broken and so this syntax shouldn’t be used).
2. In this case, to install your package you should probably create PORTDIR_OVERLAY in /etc/make.conf (e.g., PORTDIR_OVERLAY="/usr/local/portage", see this wiki page).
3. Your package has to be in one of the predefined categories. If you come up with something new, Portage will happily ignore it.
4. Before emerging, you should recompute the digest of your package using emerge /path/to/ebuild digest command.
5. It seems that from within the package, it is difficult to distinguish whether the package is being installed for the first time vs. updated and updated vs. removed. For this, I wrote a hack, which I documented in my blog.

I’ve recently been writing Gentoo ebuilds (interestingly enough, not being a Gentoo user myself) and faced the follwing problem: “How to distinguish between a package upgrade and a package removal from within the package?”

Gentoo has only a standard set of “hooks” that get called during package installation (related to install and uninstall, but not to upgade). The upgrade stage P1 -> P2 is performed as follows:

• P2.configure/compile/install
• P1.prerm()
• remove files from P1 that are not in P2
• P1.postrm()

This works fine for simple file replacement tasks, which are handled automatically. However, there are situations where you want something more sophisticated:

• preinst/postinst: configure the system to use the package (services, system variables, etc).
• prerm/postrm: remove the remnants of system configuration when the package is deleted.

For this to work, package P1 needs to know if it’s being upgraded and removed (no system conf. needed). Simiarly, P2 needs to know if P1 exists and the system has already been configured (so that the configuration is not removed on update). To make things even more difficult, a package can be upgraded, downgraded or reinstalled (for reinstall rm not called). After a bit of experimenting and scratching my head, here’s what I came up with:

#detecting upgrades while installing
pkg_preinst () {
    has_version "<${PF}" && UPGRADE=1 || UPGRADE=0
    has_version ">${PF}" && DOWNGRADE=1 || DOWNGRADE=0
    has_version "=${PF}" && REINSTALL=1 || REINSTALL=0

   #now in the preinst or postinst code we can 
   #perform different actions depending on the variables set
}

#detecting upgrades while removing
pkg_prerm () {
    has_version ">${PF}" && UPGRADE=1 || UPGRADE=0
    has_version "<${PF}" && DOWNGRADE=1 || DOWNGRADE=0  
    [ -n -${A} ] && REINSTALL=1 || REINSTALL=0

  #perform different actions depending on the variables set 
  #(either here or in posrm - the variables have to be be set right here
}

I wonder whether there’s abetter solution to this problem?

I recently had to upgrade my PostgreSQL 7.4 to 8.0 in Debian and learned that there is a cool way of doing this. This procedure is described in /usr/share/doc/postgresql-common/architecture.html.

In a nutshell:

1. Install the new version in packages: postgresql-*-8.0. Mote that all the pg_* tools still point to the old versions.
2. Make sure that both servers are up and running.
3. Run: “pg_upgradecluster 7.4 main” to upgrade the old database to the new one.
4. Now all the pg_* tools point to the new versions, also the new server will listen on the standard port. Check if everything works fine.
5. Run “pg_dropcluster 7.4 main” to delete the old database.

Cool. I wonder how Gentoo does this