Tadek’s Blog

After almost 3.5 years of work I submitted my PhD thesis today. I feel relieved, although I still haven’t fully realized this yet. It will come… On the funny side, now after having (hopefully) done most of the work, I am currently not even at the first out of 11-point [Ablaufplan](http://www.informatik.uni-freiburg.de/dekpamt/promotion/allgemeines/promotion_ablauf.html) describing the quest of a poor PhD student. However, now that the machinery has been started, actionable items for me start at point 8 (“pass the defense”), which will take place some time in October

Finishing the blog entry, here’s a funny thing that happened to me on the train. I was carrying the a cardboard box, filled up with 5 copies of a nicely printed and bound dissertation. During the passport control (rarely happens, but today was my lucky day) a very bored customs officer got interested in what was in the box. The conversation followed like this.

Customs Officer: So, what is in this box?
Me: Papers.
C: How much are they worth?
M: Well… nothing? (I did not want to go into details of three-and-a-half years of pain or how much I was paid during that time)
C: Can I see?
M: Sure.
C: (opens the box takes one and starts browsing) Are they all the same?
M: Yes. Would you like to read?
C: (at this time I got a kind of look which tells you that you shouldn’t be joking with a customs official). So… were they printed in Switzerland?
M: Yes.
C: And you say they are worth “nothing”?
M: Exactly.
C: Why were they printed in Switzerland and are being brought here?
M: Because I work there?
C: Well…OK. Do you carry anything else that is also worth “nothing”?
M: No, I don’t think so.
(custom officer leaves)

I recently discovered a potential security problem with the __configuration__ of Linux-based masquerading firewalls, which (I must admit I fell prey of).

Suppose I a server with one outgoing IP address, which has two network cards (`eth0` – outgoing link and `eth1` – internal link). I want to set up masquerading of the internal network 192.168.0.0/24.

The way I would proceed is:

1. Enable packet routing:

net.ipv4.ip_forward = 1

2. Configure a masquearing rule:

iptables -t nat -A POSTROUTING -j MASQUERADE

3. Set up a firewall rules:

iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -i eth1 -j ACCEPT
iptables -A INPUT -m conntrack –ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp -m multiport –dports -j ACCEPT
iptables -P INPUT DROP

Does it work and is it secure? Yes… well… not really. Suppose the atacker is on the same network as I am and sets my host as his router. He will then be able to:

1. Connect to my internal hosts (which will be helpfully masqueraded).
2. Use my server to masquerade his connections (only on the allowed ports, but still).
3. If the server has another interface (e.g. running an IPsec tunnel to a restricted network, it’s getting __really scary__.

What can I do:

1. Control the FORWARD chain:

iptables -A FORWARD -m conntrack –ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i eth1 -j ACCEPT
iptables -P FORWARD DENY

2. Block the incoming connections in `iptables -A INPUT -p tcp -m multiport –dports -j ACCEPT` by specifying the destination IP address. However, this may be tricky, if the server uses a DHCP address (otherwise you wouldn’t be using MASQUERADE in the first place).

3. Change masquerade so that it has an interface specified: `iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE`. This only partially solves the problem, as the packets will still be forwarded to internal networks.

After having thought about it for at least half a year and having researched the topic thoroughly for a good weekend, I finally got to implementing anti-spam and anti-virus measures on my server. It turned out to be more complex than I had initially thought (as always), but it seems to be working now.

To give a bit more background, I am running Postfix with Courier-IMAP and PostgreSQL as database backend. E-mail accounts reside in a virtual folder and have no corresponding Unix accounts.

I decided to use maildrop (I discussed Postfix and procmail issues [here](http://tadek.pietraszek.org/blog/2006/02/05/postfix-virtual-mailboxes-and-procmail-filtering/)) and followed [this tutorial](http://www.xs4all.nl/~jaspersl/howto/), with the following exceptions:

* I had to backport a few packages to sarge (wrote about it [here](http://tadek.pietraszek.org/blog/2006/02/05/backporting-debian-packages-sid-sarge-experiences-with-pbuilder/)).
* I found out by trial and error that two packages `courier-maildrop` and `maildrop` have the same program working differently (essentially, maildrop from the maildrop package works, the other one doesn’t!)
* I added a custom clamAV source to my `sources.list` files:

deb http://ftp2.de.debian.org/debian-volatile sarge/volatile main
* I wrote my own `/etc/maildroprc`

The idea is to have e-mail moved automatically to a folder containing spam if (and only if) such a folder exists. What I came up with is the following

#This is the folder into which spam messages are delivered
SPAMFOLDER="$DEFAULT/.caughtspam/"

#run the message through SpamAssassin
exception {
    xfilter "/usr/bin/spamc -u $LOGNAME"
}

# if the message is marked as spam AND SPAMFOLDER exists - deliver there
#I have no idea how to check it other than executing [ -d ] in a shell
SPAMFOLDEROK=`[ -d $SPAMFOLDER ]; echo $?`
if ( /^X-Spam-Flag:.*YES/ && $SPAMFOLDEROK == 0 )
{
  exception {
          to $SPAMFOLDER
  }
}

What still needs to be done is:

* automatic training on users’ emails (to enable per-user training)
* inclusion of user-specific rules (still need to thnik about it a bit as it has serious security implications).

Useful links:

* – Maildrop filtering language
* SpamAssasin GTUBE test
* – Sending EICAR messages to test anti-virus

For some time I’ve been running an ISP-grade e-mail hosting system on our server (it might be a bit of an overkill, I know), using Postfix, and Courier IMAP and Postgres as database backend. This is not the topic of this post, but followed [this tutorial](http://workaround.org/articles/ispmail-sarge/) while setting it up and changed some MySQL-specific things to Postgres (BTE: [postfix wiki article](http://postfixwiki.org/index.php?title=Virtual_Users_and_Domains_with_Courier-IMAP_and_MySQL) also discusses this topic).

The system works fine, but what’s been on my mind is how to enable server-side e-mail filters (rules, etc.). A simple example could be to deliver spam messages directly to “spam” folder, but other things could also be interesting (e.g. rule based filtering, autoreplying, folder sorting, etc.). All in all, what I was looking for is “procmail for virtual mailboxes”.

The most obvious option `mailbox_command = /usr/bin/procmail` in `main.cf` doesn’t work, because it only refers to local delivery (done using `local`, not `virtual` command). Looking into this matter a bit further, here are the possible soulutions I found on the web:

### Change virtual delivery to `local` ###
In this case postfix should just work, however you need to make sure that it delivers mail to correct mailboxes (it’s not trivial). Also, if users have per-user procmail files, they can probably easily access each others mailboxes (essentially no file system-level permissions here).

[This post and the followup discussion](http://irbs.net/internet/postfix/0306/0240.html) discuss this topic (in short, setting `virtual_transport=local` or `virtual_transport=postfix` is not such an easy solution).

There’s an interesting discussion [here](http://www.colug.net/pipermail/colug432/2005-April/000583.html) (check all the e-mails) and [one post](http://www.colug.net/pipermail/colug432/2005-April/000576.html) suggests `virtual=local`. Again, I’m not sure how well it works. Essentially, the problem is that procmail is not aware of users stored in the database.

### Change virtual delivery to `maildrop` ###
Maildrop can connect to the database and check user account there (this option is supposedly [not enabled in the debian package](http://www.marlow.dk/site.php/tech/postfix)).

Links:

* [Gentoo howto](http://gentoo-wiki.com/HOWTO_Email_Virtual_Server_Maildrop_and_Spam_Assasin)
* [Virtual readme](http://www.postfix.org/VIRTUAL_README.html) disusses how to setup Postfix to deliver some mail using maildrop.
* [Postfix Maildrop Howto](http://www.postfix.org/MAILDROP_README.html) even more specific.
* [Another howto on Debian](http://www.xs4all.nl/~jaspersl/howto/)

### Use Courier’s MTA and `procmail` ###
[This site](http://jastram.de/story.php?id=177) briefly mentions how to enable it. Haven’t tried it.

—-

To summarize, I think that the most reliable option is to use maildrop with database support. I will give it a try…

Everybody has worries, which stay in our minds. For example, Currently, I am worrying about at least a dozen of things (maybe Poles are a worrying nation, or it is just me, don’t know) and, you must admit, it’s hard to stay productive being occupied with worrying.

Hence, during our lunch discussion yesterday we came up with the idea of _worriors_, to whom you can outsource your worries. To be completely honest this idea came from [My Outsourced Life](http://www.smartmoney.com/esquire/index.cfm?Story=20050909-outsource), I think the notion of “worriors” is ours

The idea is to have a website, where you can submit your worries, which a professional worrior (human or automated) can worry about. You can log in and check what they are currently worrying about or maybe even sign up for a comforting e-mail:

> Don’t worry.
> We are currently worrying about <> for you.
> Yours professional worriors”

This can also be a “worries outlet”: discussions, support groups and of course a great chance for advertising.

[TaskJuggler](http://www.taskjuggler.org/) is _an extremely powerful_ open-source project management software. Unlike other tools of this sort it is almost completely text-file driven and batch mode (yes, it has a nice GUI as well, but the operation is editing of the text project file). Also, the reports it generates are completely scriptable (actually there’s no other way of doing them). This is different from you’d expect, but maybe for us geeks it’s better to have something that works this way?

It is really powerful and looks actually usable (unlike some other open-source projects like [planner](http://planner.imention.org), which aspire to be but are IMO not even close).

Additional advantages I see are:
- easy to track changes if you keep the project file in CVS/SVN
- can easily script it, so that you can see your project status on the website?

Maybe I will use one day?

[Bettershopper](http://bettershopper.g-blog.net/) is a really cool plugin showing doing automatic price comparisons on amazon.com for a number of amazon.XX, alibris or other sites and converts them to your favorite currency. Saves a lot of time and money

I’ve been using version 1.1, which at some point stopped working Now I discovered that version 1.2 is [on its way](http://bettershopper.g-blog.net/news/) and you can download 1.2beta2, which works great for me (you can get it [here](http://bettershopper.g-blog.net/xpi/bettershopper-1.2b2.xpi)). BTW: it also works with FireFox 1.5.

Great idea and execution. Well done! Looking forward to trying 1.2 out!

I noticed that our [Gallery2](http://gallery.ibao.net) webpage has been ignored by Google Images. I researched this topic a bit and found the [list of search engine optimization tips for Gallery](http://gallery.menalto.com/node/36854), among which there was a line about avoiding “302 redirects”.

As it turns out Gallery uses a “302 redirect” to load “main.php“ on the initial load and it is also redirected from “index.php“, which is claimed to incur some anti-spam penalties for the Google engine. I don’t know whether this is true or not, but to be on the safe side, I decided to remove the reidirects.

Here’s what I did:

1. Move my site-specific “index.php“ somewhere else.
2. “ln -s main.php index.php“
3. Add the following line in the site-specific “config.php“:

define(‘GALLERY_MAIN_PHP’, ‘index.php’);

Everything seems to work now. We will see in a couple of weeks how well it works for the search engines

The new WP2.0 has been out for more than two weeks now and seeing there haven’t been any urgent patches/bug fixes/etc I assumed it is mature enough to be upgraded. Other than that, it seems like a perfect Sunday procrastination task

I basically followed the official WP upgrade instructions, but as my WP has been somewhat customized, I had to do some extra steps. Here’s what I did:

1. Backup the database, disable the plugins (SK2 and Markdown).
2. Move the old WP directory to “blog-old“.
3. Install a new “blog“ directory, fix permissions, copy the configuration files.
4. Upgrade the blog using “upgrade.php“ script.
5. Check that everything almost works

That was easy. Now the difficult part.

First, my theme has been somewhat customized (including [Diego's "linky"](http://www.zzamboni.org/brt/2005/02/19/20/) patch). However, even if WP2.0 still uses WP1.5 theme, some things have been changed (don’t know exactly what and why). To port my changes, I used the following procedure:

1. Create a diff against “vanilla” WP1.5.2.
2. Apply the patch on the new WP2.0.
3. Fix those little changes that didn’t merge.

Second, I once fixed a bug in WP1.5.2 concerning categories counts on the main page. Basically the problem is that the counts reflect only the “posted” posts and the private posts are not counted. However, when you are logged in and click on a given category you get to see all the posts, also including the private ones. Similarly, (sub)categories containing only private items do not show. The patch for WP1.5.2 was basically a one-liner, but it doesn’t work in WP2.0 Looking at the relevant file, it turned out that the new version uses precached counts stored in the database, instead of computing them dynamically. Hence, the patch doesn’t and cannot work. There also doesn’t seem to be an easy way of fixing it, so it has to stay the way it is.

Third, I had to upgrade SpamKarma2.0 to SpamKarma2.1 (I still doesn’t want to use Akismet, which is shipped by default in WP2.0).

Fourth, I upgraded Markdown plugin, to a new verison and disabled the new WYSIWYG editor (seems to conflict with Markdown).

Finally, I made a small fix to “linky” code, as it also conflicts with Markdown plugin. As the linky doce uses “wptexturize“ to get the content of the entry Markdown is not converted correctly. I changed “wp_texturixe“ to “apply_filters(‘the_content’,…)“ as the latter calls the plugin conversion function. However, in this case I had to remove extra “p“ tags, or more precisely, remove all “p“ tags alltogether. Finally the code looks like this: