Archive for the 'Linux' Category

Talking TLS to SMTP

Tuesday, December 2nd, 2008

I recently wanted to test TLS with SMTP. I followed instructions on and got it to work in less than 30s ;-)

In short:

perl -MMIME::Base64 -e 'print encode_base64("\000user\000password")'
openssl s_client -starttls smtp -crlf -connect <ip>:<port>
auth <auth_command>
mail from:<tadek@pietraszek.org>
rcpt to:<tadek@pietraszek.org>
data
...
.

Interestingly, when I tried typing `RCPT` s_client would interpret it as “renegotiate”, which confused me a bit, but you can inhibit it with `–quiet` or type it in lowercase like I did ;-)

T.

xdu: analyzing disk-space usage

Thursday, November 27th, 2008

I recently started getting nagmails about running out of quota on my home directory. Being a very messy user, I had no idea where the space went. I tried playing with `du` manually, but it’s much easier with `xdu`:

sudo apt-get install xdu
du > /tmp/blah
xdu -n /tmp/blah

Installing SVN with Apache2 support on my server.

Thursday, March 6th, 2008

I just enabled SVN access on my server through the web interface. It was quite easy and, since we have a Postgres DB authentication, there’s no need to edit inconvenient password files ;-)

All I had to do was to enable the SVN module: link `dav_svn.{load|conf}` in `mods-enabled` and add the following line to the SSL-ed vhost.


SVNPath /var/lib/svn/foo
Dav svn

Require user user@example.com user2@example.com

Since the SSL-ed vhost already requires authentication, I didn’t have to change anything. I also had to create an SVN repository `svnadmin create –fs-type fsfs /var/lib/svn/foo` and change the permissions to `www-data`.

The checkout command is:

svn –username user@example.com –password my_secret_password co https://my.example.com/svn/foo

SVN caches the username and password, so any further operations are done without prompting you for it. If you don’t like it, you can disable it with `–no-auth-cache`.

Finally, one annoying thing. Initially, I would just try to connect without –username and SVN would first try my Unix user name and then ask for it. Unfortunately, some (but not all) users I tried in this way would get a mysterious:

svn: PROPFIND request failed on ‘/svn/foo’
svn: PROPFIND of ‘/svn/foo’: authorization failed (https://example.com)

WTH?

Apache2.0 -> Apache 2.2 upgrade issues

Tuesday, March 4th, 2008

While upgrading apache 2.0 -> 2.2 I found that two configuration options have changed:

* `AuthAutoritative` -> `AuthBasicAuthoritative` (needs to be set to Off for mod_auth_pgsql to work).
* `AuthDigestFile` -> `AuthUserFile`

Plus there has been a log of changes in `apache2.conf`. I looked at the changes we made (keeping /etc/ in SVN comes in handy, in spite of occasional pain) and pasted them to the vanilla apache2.conf that came with 2.2.

WTH: eth0 got renamed to eth2 after an upgrade

Tuesday, March 4th, 2008

While upgrading a __remote server__ from sarge to etch including the new kernel, the server did not come up. After attaching a console (thanks Hetzner!) I found out that the network interface got mysteriously renamed to eth2!

After snooping around a bit, I found out that the culprit was udev, more specifically `/etc/udev/rules.d/z25_persistent-net.rules` which says:

 # This file was automatically generated by the /lib/udev/write_net_rules
 # program, probably run by the persistent-net-generator.rules rules file.
 #
 # You can modify it, as long as you keep each rule on a single line.
 # MAC addresses must be written in lowercase.
 # PCI device 0x1106:0x3065 (via-rhine)
 SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{address}=="00:0c:76:af:2f:9d", NAME="eth0"

It also contained two entries for bogus eth0 and eth1 (usb dongle got identified as a network card?). After removing the and relabeling interfaces everything is back to normal now.

Patching debian packages

Tuesday, March 4th, 2008

While upgrading my server, I had to create a new version of Cyrus-SASL packages with the [crypt patch](http://frost.ath.cx/software/cyrus-sasl-patches/). This turned out to be more difficult as I thought as just patching the raw source would conflict with debian patches applied after that and the package building would fail. Fortunately, I learned about (dpatch)[http://www.tuxmaniac.com/blog/2008/01/25/dpatch-just-superb-a-short-how-to/], which is a standard mechanism for patching stuff in Debian.

Once I figured out how to do it, it tunred out to be very simple:

1. Unpack the original orig.tar.gz
2. Apply the Debian diff (this would create a debian subdirectory)
3. Edit changelog file to add a new release (so that we can keep track of it).
4. Run `dpatch-edit-patch fixing_foo` to create a patch. This will create a shell with mounted source package and any changes you make there will be included in the diff.
5. __Exit the shell without changing anything__. You will change things after making sure that the patch is applied in the correct order.
6. Edit `00list` to set the patch ordering.
7. Run `dpatch-edit-patch` again now, with all the previous patches applied.
8. Make the necessary changes.
9. Build the target package with `debuild-pbuilder` (this will download all the dependencies too).
10. Install the pakcages with `dpkg -i` (I could also create my private apt-repository for this. I wrote about in in a blog entry about pbuilder).
11. Pin the packages so that they will not get upgraded `echo “package hold” | dpkg –set-selection`

sed and awk – my two old friends

Sunday, September 30th, 2007

Writing some shell scripts I needed to do some a little fancier variable substitution than the standard shell offers. The heavyweight solution would be to write a perl one-liner, but this is, well…, heavyweight? ;-)

Here’s a couple of patterns I used:

  • --parameter=$(sed -re 's/ /,/g' -e 's/(^|,)/\1file:/g' <<<$INPUT) - replaces spaces with commas and prepends file to every file.
  • --parameter=$(awk '{split($0, a, /@/); printf "%s-?????-of-%05d", a[1], a[2]} <<<$INPUT)'
  • - replaces file@5 with file-?????-of-00005

  • --parameter=$(awk '{sub(/.*:/, ""); print $0}' <<<$INPUT) - removes everything before the colon.

Listing socket/network connection owners on OSX

Wednesday, July 4th, 2007

While playing with OSX I was wondering how to find out all the networks connections a particular process owns. On Linux I’d use `netstat -p` for this, which does not work on OSX.

It turns out that the solution is quite simple – `lsof -i` does the job and works on both Linux and OSX. Two other useful commands:

lsof -ai -p PID # all connections/sockets owned by PID
lsof -i:PORT # lists all connections/sockets with a particular PORT.

Link: Surviving traffic storms with Wordpress

Wednesday, May 2nd, 2007

[Interesting link on surviving traffic storms with Wordpress](http://turbochargedcms.com/2007/04/weathering-heavy-traffic-with-wordpress-and-turbocharged/): not that I currently need it, but maybe in the future… ;)

In a nutshell:

1. fine tuning of Apache (adjusting #processes, keep alives and ListenBacklog to values that match your machine’s constraints).
2. fine tuning of MySQL query caching
3. installing [WP-Cache plugin](http://mnm.uib.es/gallir/wp-cache-2/) + adaptive switching on of WP-cache plugin (only in heavy-load condition)
5. disabling some plugins (the ones that take up a lot of resources)
6. enabling Squid caching for static content.

Two friends: GeoWebStats and GeoBroStats – visualizing Apache and Bro logs with Google Maps

Tuesday, January 2nd, 2007

One of my pet (a.k.a. [procrastination](http://www.thefreedictionary.com/procrastination)) projects has been to visualize my server logs using [Google Maps](http://maps.google.com). In fact, this has been my ‘procrastination hub’ giving me excuses to work on a variety of pet projects, including:

* playing with [Bro](http://bro-ids.org) and packaging Bro for Debian
* playing with Apache logs and importing them to the relational database
* playing with Bro logs and importing them to the relational database
* learning Python and Javascript
* playing with Google Maps
* writing a web application to visualize the collected logs on Google maps
* creating a webpage documenting all the above.

As with procrastination projects, they are by definition never complete. I do have something working now, and you can see it in action (works best in a [decent browser](http://www.mozilla.com/en-US/firefox/), but should show something in IE as well).

### GeoWebStats
Visualizing Apache logs on a webpage. Here are three links (it might take a while to load them for the first time, so please be patient):

* [Blog Visitors - last 7 days](http://plum.ibao.net/webstats/?vhost=tadek.pietraszek.org&regexpmatch=%2Fblog&regexpnomatch=%2Fblog%2Fwp-%7C%2Fblog%2Ffeed&cutoff=10&start=-7&cc_locked=vhost%2Cregexpmatch%2Cregexpnomatch%2Cstart%2Cend%2Cstriplevel&cc_hash=6049f98622a55ac5cb3b87dd535820d0&submit=1)
* [Gallery Visitors - last 7 days](http://plum.ibao.net/webstats/?vhost=gallery.ibao.net&regexpnomatch=%5E%2Fd%2F%5B0-9%5D%2B%7C%5E%2Fc%2F%7C%5E%2Findex.php%7C%2Ffavicon.ico&cutoff=5&start=-7&cc_locked=vhost%2Cregexpmatch%2Cregexpnomatch%2Cstart%2Cend%2Ccutoff%2Cstriplevel&cc_hash=e5366f30cae72af7a33f166394e96bc6&submit=1)
* [Homepage Visitors - last 7 days](http://plum.ibao.net/webstats/?vhost=tadek.pietraszek.org&regexpnomatch=%2Fblog%7C%2Ffavicon.ico%7C%5E%2Fimages&striplevel=2&start=-7&cc_locked=vhost%2Cregexpmatch%2Cregexpnomatch%2Cstart%2Cend%2Ccutoff%2Cstriplevel&cc_hash=992cd263f64643edf909c6eea6a9624e&submit=1)

The script is quite customizable (for example you can specify the regular expressions you want to filter on, group stuff) but for security resons those demo links are locked.

### GeoBroStats
Simiarly to GeoWebStats, GeoBroStats visualizes raw TCP/UDP conections based on Bro conection summaries (this might also take a while to load):

* [Today's SMTP connections that transfered 10kB+ (mostly spammers) ](http://plum.ibao.net/brostats/?service=smtp&cutoff=10&start=0&cc_locked=service%2Cport%2Cstate%2Ccutoff%2Cstart%2Cend&cc_hash=3f41b93fc42c40fe3d60920af7e207f5&submit=1)

The script is also quite customizable, but for security resons those demo links are locked.

Let me know what you think about it. I know that the user interface is very crude and needs some work. I have also almost finished GeoWebStat’s website, but knowing me, it will take a while ;-)