Tadek’s Blog

One of my pet (a.k.a. procrastination) projects has been to visualize my server logs using Google Maps. In fact, this has been my ‘procrastination hub’ giving me excuses to work on a variety of pet projects, including:

• playing with Bro and packaging Bro for Debian
• playing with Apache logs and importing them to the relational database
• playing with Bro logs and importing them to the relational database
• learning Python and Javascript
• playing with Google Maps
• writing a web application to visualize the collected logs on Google maps
• creating a webpage documenting all the above.

As with procrastination projects, they are by definition never complete. I do have something working now, and you can see it in action (works best in a decent browser, but should show something in IE as well).

GeoWebStats

Visualizing Apache logs on a webpage. Here are three links (it might take a while to load them for the first time, so please be patient):

• Blog Visitors – last 7 days
• Gallery Visitors – last 7 days
• Homepage Visitors – last 7 days

The script is quite customizable (for example you can specify the regular expressions you want to filter on, group stuff) but for security resons those demo links are locked.

GeoBroStats

Simiarly to GeoWebStats, GeoBroStats visualizes raw TCP/UDP conections based on Bro conection summaries (this might also take a while to load):

• Today’s SMTP connections that transfered 10kB+ (mostly spammers)

The script is also quite customizable, but for security resons those demo links are locked.

Let me know what you think about it. I know that the user interface is very crude and needs some work. I have also almost finished GeoWebStat’s website, but knowing me, it will take a while

I recently had to write some Polish text on my MacBook Pro and discovered that the Polish keyboard is messed up. In fact, coming from a PC world I’ve always thought Mac’s keyboards are messed up (e.g., lack of Home/End PageUp/PageDown, which can be simulated by some weird and application-dependent two/three key combination, an almost completely useless Enter/Rename key, an annoying Eject key, which pressed accidentally generates an eject sound regardless whether you have something in your drive or not), but this time I got annoyed.

To give a bit of background, in Poland, we use nine additional letters, namely ęóąśłżźćń (and their uppercase counterparts) and historically typewriter’s keyboard had them allocated at the right side (where brackets and quotes are). Now, unless you’re a typewriter, this is not very useful (especially if you need the braces and quotes more often) and we have two Polish keyboard mappings: a typewriter’s keyboard and a programmer’s keyboard (with Polish letters generated with an Alt+<Latin letter>). As we have two z-derivatives: żź one of them is Alt+z (the more common ż) and the other is Alt+x (the less common ź).

Playing with my Mac I discovered that żź are swapped. I am not sure if there’s any rationale for it (apparently it was ok in OS9 and only changed in OSX), maybe it’s easier to press Alt+x (which gives a more commonly used character), in particular that on a PC it’s a right Alt, not the left one (in fact, I was trying to get it to be more ergonimic, I would remap the right Enter to Alt), but I found it confusing. To get a feeling what it’s like, imagine what if Apple replaced a Control key with Enter or PageUp with an eject button. Whops… they already did it. Imagine something else then

Doing a bit of research I found discovered I am not the only one annoyed with it. Somebody made a correct programmer’s keyboard and which can be downloaded from here. There are two versions: one replacing a system file and one installing a local keyboard for a user. I took the latter approach and it works great!

I’ve been using bro for quite a while on my server and consider is a great IDS. Actually, I’ve been using it mostly as a network analysis tool (connection summaries, tracking HTTP connections, analyzing headers, etc.), rather than an IDS itself, but I still think it’s great.

What has been bothering me most this time is that my cleanly-installed server with a proper package manager (I’m running Debian and I am very happy about it, regardless what some friends of mine say) is running a service installed in my home directory in a screen. In fact, as the server’s uptime is on average half a year, it’s not such a big problem, but it really bothered me

Almost a half a year ago, I started Bro’s ‘Debianization’ process, as one of my many procrastination projects (a.k.a. pet project), but I haven’t been active (maybe now that I defended my thesis I don’t need to procrastinate so much? ). Now during the Christmas break I finally managed to (almost) finish it!

The whole job turned out to be more difficult than I’d thought, but it works now. Here’s a proof:

tadekp@plum:~$ apt-cache show bro
Package: bro
Version: 1.1d-1
Priority: optional
Section: net
Maintainer: Tadeusz Pietraszek <tadek@pietraszek.org>
Depends: libc6 (>= 2.3.2.ds1-21), libgcc1 (>= 1:3.4.1-3), libncurses5 (>= 5.4-1), libpcap0.7, libssl0.9.7, libstdc++5 (>= 1:3.3.4-1), c-shell
Architecture: i386
Filename: ./bro_1.1d-1_i386.deb
Size: 3061038
Installed-Size: 8916
MD5sum: 880901a64a7fc44766e4645f445799a6
Description: Network Intrusion Detection System (NIDS)
 Bro is an open-source, Unix-based Network Intrusion Detection System (NIDS)
 that passively monitors network traffic and looks for suspicious traffic.
 .
 Bro detects intrusions by comparing network traffic against a customizable
 set of rules describing events that are deemed troublesome. These rules
 might describe specific attacks (including those defined by signatures)
 or unusual activities (e.g., certain hosts connecting to certain services
 or patterns of failed connection attempts).
 .
 Bro uses a specialized policy language that allows a site to tailor Bro's
 operation, both as site policies evolve and as new attacks are discovered.
 If Bro detects something of interest, it can be instructed to either generate
 a log entry, alert the operator in real-time, execute an operating system
 command (e.g., to terminate a connection or block a malicious host
 on-the-fly). In addition, Bro's detailed log files can be particularly
 useful for forensics.


tadekp@plum:~$ 

tadekp@plum:~$ /etc/init.d/bro status
Bro is running (pid: 2859)
Autorestart: ON
Running since: Mon Jan  1 16:11:37 CET 2007
Bro Version: 1.1d
Active log suffix: plum.07-01-01_16.11.33
tadekp@plum:~$ 

The package is in alpha stage now and I still get a few lintian errors (for example, the man page is missing), but otherwise is ok (even including the init.d scripts and checkpointing). If you’re interested in trying it out, please let me know.

Python (at least up to version 2.4) doesn’t have a much needed ?: operator. Here’s how you can hack it yourself (I’m not sure if I would use it, except for very special situations, though).

# if-then-else function taking functions as arguments.
def ite(condition, true, false):
    if condition:
        return true()
    else:
        return false()

# Call it like this.
ite(condition, lambda:arg_true, lambda:arg_false);


# Just to check if it really works
def fun(arg):
    print "Evaluated "+str(arg)
    return arg

print ite(0, lambda:fun(1), lambda:fun(0));
print ite(1, lambda:fun(1), lambda:fun(0));

I don’t like programming in bash but it does make some things very simple. Unfortunately, it is not obvious how to do some simple things like iterating through an array

Assuming that we have an array ITEMS=( a b c d ), we can use a ${ITEMS[@]} construct to iterate through all the elements in a for loop:

ITEMS=( a b c d )
for ITEM in ${ITEMS[@]}; do
    echo $ITEM
done

BTW: I found this pattern here.

Sometimes I have a list of something in a file, one line per item and want to convert it to a comma(collon,…)-separated line (with no trailing separator of course) that can be used as a command-line parameter to some other tool. A perl oneliner comes in handy:

`perl -e '@_=<STDIN>; chomp(@_); print join(";",@_);' < data_file`

1. Plotting a matrix of small charts on one page:

   #setup a (close to) square matrix for plotting
   matrix_par <- function(numplots, ...) {
      par(mfrow = c(ceiling(numplots / floor(sqrt(numplots))), floor(sqrt(numplots)) ),...);
   }
   

2. Barplots with confidence intervals

   library(gplots);
   #plot a barplot with confidence intervals
   barplot_ci <- function (y, y_ci, ...) {
       barplot2(y, ci.l=y-y_ci, ci.u=y+y_ci, plot.ci=TRUE, ...);
   }
   

3. Filtering certain columns in a data frame:

   restrict = c("val1", "val2", "val3");
   x = read.table(...);
   if (is.vector(restrict)) 
       x = x[ x$V1 %in% restrict, ];
   

4. Cummulative series with sapply:

   sum_x = sapply(seq(x), function(i) { sum(x[1:i]); });
   

5. Processing multiple files in a directory and generating output files

   #execute for all input files
   input_files = list.files(".","\.ssv");
   for(input_file in input_files) {
      #convert the filename according to this regexp
      output_file = sub("\.ssv","\.new_extension",input_file);
   }