Therefore, I decided to write my own plugin (yeah, there are already two out there, why not write my own? ) and also learn Gallery2 API. The idea is to display a google map at the bottom of each photo, showing exactly where the photo was taken. Yes, it photo-specific and there’s only one pointer on the map. I find it nonetheless very useful.

Here’s a sample output (you can also admire the beautiful scenery of Melchsee ). The plugin adds a new “block” in the template (therefore can be configured using a standard block management tool in Gallery2).

The position of the current photo is always in the middle (although you can move the map around, change the map type, zoom in and out etc.). The changes you make are stored as session cookies, and preserved between consecutive photo loads. Also, the whole panel can be hidden to speed up load and only shown on demand (show map|hide map).

Any comments? suggestions? ideas?

The plugin is currently in alpha stage, I will release it in a week or two (I want to create a webpage for it as well). In the meantime, if you’re interested in trying it out, drop me a line

BTW: I also found out that when iPhoto edits a photo, it converts Exif from Intel to Motorola (little endian -> big endian). There was a bug in exifer used in gallery, which corrpted the tags. The patch is only two lines long and can be found here (I also emailed the author):

--- gps.inc.orig        2006-08-31 10:25:27.000000000 +0200
+++ gps.inc     2006-08-31 10:36:37.000000000 +0200
@@ -116,13 +116,24 @@
                        $minutes = GPSRational(substr($data,16,16),$intel);
                        $hour = GPSRational(substr($data,32,16),$intel);



/* now we need a hack, since the whole data has been flipped in :103


the order here is sec:min:hour. However, in the motorla mode the data



has not been flipped and the order is h:m:s. This breaks compatibility



with Motorola exif. (Tadek) */

if($intel==1)
                    $data = $hour+$minutes/60+$seconds/3600;
else
                                $data = $seconds+$minutes/60+$hour/3600;
        } else if($tag=="0007") { //Time
                $seconds = GPSRational(substr($data,0,16),$intel);
                $minutes = GPSRational(substr($data,16,16),$intel);
                $hour = GPSRational(substr($data,32,16),$intel);

                /* I guess the same HACK as above. Tadek */

if ($intel==1)
                    $data = $hour.":".$minutes.":".$seconds;
else
$data = $seconds.":".$minutes.":".$hour;
            } else {
                    if($bottom!=0) $data=$top/$bottom;
                    else if($top==0) $data = 0;

Hence, during our lunch discussion yesterday we came up with the idea of worriors, to whom you can outsource your worries. To be completely honest this idea came from My Outsourced Life, I think the notion of “worriors” is ours

The idea is to have a website, where you can submit your worries, which a professional worrior (human or automated) can worry about. You can log in and check what they are currently worrying about or maybe even sign up for a comforting e-mail:

Don’t worry.
We are currently worrying about <> for you.
Yours professional worriors”

This can also be a “worries outlet”: discussions, support groups and of course a great chance for advertising.

BTW: I know this SQL is ugly like hell, but the framwork is really flexible and these queries are very easy to write. Here they go:

Last 3 days’s connections on known ports (hourly): ./conn2db2csv.pl -q "select date_trunc('hour',time), coalesce(sum(case when dport=80 or dport =443 then brecv else NULL end),0) as r80, coalesce(sum(case when dport=80 or dport = 443 then bsent else NULL end),0) as s80, coalesce(sum(case when dport=22 or dport=7322 then brecv else NULL end),0) as r22, coalesce(sum(case when dport=22 or dport=7322 then bsent else NULL end),0) as s22, coalesce(sum(case when dport=993 then brecv else NULL end),0) as r993, coalesce(sum(case when dport=993 then bsent else NULL end),0) as s993, coalesce(sum(case when dport=53 then brecv else NULL end),0) as r53, coalesce(sum(case when dport=53 then bsent else NULL end),0) as s53, coalesce(sum(case when dport=25 or dport = 587 then brecv else NULL end),0) as r25, coalesce(sum(case when dport=25 or dport = 587 then bsent else NULL end),0) as s25 from conn where dip='85.10.194.212' and time > now()-'3 days'::interval group by 1 having(sum(brecv) >0) order by 1" -d "dbi:Pg:dbname=bro;host=localhost;port=54321" -u bro -p <password> -s ' '

Last half year’s connection on known ports (daily) : ./conn2db2csv.pl -q "select date_trunc('day',time), coalesce(sum(case when dport=80 or dport =443 then brecv else NULL end),0) as r80, coalesce(sum(case when dport=80 or dport = 443 then bsent else NULL end),0) as s80, coalesce(sum(case when dport=22 or dport=7322 then brecv else NULL end),0) as r22, coalesce(sum(case when dport=22 or dport=7322 then bsent else NULL end),0) as s22, coalesce(sum(case when dport=993 then brecv else NULL end),0) as r993, coalesce(sum(case when dport=993 then bsent else NULL end),0) as s993, coalesce(sum(case when dport=53 then brecv else NULL end),0) as r53, coalesce(sum(case when dport=53 then bsent else NULL end),0) as s53, coalesce(sum(case when dport=25 or dport = 587 then brecv else NULL end),0) as r25, coalesce(sum(case when dport=25 or dport = 587 then bsent else NULL end),0) as s25 from conn where dip='85.10.194.212' and time > now()-'6 months'::interval group by 1 having(sum(brecv) >0) order by 1" -d "dbi:Pg:dbname=bro;host=localhost;port=54321" -u bro -p <password> -s ' '

The number of incoming connections and total bytes sent and received on all server ports (in a week): ./conn2db2csv.pl -q "select dport, count(*), coalesce(sum(bsent),0), coalesce(sum(brecv),0) from conn where dip='85.10.194.212' and time > now()-'1 week'::interval group by 1 having (sum(brecv)>0) order by 1" -d "dbi:Pg:dbname=bro;host=localhost;port=54321" -u bro -p <password> -s ' '

Similarly, might be interesting what the server contacted: ./conn2db2csv.pl -q "select dport, count(*), coalesce(sum(bsent),0), coalesce(sum(brecv),0) from conn where sip='85.10.194.212' and time > now()-'1 week'::interval group by 1 having (sum(brecv)>0) order by 1" -d "dbi:Pg:dbname=bro;host=localhost;port=54321" -u bro -p <password> -s ' '

Finally, scanning attempts: ./conn2db2csv.pl -q "select dport, count(*), coalesce(sum(bsent),0), coalesce(sum(brecv),0) from conn where dip='85.10.194.212' and time > now()-'1 month'::interval group by 1 having (sum(brecv)) is null order by 1" -d "dbi:Pg:dbname=bro;host=localhost;port=54321" -u bro -p <password> -s ' '

And top port scanners: ./conn2db2csv.pl -q "select sip, count(dport), count(*) from conn where dip='85.10.194.212' and time > now()-'1 month'::interval group by 1 having (sum(brecv)) is null order by 2 desc" -d "dbi:Pg:dbname=bro;host=localhost;port=54321" -u bro -p <password> -s ' '

Finally how to print such a thing using gnuplot: set timefmt "%Y-%m-%d %H:%M:%S" set logscale y plot '<file>' using 1:3 with l t 'P80', '' using 1:4 with l t 'P22', '' using 1:5 with l t 'P53', '' using 1:6 with l t 'P25'

I wrote a simple perl parser parsing Bro’s connection status and writing it to a relational database (in this case Postgres). The parser can also run in “query” mode, producing a comma/space separated data for easy visualization (using AfterGlow or even GnuPlot).

Here’s how I run my collection: sudo bin/bro -i eth0 -f "host 85.10.194.212" conn

And parsing: tail -f conn.log | ../conn2db2csv.pl -i -d "dbi:Pg:dbname=bro;host=localhost" -u bro -p <password>

And plotting: ./conn2db2csv.pl -q "select time::date,dport, sum(brecv) from conn where dip='85.10.194.212' group by 1,2 having(sum(brecv) >0) order by 1,2" -d "dbi:Pg:dbname=bro;host=localhost" -u bro -p <password>

This can be nicely used by gnuplot. For example to plot daily traffic on let’s say port 22: ./conn2db2csv.pl -q "select time::date, sum(brecv) from conn where dip='85.10.194.212' and dport = 22 group by 1 having(sum(brecv) >0) order by 1" -d "dbi:Pg:dbname=bro;host=localhost" -u bro -p <password> -s ' ' > testplot

gnuplot set timefmt "%Y-%m-%d %H:%M:%S" set xdata time plot 'testplot' using 1:3

GeoTagging The idea is simple: when you go hiking, sightseeing, travelling you put a GPS on top of your backpack. You go and take photos as usual. When you’re back home, you connect the GPS to the computer and download the saved track. You then run a program that correlates the time when the images were taken with your position from the track and encodes this information into EXIF. As a result the images are GeoTagged and their position can be displayed by a GeoTagging-aware software.

Downloading GPS After having a look at different options, I chose two candidates for downloading the track: gpstrans and GPS::Garmin.

Note that these both support only Garmin GPSs. I don’t know if there is any universal track transfer protocol (probably not). Everything that claims to be cross-GPS typically is limited to reading the current position via NMEA.

GPStrans works ok, although it’s a bit old and doesn’t work well with my eTrex. The problems are: waypoints are corrupted, track does not contain the information about new segments and the output format is strange (this can be fixed). Afterall, I must admit that unlike other tools it worked at the first try.

GPS::Garmin gave me much more trouble. First it tunred out that it relied on some undocumented behavior of Device::Serial and at the end it turned out that it was doing a non-blocking read and considered that it would always get data. While this might have been true a couple of years ago, since then computers have gotten faster and it stopped working. After a couple of hours I fixed it by adding: $PortObj->read_const_time(5000); $PortObj->read_char_time(5000); somewhere in GPS::Serial.

The second problem was that the code relied on particular product codes of Garmin deivces and changed its behavoir accordingly. The problem is that it’s not exactly how Garmin protocol was written (unless you want to encode the behavior of all their products).

Garmin Protocol It’s fairly simple, although implementing it correctly can take a good evening. The problem is that the number of commands is limited (e.g., get waypoints, get track, get route), however the interpretation of data received differs depending on the device. This is a bit strange, but it is the way it is. One way of going around it is to know which devices do what (GPS::Garmins’s approach). The better way could be to query which protocol version is supported (e.g., A100, A103, A108) and load the correct handler accordingly. Wonder why they didn’t do it…

Image::ExifTool I really love it. It’s a well-maintained, and a fully functioned EXIF read and manipulation library. Adding GPS data to an image is virtually 6 lines of code (taking error checking out): $exifTool->ExtractInfo($file); $exifTool->SetNewValue(GPSLatitudeRef => ($lat > 0)?'N':'S', Group=>'GPS'); $exifTool->SetNewValue(GPSLongitudeRef => ($lon > 0)?'E':"W", Group=>'GPS'); $exifTool->SetNewValue(GPSLatitude => abs($lat), Group=>'GPS'); $exifTool->SetNewValue(GPSLongitude => abs($lon), Group=>'GPS'); $exifTool->WriteInfo($file, $file."-new");

Ok, the real code is more complicated but it’s what it does:

• Read saved GPS track (array of arrays) and sort it by the timestamp
• Process files from the command line: check if they have EXIF, are not already GeoTagged or have some other problems.
• Do the binary search on the sorted timestamps and find the correct two elements.
• If the second one does not start a new segment (or the timestamp difference is not too big) approximate the position from the two points.
• Write the data back to the file, making backup if necessary

The program is really simple and works well. What remains to be done now is to add some more runtime options and write a manual. I also need to work a bit more on the garmin transfer program – I’m still not happy with GPS::Garmin, even after some basic fixes.

I will try these out, but I still want something more automated… maybe I will write it one day…

Here’s an interesting discussion on what can an cannot be done with Linux: http://mirror.hamakor.org.il/archives/linux-il/01-2005/13574.html

One pointer is laptop-mode (Documentation/laptop-mode), although for different reasons the information there might not be accurate.

A nice summary on linux profiling: http://www.cs.utk.edu/~mucci/latest/pubs/LCSC2004.pdf

Perfsuite: http://perfsuite.sourceforge.net/ IOTrack: http://www.pdc.kth.se/~pek/iotrack/

The reality shows that most of the builtfiles are written ad hoc and are of varied quality. For example, given the source code application X, there’s no common way of specifying how to build and install this application in a given path or how to build the documentation.

Wait a second… haven’t we had a similar problem with makefiles? Yessss, the answer is automake/autoconf. How about a similar tool for Ant, without autocont/automake’s wrinkles? How this would work? Well, I don’t know, but I’ve got some ideas: - there’s a template file (XML?), which specifies which files are in the project and what the dependencies are.

• AutoAnt (XSL?) generates a build.xml file having the following targets:
  1. clean
  2. build
  3. dist
  4. install

It also checks for required modules (e.g. Xerces, COLT, etc) and generates a startup file, which runs the application with required libraries (similar to –with-xxxx for configure.in).

Note that even if all jar files are in a standard location (/usr/share/java/*.jar) you still need to add these files to a CLASSPATH to be able to run the application. SOmetimes these files are not in the same location, but the application should still be able to run.

Well, that’s for ideas… maybe I will write it one day.

PROBLEM: flag TCP/UDP connections longer than XXX seconds as suspicious. ORIGINAL SOLUTION: use pcap interface in perl, analyze all the packets, look for SYN and FIN to determine the duration. PROBLEMS:

1. Essentailly reimplementing TCP stack
2. Problems with undefined behaviour, e.g. SYN,ACK,FIN,RST packets, closed ports, timeouts, etc.
3. Difficult to implements and no clear advantages.

IMPROVED SOLUTION: use ip_conntrack module in iptables and periodically (e.g. every 5 seconds) reread /proc/net/iptables file. Then parse the file, build the connection table and flag the ones that are longer than XXX.

PROBLEMS:

1. Time granularity of reread time (e.g. 5 seconds). Given a task – is it a problem???
2. Therefore completely missing connections shorter than 5 seconds.

YET IMPROVED SOLUTION: previous solution plus using iptables LOG traget to mark the begining of each TCP session (something like -m conntrack –ctstate NEW -j LOG) and then check for the termination with /proc/net/iptables. Advantages – the tracking of the beginning of the session is easier.

PROBLEMS: Still no amount of data transfer per session, but, really, it’s not a problem with task we defined.

IMPROVED IMPROVED SOLUTION: the same as above, but tracking easch TCP packet and tracking its size (LOG does not write the packet sizes, so have to use ULOG target). Determining the end of the connection using the previously defiend methods.

PROBLEMS: probably more usefull is actually looking at the amount of data transferred per source IP address rather than session.

USEFUL LINKS:

1. File::Tail – useful for reading log files in perl
2. ULOG – a really cool logger (better than LOG), but unfortunatelly no perl frontend for libipulog
3. ulog-acct – a nice packet logger using ulog – can be easily tuned to dump packed sizes, etc.
4. conntrack entries explained