Tadek’s Blog

After resuming my geotaggin script (see this post), I decided to do something useful with it. We’re using gallery2 to store our photos and with a [googlemap plugin](http://codex.gallery2.org/index.php/Gallery2:Modules:Map), but found it useful only for displaying a single pointer per album (see [here](http://gallery.ibao.net/Map/)). For a more fine-grained selection we needed something else.

Therefore, I decided to write my own plugin (yeah, there are already two out there, why not write my own? ) and also learn Gallery2 API. The idea is to display a google map at the bottom of each photo, showing exactly where the photo was taken. Yes, it photo-specific and there’s only one pointer on the map. I find it nonetheless very useful.

Here’s a [sample output](http://gallery.ibao.net/travel/Switzerland/melchsee/IMG_2991.JPG.html) (you can also admire the beautiful scenery of Melchsee ). The plugin adds a new “block” in the template (therefore can be configured using a standard block management tool in Gallery2).

The position of the current photo is always in the middle (although you can move the map around, change the map type, zoom in and out etc.). The changes you make are stored as session cookies, and preserved between consecutive photo loads. Also, the whole panel can be hidden to speed up load and only shown on demand (show map|hide map).

Any comments? suggestions? ideas?

The plugin is currently in _alpha_ stage, I will release it in a week or two (I want to create a webpage for it as well). In the meantime, if you’re interested in trying it out, drop me a line

BTW: I also found out that when iPhoto edits a photo, it converts Exif from Intel to Motorola (little endian -> big endian). There was a bug in [exifer](http://www.offsky.com/software/exif/index.php) used in gallery, which corrpted the tags. The patch is only two lines long and can be found here (I also emailed the author):

--- gps.inc.orig        2006-08-31 10:25:27.000000000 +0200
+++ gps.inc     2006-08-31 10:36:37.000000000 +0200
@@ -116,13 +116,24 @@
                        $minutes = GPSRational(substr($data,16,16),$intel);
                        $hour = GPSRational(substr($data,32,16),$intel);

+      /* now we need a hack, since the whole data has been flipped in :103
+                        * the order here is sec:min:hour. However, in the motorla mode the data
+                        * has not been flipped and the order is h:m:s. This breaks compatibility
+                        * with Motorola exif. (Tadek) */
+      if($intel==1)
                        $data = $hour+$minutes/60+$seconds/3600;
+                       else
+                                       $data = $seconds+$minutes/60+$hour/3600;
                } else if($tag=="0007") { //Time
                        $seconds = GPSRational(substr($data,0,16),$intel);
                        $minutes = GPSRational(substr($data,16,16),$intel);
                        $hour = GPSRational(substr($data,32,16),$intel);

+                       /* I guess the same HACK as above. Tadek */
+                       if ($intel==1)
                        $data = $hour.":".$minutes.":".$seconds;
+                       else
+                                 $data = $seconds.":".$minutes.":".$hour;
                } else {
                        if($bottom!=0) $data=$top/$bottom;
                        else if($top==0) $data = 0;

Everybody has worries, which stay in our minds. For example, Currently, I am worrying about at least a dozen of things (maybe Poles are a worrying nation, or it is just me, don’t know) and, you must admit, it’s hard to stay productive being occupied with worrying.

Hence, during our lunch discussion yesterday we came up with the idea of _worriors_, to whom you can outsource your worries. To be completely honest this idea came from [My Outsourced Life](http://www.smartmoney.com/esquire/index.cfm?Story=20050909-outsource), I think the notion of “worriors” is ours

The idea is to have a website, where you can submit your worries, which a professional worrior (human or automated) can worry about. You can log in and check what they are currently worrying about or maybe even sign up for a comforting e-mail:

> Don’t worry.
> We are currently worrying about <> for you.
> Yours professional worriors”

This can also be a “worries outlet”: discussions, support groups and of course a great chance for advertising.

Now that I have the infrastrucutre for asking meaningful queries, here’s what I could be interesting in. These can be nicely scripted and shown on the webpage using CGI (all these are sliding windows with no input parameters, thus no worries about SQL injections).

BTW: I know this SQL is ugly like hell, but the framwork is really flexible and these queries are very easy to write. Here they go:

(more…)

Since a while I have been (experimentally) running Bro on my server collecting connection summaries. This is a very nice feature of Bro, generating really accurate TCP flows (including information whether the connection was terminated correctly or not and others). The big advantage of connection summaries (over e.g. NetFlow) is their high accuracy and yet compact representation (for over 2 months ov data, I have only 21MB (250k lines) of data.

I wrote a simple perl parser parsing Bro’s connection status and writing it to a relational database (in this case Postgres). The parser can also run in “query” mode, producing a comma/space separated data for easy visualization (using AfterGlow or even GnuPlot).

Here’s how I run my collection:

sudo bin/bro -i eth0 -f "host 85.10.194.212" conn


And parsing:

tail -f conn.log | ../conn2db2csv.pl -i -d "dbi:Pg:dbname=bro;host=localhost" -u bro -p <password>


And plotting:

./conn2db2csv.pl -q "select time::date,dport, sum(brecv) from conn where dip='85.10.194.212' group by 1,2 having(sum(brecv) >0) order by 1,2" -d "dbi:Pg:dbname=bro;host=localhost" -u bro -p <password>


This can be nicely used by gnuplot. For example to plot daily traffic on let’s say port 22:

./conn2db2csv.pl -q "select time::date, sum(brecv) from conn where dip='85.10.194.212' and dport = 22 group by 1 having(sum(brecv) >0) order by 1" -d "dbi:Pg:dbname=bro;host=localhost" -u bro -p <password> -s ' ' > testplot

gnuplot
set timefmt "%Y-%m-%d %H:%M:%S"
set xdata time
plot 'testplot' using 1:3


GeoTagging: This idea has been maturing long enough and last Sunday reached its critical mass. Here’s my account on the story and a few things I learned.

GeoTagging
The idea is simple: when you go hiking, sightseeing, travelling you put a GPS on top of your backpack. You go and take photos as usual. When you’re back home, you connect the GPS to the computer and download the saved track. You then run a program that correlates the time when the images were taken with your position from the track and encodes this information into EXIF. As a result the images are GeoTagged and their position can be displayed by a GeoTagging-aware software.

Downloading GPS
After having a look at different options, I chose two candidates for downloading the track: gpstrans and GPS::Garmin.

Note that these both support only Garmin GPSs. I don’t know if there is any universal track transfer protocol (probably not). Everything that claims to be cross-GPS typically is limited to reading the current position via NMEA.

GPStrans works ok, although it’s a bit old and doesn’t work well with my eTrex. The problems are: waypoints are corrupted, track does not contain the information about new segments and the output format is strange (this can be fixed). Afterall, I must admit that unlike other tools it worked at the first try.

GPS::Garmin gave me much more trouble. First it tunred out that it relied on some undocumented behavior of Device::Serial and at the end it turned out that it was doing a non-blocking read and considered that it would always get data. While this might have been true a couple of years ago, since then computers have gotten faster and it stopped working. After a couple of hours I fixed it by adding:

$PortObj->read_const_time(5000);
$PortObj->read_char_time(5000);

somewhere in GPS::Serial.

The second problem was that the code relied on particular product codes of Garmin deivces and changed its behavoir accordingly. The problem is that it’s not exactly how Garmin protocol was written (unless you want to encode the behavior of all their products).

Garmin Protocol
It’s fairly simple, although implementing it correctly can take a good evening. The problem is that the number of commands is limited (e.g., get waypoints, get track, get route), however the interpretation of data received differs depending on the device. This is a bit strange, but it is the way it is. One way of going around it is to know which devices do what (GPS::Garmins’s approach). The better way could be to query which protocol version is supported (e.g., A100, A103, A108) and load the correct handler accordingly. Wonder why they didn’t do it…

Image::ExifTool
I really love it. It’s a well-maintained, and a fully functioned EXIF read and manipulation library. Adding GPS data to an image is virtually 6 lines of code (taking error checking out):

$exifTool->ExtractInfo($file);
$exifTool->SetNewValue(GPSLatitudeRef => ($lat > 0)?'N':'S', Group=>'GPS');
$exifTool->SetNewValue(GPSLongitudeRef => ($lon > 0)?'E':"W", Group=>'GPS');
$exifTool->SetNewValue(GPSLatitude => abs($lat), Group=>'GPS');
$exifTool->SetNewValue(GPSLongitude => abs($lon), Group=>'GPS');
$exifTool->WriteInfo($file, $file."-new");


Ok, the real code is more complicated but it’s what it does:

• Read saved GPS track (array of arrays) and sort it by the timestamp
• Process files from the command line: check if they have EXIF, are not already GeoTagged or have some other problems.
• Do the binary search on the sorted timestamps and find the correct two elements.
• If the second one does not start a new segment (or the timestamp difference is not too big) approximate the position from the two points.
• Write the data back to the file, making backup if necessary

The program is really simple and works well. What remains to be done now is to add some more runtime options and write a manual. I also need to work a bit more on the garmin transfer program – I’m still not happy with GPS::Garmin, even after some basic fixes.

For quite some time now I’ve been thinking of correlating tracklog GPS data from my GPS with the protos I’ve taken based on time. It looks such a thing alrady exists here and this blog contains a lot of good pointers.

I will try these out, but I still want something more automated… maybe I will write it one day…

Three packages I found in Debian: dstat, ifstat, systat – allow to show cumulative statistics, but not on per-process basis.

Here’s an interesting discussion on what can an cannot be done with Linux:

One pointer is laptop-mode (Documentation/laptop-mode), although for different reasons the information there might not be accurate.

A nice summary on linux profiling:

Perfsuite:
IOTrack:

Yesss, it’s a good idea. Ant is a substitute for make, designed to be free of make’s wrinkles. It is, however, only a make substitute, and allows only to write build files.

The reality shows that most of the builtfiles are written ad hoc and are of varied quality. For example, given the source code application X, there’s no common way of specifying how to build and install this application in a given path or how to build the documentation.

Wait a second… haven’t we had a similar problem with makefiles? Yessss, the answer is automake/autoconf. How about a similar tool for Ant, without autocont/automake’s wrinkles?
(more…)

Inspired by discussions with Ashish, here are some thoughts on tracking connections in Linux.

PROBLEM: flag TCP/UDP connections longer than XXX seconds as suspicious.
(more…)