Archive for the 'Personal' Category

Writing ebuilds for Gentoo – my experiences

Sunday, November 20th, 2005

Being new to the Gentoo world, I had to quickly write a few ebuilds. This process is reasonably well documented in Ebuild howto and wiki, however, some issues are not very clear.

Here’s what wasn’t obvious to me:

  1. Installing packages using ebuild and a full path to your ebuild doesn’t work (from the man-page: The implementation of emerge /path/to/ebuild is broken and so this syntax shouldn’t be used).
  2. In this case, to install your package you should probably create PORTDIR_OVERLAY in /etc/make.conf (e.g., PORTDIR_OVERLAY="/usr/local/portage", see this wiki page).
  3. Your package has to be in one of the predefined categories. If you come up with something new, Portage will happily ignore it.
  4. Before emerging, you should recompute the digest of your package using emerge /path/to/ebuild digest command.
  5. It seems that from within the package, it is difficult to distinguish whether the package is being installed for the first time vs. updated and updated vs. removed. For this, I wrote a hack, which I documented in my blog.

Detecting ebuild upgrades in Gentoo

Saturday, November 19th, 2005

I’ve recently been writing Gentoo ebuilds (interestingly enough, not being a Gentoo user myself) and faced the follwing problem: “How to distinguish between a package upgrade and a package removal from within the package?”

Gentoo has only a standard set of “hooks” that get called during package installation (related to install and uninstall, but not to upgade). The upgrade stage P1 -> P2 is performed as follows:

  • P2.configure/compile/install
  • P1.prerm()
  • remove files from P1 that are not in P2
  • P1.postrm()

This works fine for simple file replacement tasks, which are handled automatically. However, there are situations where you want something more sophisticated:

  • preinst/postinst: configure the system to use the package (services, system variables, etc).
  • prerm/postrm: remove the remnants of system configuration when the package is deleted.

For this to work, package P1 needs to know if it’s being upgraded and removed (no system conf. needed). Simiarly, P2 needs to know if P1 exists and the system has already been configured (so that the configuration is not removed on update). To make things even more difficult, a package can be upgraded, downgraded or reinstalled (for reinstall rm not called). After a bit of experimenting and scratching my head, here’s what I came up with:

#detecting upgrades while installing
pkg_preinst () {
    has_version "<${PF}" && UPGRADE=1 || UPGRADE=0
    has_version ">${PF}" && DOWNGRADE=1 || DOWNGRADE=0
    has_version "=${PF}" && REINSTALL=1 || REINSTALL=0

   #now in the preinst or postinst code we can 
   #perform different actions depending on the variables set
}

#detecting upgrades while removing
pkg_prerm () {
    has_version ">${PF}" && UPGRADE=1 || UPGRADE=0
    has_version "<${PF}" && DOWNGRADE=1 || DOWNGRADE=0  
    [ -n -${A} ] && REINSTALL=1 || REINSTALL=0

  #perform different actions depending on the variables set 
  #(either here or in posrm - the variables have to be be set right here
}

I wonder whether there’s abetter solution to this problem?

On Image GeoTagging or why I love Image::ExifTool, hate GPS::Garmin and am indifferent to Garmin’s transfer protocol.

Tuesday, November 1st, 2005

GeoTagging: This idea has been maturing long enough and last Sunday reached its critical mass. Here’s my account on the story and a few things I learned.

GeoTagging The idea is simple: when you go hiking, sightseeing, travelling you put a GPS on top of your backpack. You go and take photos as usual. When you’re back home, you connect the GPS to the computer and download the saved track. You then run a program that correlates the time when the images were taken with your position from the track and encodes this information into EXIF. As a result the images are GeoTagged and their position can be displayed by a GeoTagging-aware software.

Downloading GPS After having a look at different options, I chose two candidates for downloading the track: gpstrans and GPS::Garmin.

Note that these both support only Garmin GPSs. I don’t know if there is any universal track transfer protocol (probably not). Everything that claims to be cross-GPS typically is limited to reading the current position via NMEA.

GPStrans works ok, although it’s a bit old and doesn’t work well with my eTrex. The problems are: waypoints are corrupted, track does not contain the information about new segments and the output format is strange (this can be fixed). Afterall, I must admit that unlike other tools it worked at the first try.

GPS::Garmin gave me much more trouble. First it tunred out that it relied on some undocumented behavior of Device::Serial and at the end it turned out that it was doing a non-blocking read and considered that it would always get data. While this might have been true a couple of years ago, since then computers have gotten faster and it stopped working. After a couple of hours I fixed it by adding: $PortObj->read_const_time(5000); $PortObj->read_char_time(5000); somewhere in GPS::Serial.

The second problem was that the code relied on particular product codes of Garmin deivces and changed its behavoir accordingly. The problem is that it’s not exactly how Garmin protocol was written (unless you want to encode the behavior of all their products).

Garmin Protocol It’s fairly simple, although implementing it correctly can take a good evening. The problem is that the number of commands is limited (e.g., get waypoints, get track, get route), however the interpretation of data received differs depending on the device. This is a bit strange, but it is the way it is. One way of going around it is to know which devices do what (GPS::Garmins’s approach). The better way could be to query which protocol version is supported (e.g., A100, A103, A108) and load the correct handler accordingly. Wonder why they didn’t do it…

Image::ExifTool I really love it. It’s a well-maintained, and a fully functioned EXIF read and manipulation library. Adding GPS data to an image is virtually 6 lines of code (taking error checking out): $exifTool->ExtractInfo($file); $exifTool->SetNewValue(GPSLatitudeRef => ($lat > 0)?'N':'S', Group=>'GPS'); $exifTool->SetNewValue(GPSLongitudeRef => ($lon > 0)?'E':"W", Group=>'GPS'); $exifTool->SetNewValue(GPSLatitude => abs($lat), Group=>'GPS'); $exifTool->SetNewValue(GPSLongitude => abs($lon), Group=>'GPS'); $exifTool->WriteInfo($file, $file."-new");

Ok, the real code is more complicated but it’s what it does:

  • Read saved GPS track (array of arrays) and sort it by the timestamp
  • Process files from the command line: check if they have EXIF, are not already GeoTagged or have some other problems.
  • Do the binary search on the sorted timestamps and find the correct two elements.
  • If the second one does not start a new segment (or the timestamp difference is not too big) approximate the position from the two points.
  • Write the data back to the file, making backup if necessary

The program is really simple and works well. What remains to be done now is to add some more runtime options and write a manual. I also need to work a bit more on the garmin transfer program – I’m still not happy with GPS::Garmin, even after some basic fixes.

GeoTagging in EXIF

Friday, October 28th, 2005

For quite some time now I’ve been thinking of correlating tracklog GPS data from my GPS with the protos I’ve taken based on time. It looks such a thing alrady exists here and this blog contains a lot of good pointers.

I will try these out, but I still want something more automated… maybe I will write it one day… ;-)

Template replacement with M4

Thursday, October 20th, 2005

Trying to automate something I had to run a program with a configuration file modified for each run. The easy way to do this is to create a template file and generate the correct configuration file with variable substitution for each run.

This task could be done in a number of ways (sed, perl, shell), to name a few, but as Diego pointed out there is already a tool for this m4. It’s a bit archaic and has odd syntax (think strange quotes or strangely named built-in macros), but it’s ideal for my simple task.

At the end I run it like this: m4 -Dparam1=value1 -Dparam2=value2 <infile> > <outfile>

As simple as that. Thanks Diego! BTW: some pointers to M4: manual and linux journal article.

Slow aggergate queries in PostgreSQL

Monday, October 17th, 2005

Postgres is strange. If you make a query

select min(column_with_index) from table;

the database performs a sequential scan – you can only imagine how long it takes on a 10M record database. This seems to be a feature (!@!#!@), but indices can will used if you use the following workaround:

select column_with_index from table order by 1 (desc) limit 1;

This trick is explained here.

Blog is going public, some WP patches

Sunday, October 16th, 2005

After almost a day of editing and reorganizing, I think the blog is ready to go on-line. Many things are really chaotic, I nonetheless hope that someone might find something useful here.

I also did some minor patching of WP, making future updates will be close to impossible. Hope it will not be that bad…

To summarize today’s changes: 1. “linky” category, inspired by Diego’s blog, which was inspired by Fernando Graphicos, inspired by…. ;-)

I basically followed Fernando’s suggestions, shamelessly stealing some of Diego’s CSS (not too much though). The changes I had to make included editing of all the files containing a loop “while(have_posts()…”.

  1. Displaying hierarchical categories, instead of the flat strucutre. The change was to change “hierarchical=0″ to “hierarchical=1″ in sidebar.php.

  2. Displaying proper categories, depending on the current user. This was a bit tricky one: Depending on the user logged in, I wanted to have the correct list of categories on the main page. The patch is one line in template-functions-category.php”:

Replacing the line (somewhere around line 290): WHERE post_status = ‘publish’ with WHERE ( post_status = ‘publish’”. (($user_ID)?” or (post_status=’private’ AND post_author = $user_ID)”:”") . “)

Also $user_ID has to be declared as global somewhere before.

  1. (FAILED) Correct counting of categories. With a hierarchical system, I wanted to have a correct number of categories, e.g. if category A (5) has two sub categories A1 (2) and A2 (7), I would like A to be displayed as (14) or (5/9), as this is the number of posts you’d see if you click on it. … or is it? ;-)

I implemented is as follows: - storing a hash table of parent nodes while reading the categories (line 275) - updating the total number of posts by summing it up the chain (while ($id = $parent[$id])) {} (line 295).

This would have worked, but it has a BIG problem: a post can have both categories A and A1 set and thus would be counted twice. To avoid this, one would have to change the entire logic of a query: A single SQL query would have to be replaced by 2*n queries (for each category, querying the child categories -> using get_category_children() from classes.php) and then querying the number of posts in each of the categories. While it would have been feasible, I thought it’s not worth it and I gave up.

Fixing directory permissions with find

Wednesday, September 28th, 2005

Screwing directories and files with setgid bit: chmod -R 2775 photos

Fixing file permissions with find: find . -type f -exec chmod 664 {} ‘;’

Monitoring resource usage in Linux

Wednesday, September 14th, 2005

Three packages I found in Debian: dstat, ifstat, systat – allow to show cumulative statistics, but not on per-process basis.

Here’s an interesting discussion on what can an cannot be done with Linux: http://mirror.hamakor.org.il/archives/linux-il/01-2005/13574.html

One pointer is laptop-mode (Documentation/laptop-mode), although for different reasons the information there might not be accurate.

A nice summary on linux profiling: http://www.cs.utk.edu/~mucci/latest/pubs/LCSC2004.pdf

Perfsuite: http://perfsuite.sourceforge.net/ IOTrack: http://www.pdc.kth.se/~pek/iotrack/

Hula Project – Hula

Monday, August 22nd, 2005

Something, which might be a nice postfix/opengroupware/cyrus-imap/… replacement. All in one. A nice idea, but I feel somewhat resitant… Hula Project – Hula