Well — almost….. Argh.

I look at the URL, and it’s http://www.tnt.com/webtracker/tracking.do.

No sign of my tracking number, as obviously they’re submitting the form using METHOD=POST. I.e.: instead of encoding the request in the query string (../tracking.do?id=123456&..), they are submitting the query encoded as POST data.

This means you can’t bookmark the page, and you must either keep it open and refresh it (answering: “yes, I want to re-submit the data”), or each time go to the search form and enter the tracking number again.

If you bookmark the page, then opening the bookmark will give you this page:

Tracker

Sorry we are unable to fulfil your request
Please re-submit your enquiry.
Intellectual and other property rights to the
information contained in this site are held by
TNT Holding B.V. with all rights reserved © 2008

Well, thank you very much.

Luckily, it turns out that if you dig around the HTML for the search page and find the form variables (or use a HTTP sniffing tool like Fiddler2), their backend supports submitting the form using a query string (METHOD=GET) also:

http://www.tnt.com/webtracker/tracking.do?
respCountry=us&respLang=en&navigation=1&page=1
&sourceID=1&sourceCountry=ww&plazaKey=&refs=
&requesttype=GEN&searchType=CON
&cons=your-tracking-number-goes-here

Yay, bookmarked!

Now .. why is this hidden away behind METHOD=POST?

Sure, it could be out of “security” – so that your tracking number would not be kept in the browser’s history for others to have a peek at, and possibly to avoid someone intercepting your package.

Well, fine! But then – why not offer a link from the results page, for those of us who hopefully know what we are doing?

And seriously:

• Most modern browsers support “privacy mode”, if you’re doing sensitive stuff. (And pretty much all browsers have a “clear history” function hidden away somewhere)
• If knowing the tracking number is all it takes to hijack and intercept my package, then what kind of security is that..?

Short story:

Problems:

• mail() doesn’t work since it won’t have access to sendmail or whatever MTA binaries that are most likely located outside your chroot jail
• You’ll either need to use something else than mail() – e.g. a replacement library/class – or make some workaround to get mail() working
• Most likely a workaround is not what you want, since it means having to patch e.g. WordPress, Mediawiki and all sorts of existing software that already uses mail() (WordPress has a plugin to replace the call to mail() but I couldn’t get it working either)
• Even after trying a sendmail binary replacement (I went with mini_sendmail), something was just not working (I tried ssmtp and nbsmtp also, to no luck. In hindsight: Probably related to problem #2 listed below)
• Yes, that included adding various libraries and stuff to the chrooted area. (hint: ldd. l2chroot is also nice)

Problem 1: mini_sendmail failing to detect the current username

Tested using chroot from the shell first:

# chroot /var/www /bin/mini_sendmail
/bin/mini_sendmail: can't determine username

So what’s that about?

Well, there’s a segment of code that tries to figure out the login/username using getlogin() and it even has some #ifdef’s to try getpwuid(getuid()) instead, but for some reason both of them fail.

Manually setting it to some dummy value seemed to be the solution:

username = "blah"; // doesn't work:  getlogin();

Problem 2: replacement binaries were simply just not working at all from Apache/PHP

The symptom was …. PHP’s mail() was just returning “false”. Nothing in the Apache error logs or anywhere else.

I even tried creating a C program that just logged the arguments and any STDIN input to a file, pointed the php.ini file to it, but nothing appeared in the output file either.

I tried calling the binaries using system() and exec() from PHP, but they returned “” (blank string) and nothing was logged in the Apache logs. WTF?

Turns out you need to have /bin/sh in your chroot jail, or else PHP refuses to run. I guess this is because it uses system(3) which forks a shell, instead of using exec(3) or execve(2).

I don’t know. Adding /bin/sh worked. End of story.

Problem 3: mini_sendmail screwing up while parsing email addresses from mail headers

When I tried to get this working with Mediawiki, I ran into another problem: the activation mails weren’t being sent out. A little bit tcpdump was all it took to notice what was going on there: the following command was being rejected by the MTA:

RCPT TO:<Real Name <foo@example.org>

Here the arg_dumper C binary I wrote came handy. This was what it showed being passed to mail() from Mediawiki:

To: Real Name <foo@example.org>

Two problems here:

1. The ‘RCPT TO’ command should not include the descriptive name of the recipient; only the address should be included
2. Where did the last > go in the SMTP ‘RCPT TO:’ command, anyhow?

Turns out it was the add_recipient() method in mini_sendmail that doesn’t handle this very well:

    /* Skip leading whitespace. */
    while ( len > 0 && ( *recipient == ' ' || *recipient == '\t' ) )
        {
        ++recipient;
        --len;
        }

    /* Strip off any angle brackets. */
    while ( len > 0 && *recipient == '<' )
        {
        ++recipient;
        --len;
        }
    while ( len > 0 && recipient[len-1] == '>' )
        --len;

So, what does it do? First it chops off any whitespace and leading opening angle brackets (‘<’). Then it chops off any trailing closing angle brackets. (‘>’).

This effectively chops off the last character in the Mediawiki template. So, I modified that to something like:

    /* Skip leading whitespace. */
    while ( len > 0 && ( *recipient == ' ' || *recipient == '\t' ) )
        {
        ++recipient;
        --len;
        }

    /* Strip off any angle brackets.  (fixed?) */

  if ( (len > 0) && (recipient[len-1] == '>') )
    {
    while ( len > 0 && *recipient != '<' )
        {
        ++recipient;
        --len;
        }

    /* skip leading < we stopped at */
    --len;
    ++recipient;

    while ( len > 0 && recipient[len-1] == '>' )
        --len;
    }
}

I.e.: only do this bracket-stripping if there is a trailing bracket, and chop off anything before the first bracket.

..which works with all of these:

To: Real Name <foo@example.org>
To: <foo@example.org>
To: foo@example.org

Produces:

RCPT TO:<foo@example.org>

…

Yay.

I can haz mod_chroot’ed mail().

Hearing “DING” and Windows startup and shutdown sounds from people in the IT department or who are supposed to be “system administrators” really scares me…

It makes me wonder what other annoying, useless default features they’ve left enabled… :P

I choose to write “watermark” quoted, as this is really more tagging than watermarking, because I doubt they will be applying on-the-fly digital audio processing to watermark, followed by encoding each mp3 specifically for each customer. Let’s assume they’re going to just modify stocks of ready-encoded mp3s.

MP3 file format layout

Before continuing it’s important to understand how MP3 files are stored, so let’s take a closer look at how they are structured internally:

ID3v2 tag (optional)
LAME header (optional)
MPEG frame 1
MPEG frame 2
..
MPEG frame N
ID3v1/ID3v1.1 tag (optional)

ID3 tags

The ID3 tags contain various metadata about the file, which is normally either entered automatically by the encoder or edited manually by the person encoding the files. Two standards exist:

ID3v1 – which later got a slight modification and became ID3v1.1 – is a 128-byte fixed length structure located at the end of files.

ID3v2 which is a more dynamic structure, typically located at the beginning of the file. This allows for a plethoria of different information to be stored, with several extensions being made all the time. (e.g. Lyrics3 which allows the lyrics transcript of a song to be embedded into the file)

MPEG frames

MPEG frames are small chunks of audio data. The size of these frames will depend on the bitrate used in encoding the file, but each header will be prefixed with a 4 byte frame header.

These headers contain information needed to interpret and make use of the encoded data (e.g. which MPEG encoding method is used, which bitrate the frame is encoded at, the sampling frequency etc) but also non-functional data such as a “protection bit”, “private bit”, “copyright bit”, “original bit” etc.

For a full description of the MPEG headers see e.g. this article at mp3-tech.org or search the web.

LAME header

Some encoders, such as LAME (and Xing), add frames that appear as regular MPEG frames, but that actually contain additional meta data about the encoding parameters used etc.

SO WHERE DOES THIS LEAD US?

So where does this lead us? Let’s have a look at a few (but probably not all) ways of tagging mp3 files with some kind of watermarks. (with a varying depth of detail)

1. Adding out-of-stream data to the MPEG stream

Because MPEG frame headers contain a sync marker so that a player can check if the next location it is about to read is a valid MPEG frame, this means that you can place out-of-stream data in between frames.

This will just make players skip a few bytes until it finds a valid MPEG frame header sync marker, but other applications can choose to store custom data in here.

2. Using the unused MPEG frame header bits

As mentioned above, MPEG frame headers Bits like the “original bit”, “private bit” etc are not of much use to players, so for each frame of the MP3 file you can store up to several bits of information.

Spread across the entire file, depending on the length of the song, this would allow for quite a lot of data which can be used as tracking markers once read by a special program.

3. Using the IDv1 tag

I find this not very likely, but as a method you could choose to utilize the “comment” field of the ID3v1 header to add some sort of numeric ID etc.

4. Using the ID3v2 tag

There’s primarily two methods here:

a) Using an unused or little-used, or even creating a custom ID3v2 block

b) Using the unused padding space of the ID3v2 tag

5. Constructing special MPEG frames like the LAME header

You could divide this into two methods also:

a) Frames using one of the undefined or unallowed bit combinations of a frame header to mark it as invalid, so that the player will skip it. Then custom data can be stored in the actual data portion of the frame. (I think this is what LAME does)

b) Special valid encoded frames consising of an audio watermark (e.g. some audio wave) could be added. This would probably play back as a short click or noise, though, so it might not end up sounding too good.

6. Using the MPEG frame header CRC

The encoder can optionally add checksums to each MPEG frame, so the validity of the file can be tested for corrupted frames etc.

Seeing that not many players actually test for this even if it is present, this could be used to insert 2 bytes of data per frame header.

It could be a bit risky doing this, though, should any players actually choose to test the validity of frames against the CRC.

7. Using variations of volume in each MPEG frame

Quite honestly this is beyond what I know too much about, but I know certain tools (e.g. mp3gain) can normalize or change the volume of an mp3 without transcoding (re-encoding), and thus offers a non-destructive way of doing this.

This is based on the fact that data is (I think) stored as floating point values. These values again are stored using a “sign * mantissa * radix ^ exponent” format, which means that you can increment or modify these by fixed values back and forth and introduce gradual change with the option of still getting back to the original values (up to a certain level I guess).

Utilizing this, I guess you could somehow introduce short changes in volume from frame to frame that would go by unnoticed by the listener, but through analysis could be detected. E.g. think morse code.

I don’t know if this would work. Maybe. It’s just an idea..

CONCLUSION

There’s a lot of places you can hide information in an MP3 file. However, it does not take more than having 2 different watermarked copies of the same file to figure out where data is being stored.

Of course, one could combine all the methods described above, or even additional ones, but in the end all of this information could be stripped away leaving only the MP3 audio data left.

We’ll find out soon enough as these files hit the streets..

Update
Someone pointed out the fact that on-the-fly watermarking and transcoding would be perfectly well possible on smaller ranges of the files, an option I never considered…..

Update 2011
This article is pretty much outdated, as todays computing power allows for on-the-fly watermarking and re-encoding. However, the methods described are still valid, but not very hard to defeat.