Well — almost….. Argh.

I look at the URL, and it’s http://www.tnt.com/webtracker/tracking.do.

No sign of my tracking number, as obviously they’re submitting the form using METHOD=POST. I.e.: instead of encoding the request in the query string (../tracking.do?id=123456&..), they are submitting the query encoded as POST data.

This means you can’t bookmark the page, and you must either keep it open and refresh it (answering: “yes, I want to re-submit the data”), or each time go to the search form and enter the tracking number again.

If you bookmark the page, then opening the bookmark will give you this page:

Tracker

Sorry we are unable to fulfil your request
Please re-submit your enquiry.
Intellectual and other property rights to the
information contained in this site are held by
TNT Holding B.V. with all rights reserved © 2008

Well, thank you very much.

Luckily, it turns out that if you dig around the HTML for the search page and find the form variables (or use a HTTP sniffing tool like Fiddler2), their backend supports submitting the form using a query string (METHOD=GET) also:

http://www.tnt.com/webtracker/tracking.do?
respCountry=us&respLang=en&navigation=1&page=1
&sourceID=1&sourceCountry=ww&plazaKey=&refs=
&requesttype=GEN&searchType=CON
&cons=your-tracking-number-goes-here

Yay, bookmarked!

Now .. why is this hidden away behind METHOD=POST?

Sure, it could be out of “security” – so that your tracking number would not be kept in the browser’s history for others to have a peek at, and possibly to avoid someone intercepting your package.

Well, fine! But then – why not offer a link from the results page, for those of us who hopefully know what we are doing?

And seriously:

• Most modern browsers support “privacy mode”, if you’re doing sensitive stuff. (And pretty much all browsers have a “clear history” function hidden away somewhere)
• If knowing the tracking number is all it takes to hijack and intercept my package, then what kind of security is that..?

Short story:

Problems:

• mail() doesn’t work since it won’t have access to sendmail or whatever MTA binaries that are most likely located outside your chroot jail
• You’ll either need to use something else than mail() – e.g. a replacement library/class – or make some workaround to get mail() working
• Most likely a workaround is not what you want, since it means having to patch e.g. WordPress, Mediawiki and all sorts of existing software that already uses mail() (WordPress has a plugin to replace the call to mail() but I couldn’t get it working either)
• Even after trying a sendmail binary replacement (I went with mini_sendmail), something was just not working (I tried ssmtp and nbsmtp also, to no luck. In hindsight: Probably related to problem #2 listed below)
• Yes, that included adding various libraries and stuff to the chrooted area. (hint: ldd. l2chroot is also nice)

Problem 1: mini_sendmail failing to detect the current username

Tested using chroot from the shell first:

# chroot /var/www /bin/mini_sendmail
/bin/mini_sendmail: can't determine username

So what’s that about?

Well, there’s a segment of code that tries to figure out the login/username using getlogin() and it even has some #ifdef’s to try getpwuid(getuid()) instead, but for some reason both of them fail.

Manually setting it to some dummy value seemed to be the solution:

username = "blah"; // doesn't work:  getlogin();

Problem 2: replacement binaries were simply just not working at all from Apache/PHP

The symptom was …. PHP’s mail() was just returning “false”. Nothing in the Apache error logs or anywhere else.

I even tried creating a C program that just logged the arguments and any STDIN input to a file, pointed the php.ini file to it, but nothing appeared in the output file either.

I tried calling the binaries using system() and exec() from PHP, but they returned “” (blank string) and nothing was logged in the Apache logs. WTF?

Turns out you need to have /bin/sh in your chroot jail, or else PHP refuses to run. I guess this is because it uses system(3) which forks a shell, instead of using exec(3) or execve(2).

I don’t know. Adding /bin/sh worked. End of story.

Problem 3: mini_sendmail screwing up while parsing email addresses from mail headers

When I tried to get this working with Mediawiki, I ran into another problem: the activation mails weren’t being sent out. A little bit tcpdump was all it took to notice what was going on there: the following command was being rejected by the MTA:

RCPT TO:<Real Name <foo@example.org>

Here the arg_dumper C binary I wrote came handy. This was what it showed being passed to mail() from Mediawiki:

To: Real Name <foo@example.org>

Two problems here:

1. The ‘RCPT TO’ command should not include the descriptive name of the recipient; only the address should be included
2. Where did the last > go in the SMTP ‘RCPT TO:’ command, anyhow?

Turns out it was the add_recipient() method in mini_sendmail that doesn’t handle this very well:

    /* Skip leading whitespace. */
    while ( len > 0 && ( *recipient == ' ' || *recipient == '\t' ) )
        {
        ++recipient;
        --len;
        }

    /* Strip off any angle brackets. */
    while ( len > 0 && *recipient == '<' )
        {
        ++recipient;
        --len;
        }
    while ( len > 0 && recipient[len-1] == '>' )
        --len;

So, what does it do? First it chops off any whitespace and leading opening angle brackets (‘<’). Then it chops off any trailing closing angle brackets. (‘>’).

This effectively chops off the last character in the Mediawiki template. So, I modified that to something like:

    /* Skip leading whitespace. */
    while ( len > 0 && ( *recipient == ' ' || *recipient == '\t' ) )
        {
        ++recipient;
        --len;
        }

    /* Strip off any angle brackets.  (fixed?) */

  if ( (len > 0) && (recipient[len-1] == '>') )
    {
    while ( len > 0 && *recipient != '<' )
        {
        ++recipient;
        --len;
        }

    /* skip leading < we stopped at */
    --len;
    ++recipient;

    while ( len > 0 && recipient[len-1] == '>' )
        --len;
    }
}

I.e.: only do this bracket-stripping if there is a trailing bracket, and chop off anything before the first bracket.

..which works with all of these:

To: Real Name <foo@example.org>
To: <foo@example.org>
To: foo@example.org

Produces:

RCPT TO:<foo@example.org>

…

Yay.

I can haz mod_chroot’ed mail().

Hearing “DING” and Windows startup and shutdown sounds from people in the IT department or who are supposed to be “system administrators” really scares me…

It makes me wonder what other annoying, useless default features they’ve left enabled… :P

Prior to SL2-B2, the only restriction for socket connections was that the connection had to be done against the same hostname/IP-address that served the Silverlight application, and that the TCP port needed to be in the range 4502-4534.

Release candidate 0 (SL2-RC0) and the final “Release-To-Web” release (SL2-RTW) didn’t introduce any new changes.

So, the restrictions are now:

1. Connections can only be made to TCP ports in the range 4502-4534
2. A socket policy server must be running on port 943 on the same server
3. The policy returned by the policy server must match with the application performing the request

The arguments for introducing the policy server seems to be:

1. Added security
2. A server hosting a socket server is not likely to also run a web server

Some related links:

1. http://timheuer.com/blog/archive/2008/06/06/silverlight-sockets-requires-policy-server-beta-2.aspx
2. http://silverlight.net/blogs/msnow/archive/2008/06/26/full-implementation-of-a-silverlight-policy-server.aspx

Added security?

The first point is of course a partially valid one. If you have a socket service, you might want to limit who has access to the service in order to prevent unauthorized use of it, or prevent so-called Cross-Site Request Forgery (XSRF) attacks.

Basically this is what’s going on:

1. The user goes to http://server1.example.org/foo.html and is served a HTML page with an embedded Silverlight application
2. The Silverlight app tries to do a connection against server2.example.org:4502
3. Silverlight intercepts the request and does a connection against server2.silverlight.example.org:943 and requests the policy XML
4. Silverlight checks if the policy file accepts connections to server2.example.org:4502 from server1.example.org/foo.html or server1.example.org or *.example.org or some kind of match
5. The socket request is then either allowed or disallowed, based on the outcome of if a policy server answered, and gave an XML allowing the request

This also is more or less the same process with web service calls or web requests (http/https) from within a Silverlight application, with the change being that the access policy file is then read from http://<same-server-and-port-as-the-app-was-served-from>/ClientAccessPolicy.xml (or the Flash crossdomain.xml file).

However, the limitation of what services Silverlight applications are able to call is implemented in the Silverlight plugin itself! If you make an external application (i.e. not in Silverlight) in <your favorite programming language> that does socket, http/https or web service calls against these services, these are of course not checked against the access policy files first.

In other words: It leaves these services wide open to any other non-restricted applications doing calls against them. This only protects against XSRF attacks, and nothing else.

For the services to be really secure and limit requests, these access checks need to be implemented in the socket services or in the web services etc themselves. You simply cannot rely on that some policy file has been checked in advance. Of course, in a socket server you can’t figure out which web page the application originated from* – only what remote IP and port the caller has – so the policy files do still make a little bit of sense there. (* You could maybe record the IP address in an ASPX page serving the Silverlight app and do something smart there and check this again in the socket)

“A server hosting a socket server is not likely to also run a web server”

Now, I don’t know where this idea came from, or what the reasoning behind it was. My reactions are:

1. Why not also do a request against and HTTP service on the same server, in case there is a web server running there already?
2. Why not allow the policy server port to be configurable? If someone is already running a socket server there in the 4502-4534 range, surely they will be able to set up a service running on port 943 also. (Yes, some OSes limit listening sockets below port 1024 or 4096 to the root/administrator user, but how likely is that?). Also, it’s not like this will be directly insecure either, as the requesting client can’t affect which socket servers are running on the remote server anyhow, AND the policy file needs to be checked. I guess one argument could be that this allows for an administrator to manage what goes on, instead of permitting the individual socket server developers to do that.
3. The policy file is a standard XML file. It is served by a client connecting to a service, issuing a request, and getting the file back. Does this sound vaguely familiar with another, very common protocol? Why insist on creating a new, custom protocol for this, instead of just basing this on HTTP? (see #1)

(Regarding #1, this makes me think of another weird decision: leaving out double-click mouse events or mouse-wheel support just because some interfaces might not have anything equivalent. It’s like limiting the screen resolution to 640×480 just in case some users don’t have a monitor capable of a higher resolution, or leaving out mouse support all-together in case some devices might only have keyboard, and no joystick/mouse/touch screen etc. I don’t get it — it reduces the user experience at the cost of a hypothetical uncommon scenario)

Firewalls

Most non-standard ports will normally be blocked by most “secure” firewall settings, and if you’re behind a corporate firewall, chances are you’ll experience even more restrictive policies with only a very few ports being open, e.g. port 80 and 443 for http/https.

Adding both a limit of a policy server running on port 943, AND the request being limited to ports 4502-4534 will make this virtually impossible to get to work in most corporate settings, and also other restrictive firewall setups.

It can be argued in reverse also; enforcing this range makes it easier to standardize the port range for access lists. Yes, but; from my experience with firewall administrators and IT departments, they are pretty sensitive about what they will put into their rules. They will most likely only open up the range to a select destination IP or range, and then they could just as well have done that for another port range. Opening up 4502-4534 to all target destinations is not likely to happen.

The common work-around for this problem, is for people to just set up e.g. a non-HTTP service on a the common HTTP port, or something similar. As long as the firewall isn’t inspecting the traffic and doing protocol analysis, this usually works fine. If you have packet inspection, you’ll probably be screwed anyhow.

Secondly, it limits calls against standard services, such as chat servers, web servers, ftp servers, etc since these do not typically run in the 4502-4534 range, but usually have their own defined ports. If you want to be able to connect to such services, you now need to either reconfigure them to use this port range, or write a proxy server that proxies/tunnels connections and traffic to the correct ports.

My conclusion

My conclusion to this, is that Microsoft weighed the security aspects of this to be much more imporant that offering developers flexibility and possibilities, a choice I can clearly understand and respect. However, in that case I think they really outdid themselves in adding obstacles.

Or; they just didn’t carefully think this through before rushing out SL2-RTW.. (because let’s admit it; one RC, ~2 weeks, hardly any changes?)

What I feel would have sufficed:

1. Access policy file must be present either on either a provided port (default 943 if they insist), or on http/https on the same server.
2. Use something equivalent to HTTP instead of that custom protocol. It could perfectly well be VERY limited (“GET /ClientAccessPolicy.xml\n\n” + dummy headers + XML response).
3. No port range restriction, other than what is enforced through the access policy file. (Or at least offer the possibility of requesting “elevated privileges” from the user in form of a dialog or something for using non-SL-range ports, e.g. like Java does).

I choose to write “watermark” quoted, as this is really more tagging than watermarking, because I doubt they will be applying on-the-fly digital audio processing to watermark, followed by encoding each mp3 specifically for each customer. Let’s assume they’re going to just modify stocks of ready-encoded mp3s.

MP3 file format layout

Before continuing it’s important to understand how MP3 files are stored, so let’s take a closer look at how they are structured internally:

ID3v2 tag (optional)
LAME header (optional)
MPEG frame 1
MPEG frame 2
..
MPEG frame N
ID3v1/ID3v1.1 tag (optional)

ID3 tags

The ID3 tags contain various metadata about the file, which is normally either entered automatically by the encoder or edited manually by the person encoding the files. Two standards exist:

ID3v1 – which later got a slight modification and became ID3v1.1 – is a 128-byte fixed length structure located at the end of files.

ID3v2 which is a more dynamic structure, typically located at the beginning of the file. This allows for a plethoria of different information to be stored, with several extensions being made all the time. (e.g. Lyrics3 which allows the lyrics transcript of a song to be embedded into the file)

MPEG frames

MPEG frames are small chunks of audio data. The size of these frames will depend on the bitrate used in encoding the file, but each header will be prefixed with a 4 byte frame header.

These headers contain information needed to interpret and make use of the encoded data (e.g. which MPEG encoding method is used, which bitrate the frame is encoded at, the sampling frequency etc) but also non-functional data such as a “protection bit”, “private bit”, “copyright bit”, “original bit” etc.

For a full description of the MPEG headers see e.g. this article at mp3-tech.org or search the web.

LAME header

Some encoders, such as LAME (and Xing), add frames that appear as regular MPEG frames, but that actually contain additional meta data about the encoding parameters used etc.

SO WHERE DOES THIS LEAD US?

So where does this lead us? Let’s have a look at a few (but probably not all) ways of tagging mp3 files with some kind of watermarks. (with a varying depth of detail)

1. Adding out-of-stream data to the MPEG stream

Because MPEG frame headers contain a sync marker so that a player can check if the next location it is about to read is a valid MPEG frame, this means that you can place out-of-stream data in between frames.

This will just make players skip a few bytes until it finds a valid MPEG frame header sync marker, but other applications can choose to store custom data in here.

2. Using the unused MPEG frame header bits

As mentioned above, MPEG frame headers Bits like the “original bit”, “private bit” etc are not of much use to players, so for each frame of the MP3 file you can store up to several bits of information.

Spread across the entire file, depending on the length of the song, this would allow for quite a lot of data which can be used as tracking markers once read by a special program.

3. Using the IDv1 tag

I find this not very likely, but as a method you could choose to utilize the “comment” field of the ID3v1 header to add some sort of numeric ID etc.

4. Using the ID3v2 tag

There’s primarily two methods here:

a) Using an unused or little-used, or even creating a custom ID3v2 block

b) Using the unused padding space of the ID3v2 tag

5. Constructing special MPEG frames like the LAME header

You could divide this into two methods also:

a) Frames using one of the undefined or unallowed bit combinations of a frame header to mark it as invalid, so that the player will skip it. Then custom data can be stored in the actual data portion of the frame. (I think this is what LAME does)

b) Special valid encoded frames consising of an audio watermark (e.g. some audio wave) could be added. This would probably play back as a short click or noise, though, so it might not end up sounding too good.

6. Using the MPEG frame header CRC

The encoder can optionally add checksums to each MPEG frame, so the validity of the file can be tested for corrupted frames etc.

Seeing that not many players actually test for this even if it is present, this could be used to insert 2 bytes of data per frame header.

It could be a bit risky doing this, though, should any players actually choose to test the validity of frames against the CRC.

7. Using variations of volume in each MPEG frame

Quite honestly this is beyond what I know too much about, but I know certain tools (e.g. mp3gain) can normalize or change the volume of an mp3 without transcoding (re-encoding), and thus offers a non-destructive way of doing this.

This is based on the fact that data is (I think) stored as floating point values. These values again are stored using a “sign * mantissa * radix ^ exponent” format, which means that you can increment or modify these by fixed values back and forth and introduce gradual change with the option of still getting back to the original values (up to a certain level I guess).

Utilizing this, I guess you could somehow introduce short changes in volume from frame to frame that would go by unnoticed by the listener, but through analysis could be detected. E.g. think morse code.

I don’t know if this would work. Maybe. It’s just an idea..

CONCLUSION

There’s a lot of places you can hide information in an MP3 file. However, it does not take more than having 2 different watermarked copies of the same file to figure out where data is being stored.

Of course, one could combine all the methods described above, or even additional ones, but in the end all of this information could be stripped away leaving only the MP3 audio data left.

We’ll find out soon enough as these files hit the streets..

Update
Someone pointed out the fact that on-the-fly watermarking and transcoding would be perfectly well possible on smaller ranges of the files, an option I never considered…..

Update 2011
This article is pretty much outdated, as todays computing power allows for on-the-fly watermarking and re-encoding. However, the methods described are still valid, but not very hard to defeat.