Reverse-engineering Gmail: Initial remarks

For the last week and a bit, I have been trying to do a particular something with Gmail. (Specifically, get at the Message-ID headers of messages.) This has been mostly a failure, but that’s not so surprising, as I had little experience with “all this web stuff”: JavaScript, AJAX, DOM, browser incompatibilities, Firebug, Greasemonkey… round up the usual buzzwords. I have learnt a bit, though, and thought it might help others starting in a similar situation. (And there’s also the hope that someone might actually find this and help me!)

The story so far
Gmail was launched in April 2004. Since then, it has been through many changes, the latest around October 2007 when there came to our inboxes a “Newer version”, also sometimes called “Gmail 2”. (Note that officially Gmail is still in Beta; it hasn’t even released a 1.0!)
When Gmail was released the set of practices that go by the name of “AJAX” was still new and unfamiliar; it has been refined and better-understood since. (And it turns out to require neither asynchrony nor JavaScript nor XML.)

Johnvey Hwang reverse-engineered much of Gmail’s original version, and even made a “Gmail API” out of it. It no longer works of course, and the site is often down too, but it’s available on the Wayback Machine and the section documenting “the Gmail engine and protocol” is still worth a read, if only for its glimpse into the labyrinthine ways in which Ajax applications can work. He turned it (in May 2005) into a SourceForge project (“Gmail API”), last updated June 2005, and the associated Google Group (” Gmail Agent API”) is also largely defunct and indicates that the API, or whatever came of it, has not been working since the changes in October 2007, at any rate.

My goal
At this point, I might as well reveal what I want to do: I want to make it easy to get the “Message-ID:” header of messages in Gmail. (I like to read email in Gmail but not to send, so one way to reply to a specific message would be to get the Message-ID and ask my other mail client to reply to the message with that message-ID.) In the current interface, the (only) way of getting it is to click on the pulldown menu next to “Reply”, and click on “Show original”. This will open up a page that contains the raw text of the message with all its headers, and “Message-ID:” is always one of them. Since I use Firefox, I’ve been trying to make this easier with a Greasemonkey script.

Trap-patching the P() function
As Greasemonkey scripts for Gmail go, much useful information comes from Mihai Parparita, who wrote many Greasemonkey scripts for Gmail. Quoting from here:

As others have documented, Gmail receives data from the server in form of JavaScript snippets. Looking at the top of any conversation list’s source, we can see that the D() function that receives data in turns calls a function P() in the frame where all the JavaScript resides. Since all data must pass through this global P() function, we can use Greasemonkey to hook into it. This is similar to the trap patching way of extending Classic Mac OS. Specifically, the Greasemonkey script gets a hold of the current P() function and replaces it with a version that first records relevant data in an internal array, and then calls the original function (so that Gmail operations are not affected).

Clever. This same information is also documented at Greasespot wiki, with a few remarks on what different parameters to P() mean. Alas, it no longer works, because Gmail changed their functions around and renamed all of them, so there is no P() function anymore, and I can’t find what the new equivalent is, or if there is one.

Changes of October 2007
Gmail made certain changes in October 2007, including introducing a “newer version”, but also changing the “older version” that is still available: so it’s not really the older version. As far as Greasemonkey scripts go, another change was in January 2008, where they made all the Javascript load in a separate iframe. So “unsafeWindow” in a Greasemonkey script now refers to this iframe (which is the first frame, frame[0], in the window, and can also be got as top.js). So any scripts written in September 2007 or earlier are certainly useless now.

A lesson from all this is that Gmail will always be a moving target, and one must consider whether it’s worth chasing it.

Gmail’s Greasemonkey “API”:
Sometime in November 2007 or so, after the latest changes, Google even released a basic Greasemonkey API for Gmail, which lets you do a few things, like adding things to the pane at the left. It is too limited for what I need, but it works very well for what is meant for, and is also very well-documented, by Mark Pilgrim with his usual “Dive Into” excellence. It is comprehensive, accurate, well-illustrated and to-the-point, and great as documentation goes; it just happens that the API doesn’t provide what I need.

Some observations
Back to what I’m trying to do. Currently, the actions in the menu next to “Reply”, namely “Reply to all”, “Forward”, “Filter messages like this”, … “Show original” etc., do not actually appear in the DOM multiple times once attached to each message. Instead each of these actions corresponds to exactly one node (each) in the DOM, like these:

<div act="27" style="padding-left: 19px;" class="SAQJzb" id=":t6">Filter messages like this</div>
<div id=":t8" class="R10Zdd" act="29" style="padding-left: 19px;">Add to Contacts list</div>
<div id=":tc" class="SAQJzb" act="32" style="padding-left: 19px;">Show original</div>

etc. The IDs change, and the class name also seems to randomly change between “SAQJzb” and “R10Zdd”; the only constant between the action and the node is the “act” attribute. “Show original” is always act=32. So when you click on the down-arrow button next to Reply, this menu comes up, and when you click on something in the menu, it somehow uses the information about where this menu came up and what you clicked, to find out which message to act on.

This means that simply simulating a click on the node (initMouseEvent, etc…) does not work; we also have to somehow give it the information on what message to act on. How to do this is one thing I’m trying to find out.

The other way involves the fact that Gmail also has its own “ID” for each message. When you are looking at a thread (“conversation”) that contains a single message, it is the same as what is in the URL, e.g. if the URL is something like https://mail.google.com/mail/#inbox/11c177beaf88ffe6, Gmail’s ID of the message is 11c177beaf88ffe6. But when you’re looking at a thread containing more than one message, the ID in the URL is just that of any of the messages in the thread (usually the first one, but you can use the ID of a different message in the URL and it will show the same thread). And when you click on the “Show original” link, the URL is something like https://mail.google.com/mail/?ui=2&ik=1234567890&view=om&th=11c177beaf88ffe6 where 1234567890 is a constant (probably depending on the user) and “om” probably stands for “original message”, and the “th” parameter is the ID of the message. So if I can somehow find a way of getting the ID of messages (like the trap-patching P() method, except that it should work for the current version), then it is possible to get the Message-ID headers of messages too.

Neither has worked out yet, but I’m trying…
(And I have more to say, but will post when things actually work.)

Have you been high today?

If not, you need to see this (besides needing a bun to bite Benny Lava): [Make sure you read the subtitles before, not after the corresponding sounds.]

Where Have I Seen…

I have, for a long time, dreamed of a Firefox extension that would do this. Now, finally, empowered by Greasemonkey, I wrote one myself. (Technically, a Greasemonkey user script is not a Firefox extension, but it can be easily converted into one, and a Greasemonkey script is the Right Thing anyway.)

What it does: on any IMDB cast listing, show for each actor what other movies you have seen him/her in. Here’s a screenshot:

Initial version, not yet a Greasemonkey script [Batman Begins]

Initial version, after feeding it a more complete list of movies I’ve seen.

Update: It now runs in the browser entirely. Did you know Roy Batty was in Sin City?

Clearly there is a lot of work left to do, but it took less than half a day to learn Greasemonkey (and parts of JavaScript) and write it, and it’s already usable! By me, that is.
[The following was true at the time of writing but is no longer true.] I don’t know if anyone else would be interested in this, but it currently won’t work anyway except when running on my laptop. This is its crazy “design”: on any IMDB page with a cast listing, it first looks for each actor on the page, and extracts their ID. (Reasonable so far.) To find their other movies, it then makes a xmlhttp request to a PHP script running on my laptop, which then calls a Python script and returns its raw output inside ‘pre’ tags. Now you know. The reason for this nonsense is that there was no JavaScript API/library for IMDB while there was one for Python, so it was really easier to just use the latter, and the only way available of interacting with the “outside world” from a Greasemonkey script is through xmlhttp requests, and…
Anyway it’s not all that hard to parse each actor’s “other movies” through JavaScript myself, if that’s all I’m doing, so I might get to that eventually. (I also considered keeping IMDB’s data locally and parsing the text files, but they’re huge and not very well-formatted: No IDs, for example.)

It’s currently named “WHIS”, can you think of a better name? :)

Update: It’s now a full-fledged Greasemonkey script, and is up on userscripts.org.

JavaScript as a programming language

JavaScript has a bad reputation. It does have a few warts, but at its core it’s an okay language, and much of the ire ought to be directed at its users instead: amateur programmers with poor taste. JavaScript can’t be blamed for making it easy for them to implement their ill-considered ideas, but it often is. It is also significant that JavaScript has a somewhat unique relationship with its “users”, in the following sense. With most programming languages, one doesn’t know one is using the language unless one is programming in it. But think about what it means to have JavaScript enabled on your browser: you are basically inviting every random website to run code on your browser, and while your browser happily gives up control over all its aspects to the random code, it gives you no control over what code runs! Is it any surprise that a lot of people simply hate JavaScript for what it makes possible? The only surprising thing is that it took so long for control to be returned to the user of the browser, in the form of Greasemonkey. “Client-side” is a misnomer when only the computation happens on the client side but everything is decided by the server.