The Lumber Room

"Consign them to dust and damp by way of preserving them"

Archive for the ‘compknow’ Category

Converting a data URL (aka data URI) to an image on the commandline (Mac OS X)

with 2 comments

This is trivial, but was awfully hard to find via Google Search. Eventually had to give up and actually think about it. :-)

So, a data-URI looks something like the following:

data:image/png;base64,[and a stream of base64 characters here]

The part after the comma is literally the contents of the file (image or whatever), encoded in base64, so all you need to do is run base64 --decode on that part.

For example, with the whole data URL copied to the clipboard, I can do:

pbpaste | sed -e 's#data:image/png;base64,##' | base64 --decode > out.png

to get it into a png file.

Written by S

Sun, 2015-09-27 at 19:58:17

Posted in compknow

Using Stellarium to make an animation / video

leave a comment »

(I don’t have a solution yet.)

I just wanted to show what the sky looks like over the course of a week.

On a Mac with Stellarium installed, I ran the following

/Applications/Stellarium.app/Contents/MacOS/stellarium --startup-script stellarium.ssc

with the following stellarium.ssc:

// -*- mode: javascript -*-
core.clear('natural');  // "atmosphere, landscape, no lines, labels or markers"
core.wait(5);
core.setObserverLocation('Ujjain, India');
core.setDate('1986-08-15T05:30:00', 'utc');
core.wait(5);
for (var i = 0; i < 2 * 24 * 7; i += 1) {
    core.setDate('+30 minutes');
    core.wait(0.5);
    core.screenshot('uj');
    core.wait(0.5);
}
core.wait(10);
core.quitStellarium();

It took a while (some 10–15 minutes) and created those 336 images in ~/Pictures/Stellarium/uj*, occupying a total size of about 550 MB. This seems a start, but Imagemagick etc. seem to choke on creating a GIF from such large data.

Giving up for now; would like to come back in future and figure out something better, that results in a smaller GIF.

Written by S

Mon, 2015-09-14 at 20:10:10

Typing Kannada on Mac OS X

with 26 comments

(Thanks to this and this.)

Turns out it’s very easy, and we can basically use the same input method (UIM) as in Linux.

  1. Get MacUIM from its website
  2. Install it.
  3. Go to System Preferences -> Language & Text -> Input Sources System Preferences -> Keyboard -> Input Sources, and turn on MacUIM. (Click + and add “MacUIM (Roman)” under English.) Tick “Show Input menu in menu bar” too. Screen Shot 2013-04-07 at 12.33.19 AM
  4. I now have three input methods: US, EasyIAST (see earlier post), and MacUIM (Roman).
  5. Go to System Preferences -> MacUIM -> General, and in Input method, choose m17n-kn-itrans
  6. Go to System Preferences -> MacUIM -> Helper, tick “Use Helper-Applet”, and in the list at the right, tick m17n-kn-itrans.
    Screen Shot 2013-04-07 at 5.45.57 PM
  7. [Just for me] I have some changes to kn-itrans.mim, to make it closer to HK (and remove nonsense like “RRi” or whatnot just to type ಋ): download this file kn-itrans.mim, and remove the pdf extension. It goes into /Library/M17NLib/share/m17n/kn-itrans.mim

Written by S

Sun, 2013-04-07 at 01:14:35

Posted in compknow

My Git personal reference

with 9 comments

Various git things I’ve had to look up from time to time.

(Always, while doing anything dangerous, have a gitk window open. Look, don’t guess. (Haven’t used gitk in a while.) And if you’re sharing your repository publicly, you can forget about most of the below.)

Git reset

What git reset <commit> does:
* Reset HEAD to the given commit
* (If not --soft) copy this new HEAD to the index
* (If --hard) copy contents of index to working dir

So, for example:
* To “undo commit”: git reset HEAD~1
This resets HEAD to HEAD~1, without copying this new HEAD to the index. So it’s as if you didn’t make the commit. You can go and commit in another branch if you want.

Copy commits from another repository

# 1. Add the other repo as a remote
git remote add other_repo_nickname <other repo's path/url>
# 2. Fetch its data. (pull = fetch + merge, so we want only fetch, not pull)
git fetch other_repo_nickname
# 3. The rest should be familiar
git cherry-pick <commit>

Note that you don’t need to specify the remote’s name for cherry-picking. Once you have fetched, all commits, even those originally from the other repo, can be identified just by hash. (If you want to refer to commit by branch, then you can identify it with “other_repo_nickname/branch_name”.)

Swap commits (reorder top two commits)

 git rebase -i HEAD~2 

and in your editor, reorder the two “pick” lines. (See here.)

Recovering commits deleted with reset --hard

In general, these are garbage-collected after 30 days (or when you run git prune or git gc), so you shouldn’t use reset --hard at least without doing a stash first.
If the garbage-collection hasn’t happened yet, get the sha1 hash of the commit with

 git reflog 

then make sure the commit is what you want with

 git show sha1 

and get the commit back with

 git cherry-pick sha1 

(or rebase or merge instead of cherry-pick, if that’s what you want.)

Squashing commits together, to keep your history clean

Use rebase. To squash the last n commits into one, do

 git rebase -i HEAD~n 

and change all “pick”s except the first one to “squash” (or “s”). See here and here.

Delete a specific commit

Use rebase -i, again.

 git rebase -i <commit>~1 

Delete the line for the commit you want deleted. See here and here.

Amend a specific older commit

This is tricky, and I don’t think I’ve seen it anywhere, especially for the case where there are branches that depend on it.
My solution: Find the first branch X that’s downstream from (= later than) it. Keep track of the whole tree downstream from X (take a screenshot if you must); you’ll need it. Checkout X, and do “rebase -i HEAD~[large enough number to cover the commit you want to amend]”. In the editor that pops up, keep all “pick” lines, changing only that one line you want to amend to “edit”. Save and close. Now git has stopped, allowing you to amend that commit. Edit the file. Do “commit –amend” (don’t forget to add all files you want included in that commit!). Do “rebase –continue”. You’re back at X now. Now for the first branch Y that was downstream from X, checkout Y, do “git rebase X”, and recurse on Y.
This doesn’t seem work (gets into rebase conflicts). Need to try again, and ask.

Written by S

Tue, 2011-11-29 at 10:06:52

Posted in compknow

Tagged with

Getting back non-monospaced font in WordPress’s HTML editor

leave a comment »

So apparently some farsighted folks over at WordPress decided recently (see screenshots) that everyone who uses the HTML editor is using it to write code, rather than simply because the unpredictable “Visual” editor sucks so much. If you use WordPress, don’t like this change and would like to go back to using more normal fonts, (you can go complain at the appropriate places, or) open either Firebug console, or in Google Chrome go to View → Developer → Developer Tools and choose the console, and type

document.getElementById('content').style.cssText += "font-family: sans-serif;"

or whatever it is that you want. Making this a Greasemonkey/Stylish/whatever extension is left for others; I just want something quick to fix this annoyance.

Monospaced is fine for code, but typical monospaced fonts lack so many non-ASCII characters that all the glyph substitution makes it really ugly here.

These are (not) a few of my favourite fonts: Medley by WordPress

Edit: Looking around two days later, you can see complaints (I guess… I haven’t read them) here, here, here, etc., and the userscript here.

Written by S

Sun, 2011-06-05 at 04:35:52

Posted in compknow

Making audio louder with Audacity

with one comment

(Tried with Audacity 1.3.12 beta.)

  1. Open the file in Audacity. Go to Effects → Amplify.
  2. The amplification set is already the maximum possible without clipping. Don’t change anything, just click OK

This makes the file as loud as possible without clipping: without the loudest parts of the “signal” getting lost. If the result is not loud enough, the problem is not with the loudest parts (they are already as loud as they can be), but with the softer parts. So you need a transformation that makes the soft parts louder while keeping the loud parts the same. This is Dynamic range compression: the dynamic range (difference between softest and loudest parts) is compressed.

So, after trying “Amplify”,

  1. Download “Chris’s dynamic compressor” from here (direct link).
  2. Save the file compress.ny in /Applications/Audacity/plug-ins
  3. In Audacity, go to “Effects → Compress dynamics…” (or perhaps it’s called “Compress &dynamics…”)
  4. The first control (“Compress ratio”) is the main one. Or just leave it as it is. Click OK.
  5. If still not loud enough, go back and increase Compress ratio. Of course, increasing it means decreasing the dynamic range — increase it too much and the parts meant to be soft will be no softer than the rest.

Written by S

Sat, 2011-05-21 at 04:57:44

Posted in compknow

Converting old PS files (generated with LaTeX) to searchable PDFs

with 8 comments

pkfix fasc3a.ps fasc3a-fix.ps
ps2pdf fasc3a-fix.ps

A common (or at least, more common than it should be) scenario: you find a PostScript file of some paper, clearly written in (La)TeX, but which looks blurry on screen and you cannot copy any text. Converting to PDF with, say, ps2pdf does not help either. You curse the .ps format, and put up with the blurriness or print it out (where it looks fine) to read it.

Turns out it doesn’t have to be this way. The problem is that the PS file is using bitmap fonts, but assuming you have the scalable (Type 1) versions of those same fonts on your system, you can convert the fonts! There’s a script called pkfix, distributed with TeX Live, which will take a ps file that uses bitmap fonts and try to convert it to use scalable fonts. Just run it as

pkfix inputfile.ps outputfile.ps

This should produce a PS which isn’t blurry and is searchable, but if you prefer PDF, the usual way will work

ps2pdf outputfile.ps

or on Mac OS X if you don’t have ps2pdf for some reason, o inputfile.ps -a macps2pdf where macps2pdf comes with MacGhostView.

If the file is very old (generated with dvips from before 1996) and pkfix doesn’t work, there’s a further script called pkfix-helper that may make the file appropriate for pkfix.

BTW, if it’s your own files that are coming out blurry, something is wrong with your setup. Just install the package cm-super from CTAN—sudo tlmgr install cm-super or whatever—and no other change is needed. Or you can use the lmodern fonts with \usepackage{lmodern}, but that shouldn’t be necessary.

Written by S

Thu, 2011-05-05 at 11:42:30

Posted in compknow

Tagged with , ,

Matplotlib tutorial

The “standard” way to plot data used to be gnuplot, but it’s time to start using matplotlib which looks better and easier to use. For one thing, it’s a Python library, and you have the full power of a programming language available when you’re plotting, and a full-featured plotting library available when you’re programming, which is very convenient: I no longer find it necessary to use a horrible combination of gnuplot, programs, and shell scripts. For another, it’s free software, unlike gnuplot which is (unrelated to GNU and) distributed under a (slightly) restrictive license. Also, the plots look great. (Plus, I’m told that matplotlib will be familiar to MATLAB users, but as an assiduous non-user of MATLAB, I can’t comment on that.)

matplotlib is simple to start using, but its documentation makes this fact far from clear. The documentation is fine if you’re already an expert and want to draw dolphins swimming in glass bubbles, but it’s barely useful to a complete beginner. So what follows is a short tutorial. After this, you (that is, I) should be able to look at the gallery and get useful information, and perhaps even the documentation will make a bit of sense. (If matplotlib isn’t already present on your system, the easiest way to install it, if you have enough bandwidth and disk space to download a couple of gigabytes and you’re associated with an academic installation, is to get the Enthought Python distribution. Otherwise, sudo easy_install matplotlib should do it.)

The most common thing you want to do is plot a bunch of (x,y) values:

import matplotlib.pyplot as plot
xs = [2, 3, 5, 7, 11]
ys = [4, 9, 5, 9, 1]
plot.plot(xs, ys)
plot.savefig("squaremod10.png")

p^2 mod 10
Or, for a less arbitrary example:

import matplotlib.pyplot as plot
import math

xs = [0.01*x for x in range(1000)] #That's 0 to 10 in steps of 0.01
ys = [math.sin(x) for x in xs]
plot.plot(xs, ys)
plot.savefig("sin.png")

That is all. And you have a nice-looking sine curve:

If you want to plot two curves, you do it the natural way:

import matplotlib.pyplot as plot
import math

xs = [0.01*x for x in range(1000)]
ys = [math.sin(x) for x in xs]
zs = [math.cos(x) for x in xs]

plot.plot(xs, ys)
plot.plot(xs, zs)

plot.savefig("sin-2.png")

It automatically chooses a different colour for the second curve.

Perhaps you don’t find it so nice-looking. Maybe you want the y-axis to have the same scale as the x-axis. Maybe you want to label the x and y axes. Maybe you want to title the plot. (“The curves sin x and cos x”?) Maybe you want the sine curve to be red for some reason, and the cosine curve to be dotted. And have a legend for which curve is which. All these can be done. In that order —

import matplotlib.pyplot as plot
import math

xs = [0.01*x for x in range(1000)]
ys = [math.sin(x) for x in xs]
zs = [math.cos(x) for x in xs]

plot.axis('equal')
plot.xlabel('$x$')
plot.ylabel('$y$')
plot.title(r'The curves $\sin x$ and $\cos x$')
plot.plot(xs, ys, label=r'$\sin$', color='red')
plot.plot(xs, zs, ':', label=r'$\cos$')
plot.legend(loc='upper right')

plot.savefig("sin-cos-two.png")

Observe that we specified the plot style as “:”, for dotted. We can also use ‘o’ for circles, ‘^’ for triangles, and so on. We can also prefix a colour, e.g. ‘bo’ for blue circles, ‘rx’ for red crosses and so on (the default is ‘b-‘, as we saw above). See the documentation for plot.

Also, Matplotlib has some TeX support. :-) It parses a subset of TeX syntax (be sure to use r before the string, so that you don’t have to escape backslashes), and you can also use

plot.rc('text', usetex=True)

to make it actually use (La)TeX to generate text.

My only major annoyance with the default settings is that the y-axis label is vertical, so one has to tilt one’s head to read it. This is easy to fix:

plot.ylabel('$y$', rotation='horizontal')

You can also place text at a certain position, or turn a grid on:

plot.text(math.pi/2, 1, 'top')
plot.grid(True)

You can also annotate text with arrows.

You can have multiple figures in the same image, but it’s all rather stateful (or complicated) and so far I haven’t needed it.

Instead of saving the figure, you can use
plot.show()
and get an interactive window in which you can zoom and so on, but I find it more convenient to just save to image.

You can customize defaults in the matplotlibrc file — fonts, line widths, colours, resolution…

Other plots: Matplotlib can also do histographs, erorr bars, scatter plots, log plots, polarplots, bar charts and yes, pie charts, but these don’t seem to be well-documented.

Animation: Don’t know yet, but it seems that if you want to make a “movie”, the recommended way is to save a bunch of png images and use mencoder/ImageMagick on them.

That takes care of the most common things you (that is, I) might want to do. The details are in the documentation. (Also see: cookbook, screenshots, gallery.)

Edit: If you have a file which is just several lines of (x,y) values in two columns (e.g. one you may have been using as input to gnuplot), this function may help:

def x_and_y(filename):
    xs = []; ys = []
    for l in open(filename).readlines():
        x, y = [int(s) for s in l.split()]
        xs.append(x)
        ys.append(y)
    return xs, ys

Edit [2010-05-10]: To plot values indexed by dates, use the following.

    fig = plot.figure(figsize=(80,10))
    plot.plot_date(dates, values, '-', marker='.', label='something')
    fig.autofmt_xdate()

The first line is because I don’t know how else to set the figure size. The last is so that the dates are rotated and made sparser, so as to not overlap.

More generally, Matplotlib seems to have a model of figures inside plots and axes inside them and so on; it would be good to understand this model.

Edit [2013-03-15]: Found another good-looking tutorial here: http://www.loria.fr/~rougier/teaching/matplotlib/.

[2013-04-08: Comments closed because this post was getting a lot of spam; sorry.]

Written by S

Sun, 2010-03-07 at 23:41:37

Flashing the screen on Mac OS X

with 2 comments

Here’s one way. There’s a C program to adjust the screen’s brightness written by Nicholas Riley, also available from this blog post by Matt (Danger) West. Get it. The rest is obvious. For instance, here’s a Python script, which should have probably been written in Perl:

import os, re, time

s = os.popen('./brightness -l').read()
ob = re.findall('brightness (\d.\d+)', s)[0]

w = 0.2
for i in range(10):
    os.system('./brightness 0'); time.sleep(w)
    os.system('./brightness 1'); time.sleep(w)
    if(i==4): os.system("say beep")

os.system('./brightness ' + ob)

Tune parameters to avoid epileptic seizures.

Read the rest of this entry »

Written by S

Mon, 2009-08-17 at 16:07:05

Posted in compknow

Tagged with , , , ,

Firebug “console is undefined”

with 2 comments

If you’re using Firebug and don’t want to bother removing or commenting out all the console.log debug messages for users who aren’t, put this at the top:

if(typeof(console) === "undefined" || typeof(console.log) === "undefined") 
    var console = { log: function() { } };

This reminds me of the trick I use for C/C++ of putting debugging statements inside a macro:

D(cerr<<"The number is "<<n<<endl;);

where the macro D is defined as

#define D(A) A

when you want the debugging code to run, and

#define D(A) 

when you don’t. :-)

(The final semicolon after the D() is for Emacs to indent it reasonably.)

Update: Of course, the above are just hacky hacks to save typing. The “right” way for conditional C code is usually to use #ifdef DEBUG and the like, and the right way around the Firebug problem is to use something more general like this code at metamalcolm.

Written by S

Wed, 2009-08-05 at 14:32:15

Extreme image compression: the Twitter challenge

with 3 comments

If a picture is worth a 1000 words, how much of a picture can you fit in 140 characters?

Mario Klingemann (Quasimondo on Flickr) had a fascinating — call it crazy if you like — idea: can you encode an image such that it can be sent as a single Twitter message (“tweet”)? Twitter allows 140 characters, which seems like nothing. It’s pretty much guaranteed that you’ll be able to get nothing meaningful out of so few bits, right?

Well, he came up with this, using a bunch of clever tricks: using the full Unicode range for “characters” (Chinese, etc.) to squeeze a few more bits’ worth, representing colours as blends of just 8 colours (3 bits!), and arriving at a Voronoi triangulation through a genetic algorithm:

© Quasimondo:Flickr/CC-BY-NC (210 bytes?)

“Mona Tweeta” © Quasimondo:Flickr/CC-BY-NC (210 bytes?)

The one on the right is the real Mona Lisa, and the left one is what fits in 140 characters, specifically the message: “圑嘌婂搒孵怤實恄幖戰怴搝愩娻屗奊唀唭嚟帧啜徠山峔巰喜圂嗊埯廇嗕患嚵幇墥彫壛嶂壋悟声喿墰廚埽崙嫖嘵奰恛嬂啷婕媸姴嚥娐嗪嫤圣峈嬻尤囮愰啴屽嶍屽嶰寂喿嶐唥帑尸庠啞彐啯廂喪帄嗆怠嗙开唅恰唦慼啥憛幮悐喆悠喚忐嗳惐唔戠啹媊婼捐啸抃岖嗅怲幀嗈拀唹坭嵄彠喺悠單囏庰抂唋岰媮岬夣宐彋媀恦啼彐壔姩宔嬀”

This is pretty impressive, you’d think, for 140 characters. But it gets better. Brian Campbell started a contest on Stack Overflow, and some brilliant approaches turned up.

Boojum wrote a nanocrunch.cpp, based on fractal compression, which can do this (on the left is the original, for comparsion):

Boojum-origby Boojum [490 bytes] by Boojum (490 bytes)

Sam Hocevar wrote img2twit, which segments the image into square cells and tries to randomly assign points and colours to them until something is close. It can do this:

img2twit-origimg2twit-dec
img2twit by Sam Hocevar (250 bytes?)
You can watch a movie of the image evolving; it’s pretty cool!

There were also attempts at converting the image to a vector format and encoding that instead. Needless to say, it works well for vector-like images:
so-logoso-logo-decoded (almost perfect!)

but it’s hard to even convert some images to vector form:
autotrace by autotrace (before compression!)

Finally, this is how Dennis Lee’s record-holding “optimizing general-purpose losy image codec” DLI does:

DLI-comp

Comparison: JPEG at 536 bytes, img2twit at 534 bytes, DLI at 534 bytes

Or if you want to be fair and compare at 250 bytes, here’s img2twit and DLI:

img2twit at ~250 bytes, DLI at 243 bytes

img2twit at ~250 bytes, DLI at 243 bytes

Amazing!

For silly amusement, you can read a liberal translation of the original message, or the Reddit thread with ASCII porn.

Disclaimer: I did not participate in any of this, and I know nothing about image compression, so no doubt there are errors in the above. Please point them out. All images are copyright the respective owners, and the quote in the first line is by Brian Campbell on Stack Overflow.

Written by S

Sun, 2009-05-31 at 13:37:44

mplayer: changing speed without changing pitch (avoiding the chipmunk effect)

with one comment

In mplayer, you can change the playback speed with [ or ], but that probably changes the pitch as well (naturally). Can be amusing the first time, but not after you realise that it is actually possible to do something sophisticated to avoid this. (Wikipedia calls it Audio timescale-pitch modification) Many other media players (including VLC and even Windows Media Player(?)) can do this automatically; here’s how to do it in mplayer.

Short answer:

Start mplayer as mplayer -af scaletempo
That’s it. The catch is that you need to get an mplayer which has the scaletempo filter, and we know how much the mplayer project loves making releases. (It’s not in Ubuntu at the time of writing.)

So, either

Get such an mplayer
e.g the deb from Sourceforge (here),

or

Start mplayer as mplayer -speed 1.5 -af ladspa=tap_pitch:tap_pitch:0:-33:-90:0 foo.avi

Seems even the latter might require installing the ladspa plugins.

For more on all this, see:

  1. blog comments (with patches) at Pitch-Correct Play Speed with MPlayer
  2. Change MPlayer Playback Speed
  3. Mplayer FAQ: “How do i change mplayer speed but keep pitch the same?”

Written by S

Fri, 2009-05-29 at 21:54:31

Posted in compknow

Tagged with

Drag-and-drop on web pages

with 2 comments

Many web interfaces (like the WordPress dashboard in which I type this) allow you to drag and drop sections of the page to rearrange them. I think I first saw it when the personalised Google homepage (google.com/ig) came out, but I don’t know if it was the first. Here are some libraries that seem to help you

So it seems that all JavaScript frameworks have it. It also seems that JQuery has “won”.

But it seems YUI’s “Reordering a List” example is the closest match to what we actually want, but it seems to require too many lines of code.

Some of the rest seem even worse: unless you are actually using the library for a lot of things, it is simply too much code to just drop into a bare-bones web page you’ve been writing from scratch.

Written by S

Sun, 2008-09-14 at 22:11:37

Posted in compknow, unfinished

Tagged with , ,

Reverse-engineering Gmail: Initial remarks

with 11 comments

For the last week and a bit, I have been trying to do a particular something with Gmail. (Specifically, get at the Message-ID headers of messages.) This has been mostly a failure, but that’s not so surprising, as I had little experience with “all this web stuff”: JavaScript, AJAX, DOM, browser incompatibilities, Firebug, Greasemonkey… round up the usual buzzwords. I have learnt a bit, though, and thought it might help others starting in a similar situation. (And there’s also the hope that someone might actually find this and help me!)

The story so far
Gmail was launched in April 2004. Since then, it has been through many changes, the latest around October 2007 when there came to our inboxes a “Newer version”, also sometimes called “Gmail 2”. (Note that officially Gmail is still in Beta; it hasn’t even released a 1.0!)
When Gmail was released the set of practices that go by the name of “AJAX” was still new and unfamiliar; it has been refined and better-understood since. (And it turns out to require neither asynchrony nor JavaScript nor XML.)

Johnvey Hwang reverse-engineered much of Gmail’s original version, and even made a “Gmail API” out of it. It no longer works of course, and the site is often down too, but it’s available on the Wayback Machine and the section documenting “the Gmail engine and protocol” is still worth a read, if only for its glimpse into the labyrinthine ways in which Ajax applications can work. He turned it (in May 2005) into a SourceForge project (“Gmail API”), last updated June 2005, and the associated Google Group (” Gmail Agent API”) is also largely defunct and indicates that the API, or whatever came of it, has not been working since the changes in October 2007, at any rate.

My goal
At this point, I might as well reveal what I want to do: I want to make it easy to get the “Message-ID:” header of messages in Gmail. (I like to read email in Gmail but not to send, so one way to reply to a specific message would be to get the Message-ID and ask my other mail client to reply to the message with that message-ID.) In the current interface, the (only) way of getting it is to click on the pulldown menu next to “Reply”, and click on “Show original”. This will open up a page that contains the raw text of the message with all its headers, and “Message-ID:” is always one of them. Since I use Firefox, I’ve been trying to make this easier with a Greasemonkey script.

Trap-patching the P() function
As Greasemonkey scripts for Gmail go, much useful information comes from Mihai Parparita, who wrote many Greasemonkey scripts for Gmail. Quoting from here:

As others have documented, Gmail receives data from the server in form of JavaScript snippets. Looking at the top of any conversation list’s source, we can see that the D() function that receives data in turns calls a function P() in the frame where all the JavaScript resides. Since all data must pass through this global P() function, we can use Greasemonkey to hook into it. This is similar to the trap patching way of extending Classic Mac OS. Specifically, the Greasemonkey script gets a hold of the current P() function and replaces it with a version that first records relevant data in an internal array, and then calls the original function (so that Gmail operations are not affected).

Clever. This same information is also documented at Greasespot wiki, with a few remarks on what different parameters to P() mean. Alas, it no longer works, because Gmail changed their functions around and renamed all of them, so there is no P() function anymore, and I can’t find what the new equivalent is, or if there is one.

Changes of October 2007
Gmail made certain changes in October 2007, including introducing a “newer version”, but also changing the “older version” that is still available: so it’s not really the older version. As far as Greasemonkey scripts go, another change was in January 2008, where they made all the Javascript load in a separate iframe. So “unsafeWindow” in a Greasemonkey script now refers to this iframe (which is the first frame, frame[0], in the window, and can also be got as top.js). So any scripts written in September 2007 or earlier are certainly useless now.

A lesson from all this is that Gmail will always be a moving target, and one must consider whether it’s worth chasing it.

Gmail’s Greasemonkey “API”:
Sometime in November 2007 or so, after the latest changes, Google even released a basic Greasemonkey API for Gmail, which lets you do a few things, like adding things to the pane at the left. It is too limited for what I need, but it works very well for what is meant for, and is also very well-documented, by Mark Pilgrim with his usual “Dive Into” excellence. It is comprehensive, accurate, well-illustrated and to-the-point, and great as documentation goes; it just happens that the API doesn’t provide what I need.

Some observations
Back to what I’m trying to do. Currently, the actions in the menu next to “Reply”, namely “Reply to all”, “Forward”, “Filter messages like this”, … “Show original” etc., do not actually appear in the DOM multiple times once attached to each message. Instead each of these actions corresponds to exactly one node (each) in the DOM, like these:

<div act="27" style="padding-left: 19px;" class="SAQJzb" id=":t6">Filter messages like this</div>
<div id=":t8" class="R10Zdd" act="29" style="padding-left: 19px;">Add to Contacts list</div>
<div id=":tc" class="SAQJzb" act="32" style="padding-left: 19px;">Show original</div>

etc. The IDs change, and the class name also seems to randomly change between “SAQJzb” and “R10Zdd”; the only constant between the action and the node is the “act” attribute. “Show original” is always act=32. So when you click on the down-arrow button next to Reply, this menu comes up, and when you click on something in the menu, it somehow uses the information about where this menu came up and what you clicked, to find out which message to act on.

This means that simply simulating a click on the node (initMouseEvent, etc…) does not work; we also have to somehow give it the information on what message to act on. How to do this is one thing I’m trying to find out.

The other way involves the fact that Gmail also has its own “ID” for each message. When you are looking at a thread (“conversation”) that contains a single message, it is the same as what is in the URL, e.g. if the URL is something like https://mail.google.com/mail/#inbox/11c177beaf88ffe6, Gmail’s ID of the message is 11c177beaf88ffe6. But when you’re looking at a thread containing more than one message, the ID in the URL is just that of any of the messages in the thread (usually the first one, but you can use the ID of a different message in the URL and it will show the same thread). And when you click on the “Show original” link, the URL is something like https://mail.google.com/mail/?ui=2&ik=1234567890&view=om&th=11c177beaf88ffe6 where 1234567890 is a constant (probably depending on the user) and “om” probably stands for “original message”, and the “th” parameter is the ID of the message. So if I can somehow find a way of getting the ID of messages (like the trap-patching P() method, except that it should work for the current version), then it is possible to get the Message-ID headers of messages too.

Neither has worked out yet, but I’m trying…
(And I have more to say, but will post when things actually work.)

Written by S

Sun, 2008-08-31 at 18:45:57

Multi-tty Emacs on Ubuntu

with 7 comments

“Multi-tty support” is a feature that is currently in Emacs CVS for what will eventually become Emacs 23. It lets different Emacs windows (“frames”) share the same session. This means that you’ll always have all your buffers, and Emacs starts faster. Emacs has had this “emacsclient” feature for a long time now, where you can edit new files in an Emacs instance that is running as “server”, but what is new is that you no longer have to hunt for where you had started Emacs first — you can simply edit in a new frame. For example, if you had originally started Emacs under X, and you log in remotely, you can reuse the running Emacs process by starting a new “frame” in the terminal. Instead of explaining better what it does, I’ll let you see for yourselves:

video by EnigmaCurry

Note that it is not necessary that all (or any) of them be in a terminal — you can have multiple GTK Emacs windows too. Also, note that every new window starts with the *scratch* buffer, not necessarily the same buffer as the one on top in other windows.

Installing:
Multi-tty Emacs used to be maintained separately (which is why older posts have complicated instructions), but it has been merged into the main Emacs CVS branch for a long time now. This means that it is available on Ubuntu by simply installing the emacs-snapshot package (this has been true since 2007-09-02). On other distributions/platforms, use whatever it takes to get Emacs from CVS.

Using:
To use it, start one Emacs session, and, if you don’t have (server-start) in your .emacs already, do M-x server-start. Once done, you can either leave this Emacs window hanging around, or hide it, or whatever.
Now to start subsequent Emacs instances, use emacsclient -c. The “-c” stands for “-­-create-frame”, and means that a new Emacs “frame” is started instead of using the old one. (Emacs calls windows frames and what it calls “windows” is closer to “frames” in a web browser.)
Use emacsclient -t if you want to start it in the terminal.
It appears that you can quit each individual instance the normal way with C-x C-c; it is not necessary to use C-x 5 0 (that’s delete-frame).

Further:
Some things you can do to make life easier.

  • Put (server-start) in your .emacs if you haven’t done so already. Michael Olson suggests using (unless (string-equal "root" (getenv "USER")) (server-start)), but I don’t recall having ever needed that.
  • Automatically start an Emacs session at login and hide it, so that it runs like a “daemon” and you can simply use emacsclient -c consistently and forget about it. You can use screen -d -m emacs -nw, or use the more lightweight dtach: dtach -n /tmp/emacs emacs -nw
  • Do the above with some “name” for your emacs server. You can give your server a name by using (setq server-name "asdf") before (server-start), then you can can connect to it with emacsclient -s asdf -c (or -t). I don’t entirely see the utility of this; it might be so that you don’t start extra servers by mistake and always use the same one. There are scripts to do this.
  • You could probably combine all the above by writing a wrapper that checks if a server exists, connects to it if it does, else starts one, etc., and use that wrapper as your default system editor. That is what the wrapper script “ec” at EnigmaCurry does.

It would be nice if Emacs made this the default, though: Starting Emacs would by default work as multi-tty and share session with whatever previous Emacsen had been started, unless one explicitly told it not to do so.
Why not?

Sources: EnigmaCurry with old installation instructions, the wrapper script and the video, Mwolson’s post

Written by S

Sun, 2008-05-11 at 18:23:03

Posted in compknow

Tagged with

Zsh/Bash startup files loading order (.bashrc, .zshrc etc.)

with 32 comments

If you have ever put something in a file like .bashrc and had it not work, or are confused by why there are so many different files — .bashrc, .bash_profile, .bash_login, .profile etc. — and what they do, this is for you.

The issue is that Bash sources from a different file based on what kind of shell it thinks it is in. For an “interactive non-login shell”, it reads .bashrc, but for an “interactive login shell” it reads from the first of .bash_profile, .bash_login and .profile (only). There is no sane reason why this should be so; it’s just historical. Follows in more detail.

For Bash, they work as follows. Read down the appropriate column. Executes A, then B, then C, etc. The B1, B2, B3 means it executes only the first of those files found.

+----------------+-----------+-----------+------+
|                |Interactive|Interactive|Script|
|                |login      |non-login  |      |
+----------------+-----------+-----------+------+
|/etc/profile    |   A       |           |      |
+----------------+-----------+-----------+------+
|/etc/bash.bashrc|           |    A      |      |
+----------------+-----------+-----------+------+
|~/.bashrc       |           |    B      |      |
+----------------+-----------+-----------+------+
|~/.bash_profile |   B1      |           |      |
+----------------+-----------+-----------+------+
|~/.bash_login   |   B2      |           |      |
+----------------+-----------+-----------+------+
|~/.profile      |   B3      |           |      |
+----------------+-----------+-----------+------+
|BASH_ENV        |           |           |  A   |
+----------------+-----------+-----------+------+
|                |           |           |      |
+----------------+-----------+-----------+------+
|                |           |           |      |
+----------------+-----------+-----------+------+
|~/.bash_logout  |    C      |           |      |
+----------------+-----------+-----------+------+

In more detail is this excellent flowchart from http://www.solipsys.co.uk/new/BashInitialisationFiles.html :

Typically, most users will encounter a login shell only if either:
* they logged in from a tty, not through a GUI
* they logged in remotely, such as through ssh.
If the shell was started any other way, such as through GNOME’s gnome-terminal or KDE’s konsole, then it is typically not a login shell — the login shell was what started GNOME or KDE behind your back when you logged in; things started anew are not login shells. New terminals or new screen windows you open are not login shells either. (Starting a new window in OS X’s Terminal.app seems to count as a login shell, though.)

So typically (or sooner or later), what you will encounter are non-login shells. So this case is what you should write your config files for. This means Read the rest of this entry »

Written by S

Sun, 2008-03-30 at 19:43:56

Posted in compknow

Tagged with , , ,

Poor man’s random number generators in C++

with 3 comments

There are random number libraries, Boost for example, but ugh. It needs you to create an abstract “generator” (representing a source of randomness), a “distribution” (like “uniform”, or “exponential”), a function that samples according to that distribution using that random source, and finally a call to that function.

Meanwhile, here are poor-man implementations, with sucky randomness etc, but shorter by far. (ll is long long, ld is long double)

//uniform in [a,b)
int runiform_int(int a, int b) {return a + ((b-a)*ll(rand()))/(ll(RAND_MAX)+1);}
ld runiform(ld a, ld b) { return a + (b-a)*rand()/ld(RAND_MAX+1.0); }
ld rexp(ld lambda) { return -logl(runiform(0,1))/lambda; } //Mean: 1/λ
ld rnormal(ld mean, ld std) { //Using the Box-Muller transform
  return mean + std*sqrtl(-2*logl(runiform(0,1)))*sinl(2*M_PI*runiform(0,1));
}

Written by S

Thu, 2008-03-13 at 05:58:17

Posted in compknow

Tagged with , , ,

Emacs auto-save

leave a comment »

[The code you’re looking for is at the end of the post :)]
Following up on my previous post on auto-save…


GNU Emacs is one of the oldest programs existing, and it has had an auto-save-mode for god knows how many decades now, but it doesn’t quite do what I want. Like this other user, I was annoyed that it saves to a different file (/dir/filename is saved to /dir/#filename# while the file is being edited), and still requires you to save the file when you quit Emacs or close the buffer, thus making it not auto-save at all, but merely a backup facility guarding against Emacs crashes.

Someone at Emacswiki’s AutoSave page suggested M-x run-with-idle-timer RET SECONDS RET save-buffer RET, but I tried that for a while and it is really not a good idea; it tries to save every buffer, even those not associated with files, like completion buffers. I considered trying to figure out how to make it run on only buffers associated with files, but then I realised that it would probably be better to modify the built-in auto-save mechanism do what I want, instead of rolling my own new one, because that was smart about things like this.

C-h f auto-save-mode is actually quite sparse compared to the usual Emacs documentation. Anyway, I figured out with a bit of looking in the source files that I could edit make-auto-save-file-name function to modify what filename it used for auto-saves. On a bit more thought, I realised that it is usually never necessary to edit Emacs functions; you can always use hooks (or, in rare cases, advice). So this works: (add-hook 'auto-save-hook 'save-buffer). Yeah, it will still be saving the file in the #filename#, but it will be saving it to your file too. Actually I then realised I don’t need that either. Doing M-x apropos-variable RET auto-save RET gives several variables including:

auto-save-file-name-transforms
  Variable: Transforms to apply to buffer file name before making auto-save file name.
auto-save-timeout
  Variable: *Number of seconds idle time before auto-save.
auto-save-visited-file-name
  Variable: Non-nil says auto-save a buffer in the file it is visiting, when practical.

and setting auto-save-visited-file-name to t seemed to do what I want. Until I actually tried to use it, and discovered that whoever added that feature probably hasn’t actually used it at all — it asks a “File has changed on buffer, really edit it? (y/n/r/h)” each time it auto-saves and you try to continue editing. You could turn on auto-revert-mode to make that problem go away, but I’m unsure if that’s a good idea: sometimes I overwrite files by mistake (gcc hello.c -o hello.c) and having Emacs not auto-revert is very helpful in those cases. So I’ll stick with my earlier idea, which works perfectly, after being fixed by Bryan Murdock:

(defun save-buffer-if-visiting-file (&optional args)
"Save the current buffer only if it is visiting a file"
(interactive)
(if (buffer-file-name)
(save-buffer args)))
(add-hook 'auto-save-hook 'save-buffer-if-visiting-file)

Bryan Murdock also posted it.

Written by S

Tue, 2008-01-22 at 22:05:17

Posted in compknow

Tagged with

Goodbye Adium: Pidgin on Mac OS X… with native GTK!

with 99 comments

So you’re on Mac OS X, and want to use Pidgin on it.

First: Why not use Adium? Adium is a Free multi-protocol IM client for Mac OS X that uses libpurple, the IM library that was developed as part of Pidgin. It has several popular features such as message styles, and uses the Cocoa API native to Mac OS X, and all round looks pretty. Being a native Cocoa application, it is more well-integrated with the Mac desktop than a GTK-using application like Pidgin can ever be.

On the other hand, if you, like me, have tried Adium and have reasons for finding it unusable, then it is possible to install Pidgin on OS X too. There are two ways of doing this:
[Edit: Note that this post is from 2007. Probably a lot has changed since then.]

  1. Install Pidgin to run inside X11. Any default distribution of Pidgin should build fine on this, as long as you have all the dependencies installed. However, using an X11 app on OS X is really like entering another universe entirely… it’s like having two entirely disjoint OSes that just happen to run simultaneously. It is cumbersome, and I would not recommend it unless you are already doing much of your work inside X11 for some reason.
  2. Install Pidgin to run on the Mac desktop directly (without X11), using native GTK+ for Mac OS X and some minor modifications to Pidgin. This is very simple to do and requires only one step, described below.

Pidgin welcome dialogAdding an account

Instructions:

Read the rest of this entry »

Written by S

Fri, 2007-12-07 at 07:57:07

Posted in compknow

Tagged with , , , , , , ,

Firefox: Subscribing to feeds using Google Reader

with 2 comments

If I select a feed URL (or “live bookmark” as Firefox likes to call it), and choose to “subscribe to this feed using: Google”, I am taken to a Google page that always shows two options, “Add to Google homepage” and “Add to Google Reader”. Is there a way of directly going to Google Reader each time?

Yes: Auto add to Google Reader, a Greasemonkey script.

It probably works; but I’m using an alpha release of Firefox 3 (“Gran Paradiso”), and Greasemonkey doesn’t work for me so I can’t be sure.

Written by S

Tue, 2007-11-13 at 14:42:16

Posted in compknow

Tagged with ,