Been playing 320kbps Stereo flawlessly afaik. The bitrate is really not much of an issue.
Browsers have a much harder time dealing with binary XHRs, huge typed arrays, huge strings, and a real fucking lot of arithmetic operations.
I think the most difficult part was the debugging. Jens Nockert, Matthias Georgi & I finally used node.js and carefully diff'd the output from minimad.c (the canonical libmad example) and jsmad.
However, it quickly became obvious that comparing exact values wasn't going to cut it. Listening was also almost useless (try to debug when all you have is SCRRRCHHHHHH vs CRRRRRRKKKKKK). So we plotted graphs. With gnuplot. It was a matrix-like experience but quickly allowed us to see what was wrong.
Profiling the code via Firebug made us realize that most of the time (~28%) was spent in the synthetizer (synth.js). The I/O code is particularly expensive too, it's a mix of Uint8Arrays and javascript Strings, depending on whether you're using the local file version (filestream.js) or the http streaming version (ajaxstream.js)
One of the funniest part was when I had to code 36-bit precision integer routines in Javascript such as shifting, OR-ing, AND-ing.. I had to separate the numbers in a low word and a high word and then work with floats (which have a 54-bit mantissa in JS).
Really quite an experience!