Why the #AskObama tweet was garbled on screen

Oh come off of it. This happens everywhere on the web on probably something like 25% of websites. And it's NOT always the consuming program's fault: very often somebody upstream, e.g. the hosting company, the person that wrote the HTML, the source of an RSS feed being inserted into the page etc. etc. forgot to encode something the way somebody else expected, and you as the poor guy at the end of the chain gets a document with multiple encodings improperly embedded into it. Inevitably you have to make some bad decisions and not all corner cases are handled.

Somebody once reverse-engineered the state chart for how Internet Explorer handles documents with conflicting encoding declarations and I kid you not, it must have had >20 branches spanning a good few pages. Officially, the correct order of precedence is (http://www.w3.org/International/questions/qa-html-encoding-d...):

1. HTTP Content-Type header

2. byte-order mark (BOM)

3. XML declaration

4. meta element

5. link charset attribute

but that's not how every browser does it, because the W3C sort of declared that after things on the Real Internet (TM) had already gotten out of hand. I hate to resuscitate Joel posts but Unicode is not easy to implement right.

Because I wonder how difficult it would be to create a string that says something innocuous in UTF-8 (e.g., "When will you bring the troops home #AskObama") but in ASCII would read as something totally different, but legible (e.g., "the secret priests would take great Cthulhu from his tomb to revive His subjects and resume his rule of earth...")