Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why the #AskObama tweet was garbled on screen

hanselman.com

Post reply on HN
Print view View original

21–30 of 78 posts

Re: Why the #AskObama tweet was garbled on screen

#21

Post by StavrosK »

pavel_lishin wrote: post #14

Did they see the encoding mistake before they showed it? Because I wonder how difficult it would be to create a string that says something innocuous in UTF-8 (e.g., "When will you bring the troops home #AskObama") but in ASCII would read as something totally different, but legible (e.g., "the secret priests would take great Cthulhu from his tomb to revive His subjects and resume his rule of earth...")

I imagine quite difficult, as each character triplet when decoded with Windows-1252 would have to be one letter in Unicode, and those would have to actually form words. You'd be restricted to maybe 30 triplets.

Re: Why the #AskObama tweet was garbled on screen

#23

Post by pak »

eropple wrote: post #13
pak wrote: post #12

"This is SUCH a classic sloppy programmer mistake that I'm disappointed" Oh come off of it. This happens everywhere on the web on probably something like 25% of websites. And it's NOT always the consuming program's fault: very often somebody upstream, e.g. the hosting company, the person that wrote the HTML, the source of an RSS feed being inserted into the page etc. etc. forgot to encode something the way somebody e…

Twitter said that the message in question is UTF-8. The message recipient decoded the message with something that is not UTF-8. That's a sloppy mistake and should without doubt have been caught in testing.

How do you (or the OP) know the implementation details of the consumption path for display? The author assumes it was the direct JSON feed from Twitter. That's overly simplistic and betrays his inability to understand how these sort of things are put together. This is a public, real-time event; more likely than not tweets were filtered through one or two other components to separate the interesting material from the noise. You have no idea what those intermediary components were, who set them up, and if they were all decoding/reencoding correctly. Perhaps they were going through other Twitter users' clients, which could have contributed to the bug.

Edit: yes, there were definitely people curating the questions, so it could not have been as simple as pulling one JSON feed. http://www.theatlantic.com/politics/archive/2011/07/how-obam... https://twitter.com/#!/townhall/july-6-curators

Re: Why the #AskObama tweet was garbled on screen

#24

Post by samfoo »

luigi wrote: post #15

What modern software stack uses Extended ASCII as its default encoding? The last time I dealt with this problem, it was in 2005 or 2006 and I was working with PHP.

My guess would be that it's a flash app. Flash is (or at least used to be) horrible at internationalization issues like this.

Re: Why the #AskObama tweet was garbled on screen

#25

Post by ori_b »

pak wrote: post #23
eropple wrote: post #13

Earlier quoted context omitted.

Twitter said that the message in question is UTF-8. The message recipient decoded the message with something that is not UTF-8. That's a sloppy mistake and should without doubt have been caught in testing.

How do you (or the OP) know the implementation details of the consumption path for display? The author assumes it was the direct JSON feed from Twitter. That's overly simplistic and betrays his inability to understand how these sort of things are put together. This is a public, real-time event; more likely than not tweets were filtered through one or two other components to separate the interesting material from the…

> You have no idea what those intermediary components were, who set them up, and if they were all decoding/reencoding correctly.

...That's kind of the point. There was someone being sloppy there.

Re: Why the #AskObama tweet was garbled on screen

#27

Post by sundarurfriend »

eropple wrote: post #13
pak wrote: post #12

"This is SUCH a classic sloppy programmer mistake that I'm disappointed" Oh come off of it. This happens everywhere on the web on probably something like 25% of websites. And it's NOT always the consuming program's fault: very often somebody upstream, e.g. the hosting company, the person that wrote the HTML, the source of an RSS feed being inserted into the page etc. etc. forgot to encode something the way somebody e…

Twitter said that the message in question is UTF-8. The message recipient decoded the message with something that is not UTF-8. That's a sloppy mistake and should without doubt have been caught in testing.

The company responsible seems to have responded in the comments:

"It was definitely a mistake on our part. The problem was not the encoding on our data feed, but the HTML document was sent with ISO-8859-1. The second we inserted the twitter text into the DOM, the browsers interpreted the UTF-8 string as ISO-8859-1. Our visualizations are hosted on other platforms, and in this case the server was not configured to send UTF-8 with text/html even though the HTML file was encoded as such. It was the only issue (albeit a pretty obvious one) during an otherwise flawless event. I apologize to President Obama, Speaker Boehner, and Jack Dorsey for the mistake. If the readers of the blog think it was stupid, imagine how we felt. dev environment != production environment. If we would have just included a in the HTML head, then this would not have occurred.

The big take away is don’t make assumptions about other platforms (especially when it comes to encoding), and always include charset meta tag." [emphasis mine]

Re: Why the #AskObama tweet was garbled on screen

#28

Post by uxp »

anigbrowl wrote: post #7

I find it infuriating that this sort of thing is still a problem. I'm constantly seeing mangled apostrophes in places like Google reader too.

I would personally blame Wordpress for substituting a common apostrophe ' as the left ‘ and right quote ’ respectfully. Same with quotation marks “ and ” instead of the traditional ".

There is an option to not use "Smart Quotes", but it seems to be enabled by default.

Post reply on HN