thank you for the fascinating article! I've seen this bug in other places, and I never knew what it was (and usually brushed it off instead of digging deeper to find it).
Why the #AskObama tweet was garbled on screen
71–78 of 78 posts
Re: Why the #AskObama tweet was garbled on screen
#72Hey I work for the company responsible for the visualization behind the president and the content on http://askobama.twitter.com Let me take this very excellent opportunity to say that we are looking to hire a full time "front end" developer. You'll get to work on badass projects like the Obama Town Hall. Ideally, you'd be located in Austin. Find me on Twitter @efalcao to learn more.
FWIW, this was an intense project to pull off. 1000's of tweets per minute from Twitter, 8000 requests per second on http://askobama.twitter.com (where the same tweet was also delivered by us and rendered correctly). We're not lazy or sloppy... It basically boiled down to one server sent down the right header...the production one didn't. Unicode issues are sorta in the class of "gotcha" issue. They happen, you go "oh…
I think you did a great job, way to represent the Austin Tech scene!
Re: Why the #AskObama tweet was garbled on screen
#73Earlier quoted context omitted.
>Wasn't this in real time? No. Questions were culled from Tweets with the #AskObama hashtag starting on June 30. Some of the questions did come in close to real-time. I think the most recent ones were 5-10 minutes old when presented to the president.
So how did they choose the questions then? Based on retweets, or the Twitter team just picked the ones they liked?
They had a group of moderators who were selected to find the best questions. I believe most of the moderators were journalists or bloggers.
Re: Why the #AskObama tweet was garbled on screen
#74Earlier quoted context omitted.
FWIW, this was an intense project to pull off. 1000's of tweets per minute from Twitter, 8000 requests per second on http://askobama.twitter.com (where the same tweet was also delivered by us and rendered correctly). We're not lazy or sloppy... It basically boiled down to one server sent down the right header...the production one didn't. Unicode issues are sorta in the class of "gotcha" issue. They happen, you go "oh…
I think you did a great job, way to represent the Austin Tech scene!
Thanks so much!
Re: Why the #AskObama tweet was garbled on screen
#75Part of the problem is that UTF-8 makes things really, really simple, and bulletproof, and then people have to go and create problems again.
Listen. Any time you use an encoding other than UTF-8, you are creating incompatibilities. If your stated intention is to facilitate communication, you are failing. You are a bad person. Stop doing it. The only possible excuse for using a non-UTF-8 encoding is to frustrate communication.
(It's too fucking bad HTTP mandates that the default charset is ISO-8859-1.)
Re: Why the #AskObama tweet was garbled on screen
#76Earlier quoted context omitted.
Touchée.
Are you female and awesome, or was that just a typo?
I am neither female nor an awesome typist. Actually, I failed to remember the alt-key combination for the é, and then just pasted it in from charmap instead. I wonder why it is that my cellphone had more advanced text input facilities than my desktop. I know there are 3rd party utilities for bringing up special characters if you hold down a key and so on, but you'd think this sort of thing would have been standardized by now.
Re: Why the #AskObama tweet was garbled on screen
#77Earlier quoted context omitted.
> Or maybe I've just been fortunate enough to be in an environment where an occasional goof of this caliber doesn't have any serious consequences. Primarily, this means you don't have to support internationalization - which is hardly a bad thing, especially if you work on a startup, where worldwide distribution should be the last thing on your mind. When your product is rendered in over 80 scripts, including right-to…
And Hanselman works for Microsoft, where i18n is a big deal. So yes, for someone who's been at MS for a while i18n related issues become second nature. But if you typically are only targeting the United States, it's more understandable to not have these things on the brain.
On the other hand, Outlook still has a ridiculous bug that uses the wrong encoding when presenting HTML email - that is, it uses the encoding of the email's text-body when presenting the html-body, even if the html-body specifies a different encoding.
So, if the two bodies have differing encodings (charsets), then the HTML body will look wrong. Unless you force Outlook to always use UTF-8 for encoding emails (which is a setting, but not the default) then you'll end up sending emails that will look garbled to your recipient.
This "differing charset" scenario actually happens pretty frequently, because of the following scenario:
a) You write an email (or reply to an existing email - actually it happens most with replies).
b) Outlook's text editor decides to insert a non-breaking space (codepoint U+00A0). Perhaps it generates HTML with but before transmission this eventually turns into the single UTF-8 character 0xC2 0xA0.
c) When generating the text-body, Outlook decides to just use a plain old space, so the text body is plain ASCII.
d) Outlook, in its cleverness, then says "ooh, I can 'conserve' encoding-ness and use plain old iso-8859-1 for the text body, but I need to use UTF-8 for the HTML body because of that non-ascii character"
e) Outlook generates this email (please excuse formatting woes due to HN).
Content-Type: multipart/alternative; boundary="0016e64dbd929784310488b2b082"
This is a multi-part message in MIME format.
--0016e64dbd929784310488b2b082
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="ISO-8859-1"
yo yo
--0016e64dbd929784310488b2b082
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
yo=C2=A0yo
--0016e64dbd929784310488b2b082--
When you view the above email in Outlook, you see "yo yo" instead of "yo yo"
Re: Why the #AskObama tweet was garbled on screen
#78Earlier quoted context omitted.
And Hanselman works for Microsoft, where i18n is a big deal. So yes, for someone who's been at MS for a while i18n related issues become second nature. But if you typically are only targeting the United States, it's more understandable to not have these things on the brain.
On the other hand, Outlook still has a ridiculous bug that uses the wrong encoding when presenting HTML email - that is, it uses the encoding of the email's text-body when presenting the html-body, even if the html-body specifies a different encoding. So, if the two bodies have differing encodings (charsets), then the HTML body will look wrong. Unless you force Outlook to always use UTF-8 for encoding emails (which i…
You should file this at connect.microsoft.com. MS devs and PMs really do read and triage bug reports coming from there. The more details you put in the bug report, the better.