Why the #AskObama tweet was garbled on screen • Hacker News Board

Re: Why the #AskObama tweet was garbled on screen

#41

Post by jacobolus » Thu, Jul 07, 2011, 11:45 PM UTC

TVD wrote: post #33

sundarurfriend wrote: post #27

Earlier quoted context omitted.

The company responsible seems to have responded in the comments: "It was definitely a mistake on our part. The problem was not the encoding on our data feed, but the HTML document was sent with ISO-8859-1. The second we inserted the twitter text into the DOM, the browsers interpreted the UTF-8 string as ISO-8859-1. Our visualizations are hosted on other platforms, and in this case the server was not configured to sen…

Including is like the first thing you do when starting front-end development. Every time you don't validate...God kills a kitten. Please, think of the kittens. This has been a Public Service Announcement. Please code responsibly: http://validator.w3.org/

Just for the record, the correct spelling is now:

Re: Why the #AskObama tweet was garbled on screen

#42

Post by dthorpe » Fri, Jul 08, 2011, 12:01 AM UTC

zipdog wrote: post #31

What bugs me is not the mis-encoding (though that's a fail), but that people "struggled to understand" it... surely everyone's seen apostrophes turn into these special characters on enough web pages over the past ten years to have recognized what's happening when it does.

Blame Hollywood. Non-tech folks have been conditioned to panic whenever anything out of the ordinary appears on screen, because that's how Hollywood visualizes computer viruses and alien space invaders.

Re: Why the #AskObama tweet was garbled on screen

#43

Post by cpeterso » Fri, Jul 08, 2011, 12:55 AM UTC

luigi wrote: post #15

What modern software stack uses Extended ASCII as its default encoding? The last time I dealt with this problem, it was in 2005 or 2006 and I was working with PHP.

The article's comments says the problem was that the (implicit) character encoding of the HTML page:

"The problem was not the encoding on our data feed, but the HTML document was sent with ISO-8859-1. The second we inserted the twitter text into the DOM, the browsers interpreted the UTF-8 string as ISO-8859-1."

Re: Why the #AskObama tweet was garbled on screen

#44

Post by baddox » Fri, Jul 08, 2011, 1:07 AM UTC

zach wrote: post #17

Proof that even completely ordinary string data used in the most USA-centric domain imaginable STILL needs proper encoding.

I wouldn't call the right single quotation mark "completely ordinary string data." As far as I know, the only way it would ever get into a tweet is through some "smart correcting" client, or purposeful manual entry.

Re: Why the #AskObama tweet was garbled on screen

#45

Post by mgkimsal » Fri, Jul 08, 2011, 1:09 AM UTC

ignifero wrote: post #20

You have to give it to Microsoft. They use Word even for 140 letter documents now! (I understand spelling is a reason, but browsers have spelling now)

Downvoted, understandably perhaps, but nearly every time I see this sort of bug it's because someone has copy/pasted something from MS Word instead of using whatever native client input method was provided (usually a textarea or input field). Yes, blame the rendering script for not encoding properly, but I suspect this was Boehner's team copy/pasting from some internal MS Word doc. Cheap shot - they also likely copy/paste screenshots in to Word and mail those around too, instead of just mailing the actual graphic file. :)

Re: Why the #AskObama tweet was garbled on screen

#46

Post by old-gregg » Fri, Jul 08, 2011, 1:19 AM UTC

sundarurfriend wrote: post #27

eropple wrote: post #13

Earlier quoted context omitted.

Twitter said that the message in question is UTF-8. The message recipient decoded the message with something that is not UTF-8. That's a sloppy mistake and should without doubt have been caught in testing.

The company responsible seems to have responded in the comments: "It was definitely a mistake on our part. The problem was not the encoding on our data feed, but the HTML document was sent with ISO-8859-1. The second we inserted the twitter text into the DOM, the browsers interpreted the UTF-8 string as ISO-8859-1. Our visualizations are hosted on other platforms, and in this case the server was not configured to sen…

Was emailing involved? We've been getting so much support questions regarding encodings of HTML emails (we're a programmatic email service: http://mailgun.net) that we actually decided to become the first MTA to enforce utf-8 transcoding on the entire traffic: randomness in -> clean utf8 out and everyone is suddenly quiet&happy.

Something like this could be done as nginx/apache module: which detects encodings of the data and transcodes the HTML output into utf8 - could be useful for some cases.

Re: Why the #AskObama tweet was garbled on screen

#47

Post by ben1040 » Fri, Jul 08, 2011, 1:36 AM UTC

mgkimsal wrote: post #45

ignifero wrote: post #20

You have to give it to Microsoft. They use Word even for 140 letter documents now! (I understand spelling is a reason, but browsers have spelling now)

Downvoted, understandably perhaps, but nearly every time I see this sort of bug it's because someone has copy/pasted something from MS Word instead of using whatever native client input method was provided (usually a textarea or input field). Yes, blame the rendering script for not encoding properly, but I suspect this was Boehner's team copy/pasting from some internal MS Word doc. Cheap shot - they also likely copy/…

Cheap shot - they also likely copy/paste screenshots in to Word and mail those around too, instead of just mailing the actual graphic file. :)

When I was in consulting doing software integration work, few things infuriated me more than client "bug reports" arriving in the form of an email containing a 15 megabyte MS Word doc with a bunch of un-annotated screenshots.

I really hope Google someday opens up the awesome bug report/screencap feature in Google+, that lets you highlight part of the screen and redact sensitive parts.

Re: Why the #AskObama tweet was garbled on screen

#48

Post by sacrilicious » Fri, Jul 08, 2011, 1:40 AM UTC

markbnine wrote: post #38

What a cool shot. The prez with a common bug on his screen. And a bug I can fix! Still, even though this is an easy fix, he's going to need to open up a ticket.

My only regret is that I have but one upvote to give to this comment.

Re: Why the #AskObama tweet was garbled on screen

#49

Post by irahul » Fri, Jul 08, 2011, 1:47 AM UTC

pavel_lishin wrote: post #14

Did they see the encoding mistake before they showed it? Because I wonder how difficult it would be to create a string that says something innocuous in UTF-8 (e.g., "When will you bring the troops home #AskObama") but in ASCII would read as something totally different, but legible (e.g., "the secret priests would take great Cthulhu from his tomb to revive His subjects and resume his rule of earth...")

If you are talking about ASCII(7 bit ascii and not extended ascii), then it can't happen - the encoding is same for 0-127.

Re: Why the #AskObama tweet was garbled on screen

#50

Post by gutini » Fri, Jul 08, 2011, 1:48 AM UTC

An error like this should not detract from the value Mass Relevance delivers. Clearly an event like this or similar events like the Oscars in which they take part are better, more engaging because of their involvement.