Zed Shaw, can you make a post called "Programmers Need To Learn Unicode Or I Will Kill Them All"?
Why the #AskObama tweet was garbled on screen
51–60 of 78 posts
Re: Why the #AskObama tweet was garbled on screen
#52Earlier quoted context omitted.
Downvoted, understandably perhaps, but nearly every time I see this sort of bug it's because someone has copy/pasted something from MS Word instead of using whatever native client input method was provided (usually a textarea or input field). Yes, blame the rendering script for not encoding properly, but I suspect this was Boehner's team copy/pasting from some internal MS Word doc. Cheap shot - they also likely copy/…
Cheap shot - they also likely copy/paste screenshots in to Word and mail those around too, instead of just mailing the actual graphic file. :) When I was in consulting doing software integration work, few things infuriated me more than client "bug reports" arriving in the form of an email containing a 15 megabyte MS Word doc with a bunch of un-annotated screenshots. I really hope Google someday opens up the awesome b…
I almost wouldn't mind, except the screenshots are inevitably shrunk down so as to be unreadable.
In OSX, the cmd-shift-4 (and 3) keystroke which screenshots right in to a file are near life-transforming. Snap, drag the file into an email, and it's done. I'm sure there are utilities in Windows which do this, but having it built in is great - no apps to start or install.
Re: Why the #AskObama tweet was garbled on screen
#53Proof that even completely ordinary string data used in the most USA-centric domain imaginable STILL needs proper encoding.
I wouldn't call the right single quotation mark "completely ordinary string data." As far as I know, the only way it would ever get into a tweet is through some "smart correcting" client, or purposeful manual entry.
A healthy amount of the writing of the world still begins life in a word processor.
Any string born of this heritage is likely to have single and double (curved) quotation marks. OpenOffice, Word, Pages – they all do it and are expected to.
For that reason, I consider them to be reasonably present in ordinary string data.
Re: Why the #AskObama tweet was garbled on screen
#54Earlier quoted context omitted.
I would personally blame Wordpress for substituting a common apostrophe ' as the left ‘ and right quote ’ respectfully. Same with quotation marks “ and ” instead of the traditional ". There is an option to not use "Smart Quotes", but it seems to be enabled by default.
Wordpress is for writing, and writing should be properly typeset, and properly-typeset text has proper quotes. BTW, a note on the term “smart quotes”: that originated when word processors became “smart” about transforming the easy-to-type (but incorrect) ' and " to their proper equivalents automatically. The quotes themselves aren’t smart…they’re just quotes. Typography nerd out.
For typography, yes.
For many blogs in the hacker community, source code snippets inside You forget that Wordpress is used for many people outside of the writing community. When writing, and if the writer cares about having their word properly typeset, then the author can do so themselves. Wordpress tries to be smart about it, and covert them, but many people do not care about such features. The developers, however, do. Also, may I remind everyone, downvotes on HN are not for disagreement, they are for factually incorrect statements. tags can also be given "smart quotes", which completely breaks any strings that may be present.
Re: Why the #AskObama tweet was garbled on screen
#55Earlier quoted context omitted.
Wordpress is for writing, and writing should be properly typeset, and properly-typeset text has proper quotes. BTW, a note on the term “smart quotes”: that originated when word processors became “smart” about transforming the easy-to-type (but incorrect) ' and " to their proper equivalents automatically. The quotes themselves aren’t smart…they’re just quotes. Typography nerd out.
For typography, yes. For many blogs in the hacker community, source code snippets inside tags can also be given "smart quotes", which completely breaks any strings that may be present. You forget that Wordpress is used for many people outside of the writing community. When writing, and if the writer cares about having their word properly typeset, then the author can do so themselves. Wordpress tries to be smart about…
IMO it’s not bad that Wordpress auto-educates quotes, it’s that it does it even within code blocks. Which is inexcusable, for precisely the reason you mention.
Markdown + SmartyPants are a better solution IMO. (And you can install WP plugins that do this and that disable Wordpress’ default quote educator.)
Re: Why the #AskObama tweet was garbled on screen
#56Zed Shaw, can you make a post called "Programmers Need To Learn Unicode Or I Will Kill Them All"?
Joel Spolsky wrote something that comes close, sans the threat:
Re: Why the #AskObama tweet was garbled on screen
#57He had to do a twitter townhall because El Jefe couldn't get a G+ invite :-|
Re: Why the #AskObama tweet was garbled on screen
#58Hey I work for the company responsible for the visualization behind the president and the content on http://askobama.twitter.com
Let me take this very excellent opportunity to say that we are looking to hire a full time "front end" developer. You'll get to work on badass projects like the Obama Town Hall. Ideally, you'd be located in Austin. Find me on Twitter @efalcao to learn more.
Re: Why the #AskObama tweet was garbled on screen
#59Maybe I'm just a terrible programmer, but I think the author may be a little over emphasizing the seriousness of the bug. To me, this is one of those throw away issues that you keep in the back of your mind. Unless I'm coding something that is extremely datacentric and critical in that sense. But it's not like its on one of my "top 20 must run tests" or anything. It's always one of those issues I ignore or assume is…
You're not a terrible programmer, just an uninformed and/or slightly Anglocentric one.
Just last week, my co-worker had to waste two days debugging a two-year-old Sphinx setup (the person who implemented it no longer works with us) because a Japanese user of our blogging service wasn't seeing the post he was looking for in our search feature. The problem was that the conduit feeding the Sphinx indexer was handling Unicode incorrectly (to be specific, it was deleting certain bytes wholesale because this guy believed them never to be valid, and this broke multi-byte sequences horribly). Those two days would have been much better spent working on his current project.
Additionally, not handling Unicode correctly can leave you open to certain types of security holes!
The biggest of these is probably that Unicode means _a string is not an array of bytes_, so naïve allocators for languages with byte-array strings (read: C and its brethren) are susceptible to buffer overruns when handed multi-byte Unicode sequences when they're expecting ASCII.
Here's another one: the Unicode character space contains many, many glyphs that look almost (or in some cases exactly) like ASCII characters. RFC 3492 defines a framework for internationalized domain names (IDN) that encodes non-ASCII code points using ASCII characters, and the most common implementations of this transparently go between the two. This means that you could register "bаnkofamerica.com" (actually xn--bnkofamerica-x9j.com) and put a phishing site there, and people would happily click on the identical-looking URL and give you their bank account. This was pointed out several years ago and most modern browsers have mechanisms in place to defend against it, but your custom application might not unless you're careful to check what you're doing.
There are plenty of blog posts and articles out there designed to tell you how to be safe when dealing with Unicode (and you should assume that you will be). I highly suggest you go read one.
Re: Why the #AskObama tweet was garbled on screen
#60Hey I work for the company responsible for the visualization behind the president and the content on http://askobama.twitter.com Let me take this very excellent opportunity to say that we are looking to hire a full time "front end" developer. You'll get to work on badass projects like the Obama Town Hall. Ideally, you'd be located in Austin. Find me on Twitter @efalcao to learn more.
FWIW, this was an intense project to pull off. 1000's of tweets per minute from Twitter, 8000 requests per second on http://askobama.twitter.com (where the same tweet was also delivered by us and rendered correctly).
We're not lazy or sloppy... It basically boiled down to one server sent down the right header...the production one didn't.
Unicode issues are sorta in the class of "gotcha" issue. They happen, you go "oh shit" and fix them right away. Our "oh shit" moment just happened to come at the most intense possible moment....in front of the president, with so many watching.
Wanted to reiterate once again: We're Hiring! @efalcao on twitter. Early stage startup looking for exceptional talent.