Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unicode Is Awesome

wisdom.engineering

Post reply on HN
Print view View original

41–50 of 159 posts

Re: Unicode Is Awesome

#41

Post by ori_b »

kazinator wrote: post #9
jrochkind1 wrote: post #8

Unicode is pretty amazing. People REALLY like to complain about unicode, but where it's complicated, it's because the _problem space_ is complicated. Which it is. People are actually complaining that they wish handling global text wasn't so complicated, like, that humans had been a lot simpler and more limited with their inventions of alphabets and how they were used in typesetting and printing and what not, and that…

No, where Unicode is complicated is where the Unicode people decided to make it complicated to bolster their egos, to the detriment of everyone downstream of them. Like with most standardization, the people at the helm are the wrong people with the wrong motivations.

Interesting statement. Other than maybe han unification, what would you do differently?

Re: Unicode Is Awesome

#42

Post by saagarjha »

kazinator wrote: post #9
jrochkind1 wrote: post #8

Unicode is pretty amazing. People REALLY like to complain about unicode, but where it's complicated, it's because the _problem space_ is complicated. Which it is. People are actually complaining that they wish handling global text wasn't so complicated, like, that humans had been a lot simpler and more limited with their inventions of alphabets and how they were used in typesetting and printing and what not, and that…

No, where Unicode is complicated is where the Unicode people decided to make it complicated to bolster their egos, to the detriment of everyone downstream of them. Like with most standardization, the people at the helm are the wrong people with the wrong motivations.

> No, where Unicode is complicated is where the Unicode people decided to make it complicated to bolster their egos, to the detriment of everyone downstream of them.

Where?

Re: Unicode Is Awesome

#43

Post by paulddraper »

deathanatos wrote: post #22

I swear there should be some rule or law about how Unicode articles will inevitably muddle code units/points / grapheme clusters / bytes together. > String length is typically determined by counting codepoints. > This means that surrogate pairs would count as two characters. If you were counting code points, a surrogate pair would be 1. If it's two, you're counting code units. > Combining multiple diacritics may be s…

Also a law that the author is thinking of one and only one programming language.

> String length is typically determined by counting codepoints.

That depends entirely on what "strings" you are talking about.

In C/Go/Rust/Ruby, char*/string/std::string::String/String is bytes.

In Java/JavaScript, java.lang.String/String is UTF-16 code units.

In Python 3, str is code points.

In Swift, String is extended grapheme clusters.

In Haskell, there are various different "string" types in common use.

And in C++, std::basic_string is a generic container for whatever element type you want. (std::string specialization being for bytes.)

EDIT: Clarified that I don't disagree with parent comment; merely pointing out additional less-than-precise language.

Re: Unicode Is Awesome

#44

Post by deathanatos »

paulddraper wrote: post #43
deathanatos wrote: post #22

I swear there should be some rule or law about how Unicode articles will inevitably muddle code units/points / grapheme clusters / bytes together. > String length is typically determined by counting codepoints. > This means that surrogate pairs would count as two characters. If you were counting code points, a surrogate pair would be 1. If it's two, you're counting code units. > Combining multiple diacritics may be s…

Also a law that the author is thinking of one and only one programming language. > String length is typically determined by counting codepoints. That depends entirely on what "strings" you are talking about. In C/Go/Rust/Ruby, char*/string/std::string::String/String is bytes. In Java/JavaScript, java.lang.String/String is UTF-16 code units. In Python 3, str is code points. In Swift, String is extended grapheme cluste…

Sure, different languages have various, usually bad, definitions of length.

The point is that those two sentences themselves in the article are conflicting with each other, not that we're talking about any language in particular. (But certainly the article could go into a survey of common languages like you have.)

Re: Unicode Is Awesome

#45

Post by ridiculous_fish »

jcranmer wrote: post #10
euske wrote: post #6

I mean, awesome for whom? It might be awesome for end users as you can type in or copy/paste things without caring which language you're using. But for programmers, Unicode is a bloated monstrosity and a source of endless nightmare. Eventually, it's not going to be awesome for end users either because it will be plagued by a lot of (subtle) inconsistencies. Unicode looks a lot like a leaky abstraction to me (because…

If you think Unicode is a "bloated monstrosity and a source of endless nightmare," what would you remove from Unicode? And if you're going to respond "emoji", I'll point out that removing emoji doesn't actually remove anything that makes text processing with Unicode difficult, just makes it more likely that people will assume that what works for English works for everybody. (Side note: it is not possible to accuratel…

How about removing variation selectors? For example it's possible to turn an emoji back into text by appending a code point!

They are very painful to implement and most don't get it right.

See https://twitter.com/ridiculous_fish/status/10894210337932369...

Re: Unicode Is Awesome

#46

Post by happytoexplain »

paulddraper wrote: post #43
deathanatos wrote: post #22

I swear there should be some rule or law about how Unicode articles will inevitably muddle code units/points / grapheme clusters / bytes together. > String length is typically determined by counting codepoints. > This means that surrogate pairs would count as two characters. If you were counting code points, a surrogate pair would be 1. If it's two, you're counting code units. > Combining multiple diacritics may be s…

Also a law that the author is thinking of one and only one programming language. > String length is typically determined by counting codepoints. That depends entirely on what "strings" you are talking about. In C/Go/Rust/Ruby, char*/string/std::string::String/String is bytes. In Java/JavaScript, java.lang.String/String is UTF-16 code units. In Python 3, str is code points. In Swift, String is extended grapheme cluste…

I'm in love with Swift's approach, where the default representation is a well defined thing that both users and developers think of as "characters", but all the other representations are trivially accessible.

Re: Unicode Is Awesome

#47

Post by happytoexplain »

kazinator wrote: post #9
jrochkind1 wrote: post #8

Unicode is pretty amazing. People REALLY like to complain about unicode, but where it's complicated, it's because the _problem space_ is complicated. Which it is. People are actually complaining that they wish handling global text wasn't so complicated, like, that humans had been a lot simpler and more limited with their inventions of alphabets and how they were used in typesetting and printing and what not, and that…

No, where Unicode is complicated is where the Unicode people decided to make it complicated to bolster their egos, to the detriment of everyone downstream of them. Like with most standardization, the people at the helm are the wrong people with the wrong motivations.

As a developer who's been working intimately with user-facing strings for years, I have to disagree in the strongest possible terms. Unicode is one of the borderline zero standards that is almost angelic in its purity, with only an extremely few things I think might have served better if done differently.

Re: Unicode Is Awesome

#48

Post by ncmncm »

jrochkind1 wrote: post #8

Unicode is pretty amazing. People REALLY like to complain about unicode, but where it's complicated, it's because the _problem space_ is complicated. Which it is. People are actually complaining that they wish handling global text wasn't so complicated, like, that humans had been a lot simpler and more limited with their inventions of alphabets and how they were used in typesetting and printing and what not, and that…

There is way more than enough wacky stuff introduced by Unicode. Having dozens of letters A, for example. And giving a Japanese Kanji character the same code as a Chinese one that usually looks similar.

Re: Unicode Is Awesome

#49

Post by riquito »

BurningFrog wrote: post #39
squaresmile wrote: post #29

Unicode definitely has flaws but that doesn't mean we should throw the baby out with the bathwater and go back to "ASCII and other character sets." There's a reason we moved on from that world. However, I bet we will see another encoding coming up eventually (within 30 years) which solves the problems Unicode currently has and introduces a new set of problems as well. I saw this comment [0] about how that encoding sh…

Unicode URL has serious security problems. The canonical example is google.com vs gооgle.com.

That was solved years ago by IDN/Punycode (implemented by any browser worth their salt).

Re: Unicode Is Awesome

#50

Post by paulddraper »

happytoexplain wrote: post #46
paulddraper wrote: post #43

Earlier quoted context omitted.

Also a law that the author is thinking of one and only one programming language. > String length is typically determined by counting codepoints. That depends entirely on what "strings" you are talking about. In C/Go/Rust/Ruby, char*/string/std::string::String/String is bytes. In Java/JavaScript, java.lang.String/String is UTF-16 code units. In Python 3, str is code points. In Swift, String is extended grapheme cluste…

I'm in love with Swift's approach, where the default representation is a well defined thing that both users and developers think of as "characters", but all the other representations are trivially accessible.

Agreed. Normal "text" operations tend to work quite well.

And the other forms are accessible, e.g. if you write a text-based parser (XML, JSON, etc.), you'll probably want String.unicodeScalars.

Post reply on HN