Unicode is harder than you think • Hacker News Board

Re: Unicode is harder than you think

#21

Post by Pannoniae » Tue, Jul 25, 2023, 8:59 PM UTC

Most programs claim to support Unicode but they actually don't. They either miscount string lengths (you type a CJK character or an emoji in, string appears shorter than what the program thinks), separate them improperly or many other things. It doesn't help that by default, most programming languages also handle unicode poorly, with the default APIs producing wrong results.

I'd take "we don't do unicode at all" or "we only support BMP" or "we don't support composite characters" any day over pretend-support (but then inevitably breaking when the program wasn't tested with anything non-ASCII)

(ninjaedit: to see how prevalent it is, even gigantic message apps such as discord make this mistake. There are users on discord who you can't add as friends because the friend input field is limited to 32.... something - probably bytes, yet elsewhere the program allows the name to be taken. This is easy to do with combining characters)

Re: Unicode is harder than you think

#22

Post by NovemberWhiskey » Tue, Jul 25, 2023, 9:12 PM UTC

I used to work on a platform at a large financial services firm; it was essentially complete ignorant of anything Unicode with respect to string handling, strings were null-terminated byte streams. The platform had CSV import capability for tabular data, and it had an integrated pivot table capability based on some widgets that had been grafted onto it.

Some of the users in Hong Kong discovered that you could import CSVs with Unicode text (e.g. index compositions with Chinese company names) and they'd display in the pivot table widgets and even be exportable to reports. But only most names. Some names were truncated or turned into garbage, and I was called upon to help debug this.

My first reaction was frank amazement that this "worked" at all: apparently, the path from the dumb CSV import code through to the Unicode-aware pivot table was sufficiently clean that much of the encoded text made it through OK. I can't remember the precise details now but I think the problem turned out to be embedded nulls from UTF-16 encoding and so was completely insoluble without a gut renovation of the platform.

Re: Unicode is harder than you think

#23

Post by frizlab » Tue, Jul 25, 2023, 9:14 PM UTC

jkaptur wrote: post #19

The logical next step here is to realize that if you want to be truly internationalized, pretty much every single method of the string class in your favorite language is an antipattern and should be used with extreme caution. Seriously!

I _think_ Swift did it properly. At least that’s what they claim[1]

[1]: https://www.swift.org/blog/utf8-string/

There were many discussions on how to handle strings in the forums too. Remarkably, it is not possible to access "abc"[1] as it has unpredictable performance; instead one has to build the index and making this should make the developer realize the operation is costly. All in all most beginners in Swift hate working with strings because it’s not intuitive at first, but to be fair in almost all languages strings handling is not done properly.

Re: Unicode is harder than you think

#24

Post by spookthesunset » Tue, Jul 25, 2023, 9:15 PM UTC

veave wrote: post #15

Longhanks wrote: post #11

Earlier quoted context omitted.

Imho, unicode should stay out of politics. Country flags, vaccine syringes and pregnant men should have nothing to do with how computers handle text and writing systems.

Or when big tech banded together to change the pistol emoji to some scifi zapper.

From what I recall the problem was on some devices it was rendered as a “sci-fi zapper” or squirtgun and on others it was a fairly realistic depiction of a gun. Leading to some misunderstandings…

Re: Unicode is harder than you think

#25

Post by jkaptur » Tue, Jul 25, 2023, 9:17 PM UTC

frizlab wrote: post #23

jkaptur wrote: post #19

The logical next step here is to realize that if you want to be truly internationalized, pretty much every single method of the string class in your favorite language is an antipattern and should be used with extreme caution. Seriously!

I _think_ Swift did it properly. At least that’s what they claim[1] [1]: https://www.swift.org/blog/utf8-string/ There were many discussions on how to handle strings in the forums too. Remarkably, it is not possible to access "abc"[1] as it has unpredictable performance; instead one has to build the index and making this should make the developer realize the operation is costly. All in all most beginners in Swift hat…

Oh, interesting! The fact that "abc"[1] doesn't work is a great sign, since my contention is that "the character at index i" is not a well-defined concept.

Re: Unicode is harder than you think

#26

Post by nightpool » Tue, Jul 25, 2023, 9:29 PM UTC

Thank you for being the first article I've ever actually read to explain the difference between NFC, NFD, NFKD and NFKC in a way that I actually understood. I was a little bored through the whole UCS/UTF* history lesson because I knew a lot of it already, but the normalization and collation examples were definitely worth it

Re: Unicode is harder than you think

#27

Post by david_draco » Tue, Jul 25, 2023, 9:31 PM UTC

Pannoniae wrote: post #21

Most programs claim to support Unicode but they actually don't. They either miscount string lengths (you type a CJK character or an emoji in, string appears shorter than what the program thinks), separate them improperly or many other things. It doesn't help that by default, most programming languages also handle unicode poorly, with the default APIs producing wrong results. I'd take "we don't do unicode at all" or "…

Maybe an "Acid test" for Unicode would help? These pages seem to go into that direction: https://www.kermitproject.org/utf8.html https://web.archive.org/web/20160306060703/http://www.inter-...

Placing a fuzzy tester like "hypothesis.strategies.characters" into the CI may also be revealing.

Re: Unicode is harder than you think

#28

Post by soneil » Tue, Jul 25, 2023, 9:32 PM UTC

spudlyo wrote: post #5

If you found this essay interesting, you owe it to yourself to check out this super entertaining talk "Plain Text"[0] from NDC 2022 by Dylan Beattie. Rabbit hole warning: This video caused me to lose an entire Sunday watching Dylan's talks on YouTube, which are uniformly awesome. [0]: https://www.youtube.com/watch?v=gd5uJ7Nlvvo

This is exactly what I came here to post, too. I believe he's unusually prolific because he's part of the organisation of these conferences - but his delivery pays off regardless. I actually sent exactly the same video to a colleague this a few days ago.

Re: Unicode is harder than you think

#29

Post by alcover » Tue, Jul 25, 2023, 9:40 PM UTC

Currently working on a language, I feel dizzy after reading this.

My stdlib will provide a (byte) Buffer class with basic low-level methods but I feel like iterating through it in fancy ways should be the concern of the user or 3rd-party libraries.

I fail to see this as part of a programming language.

Am I wrong here ?

Re: Unicode is harder than you think

#30

Post by o1y32 » Tue, Jul 25, 2023, 9:44 PM UTC

Regarding the title -- anecdotally, everyone I know is sacred of encoding issues, and I don't know anyone who claims they have a great understanding of Unicode or think it is easy (including myself). It is often overlooked for sure -- people don't realize there is a problem in the code until they run into a bug, ane it turns out they are treating strings wrong from the very beginning.