Unicode is harder than you think • Hacker News Board

Re: Unicode is harder than you think

#31

Post by ekidd » Tue, Jul 25, 2023, 9:54 PM UTC

alcover wrote: post #29

Currently working on a language, I feel dizzy after reading this. My stdlib will provide a (byte) Buffer class with basic low-level methods but I feel like iterating through it in fancy ways should be the concern of the user or 3rd-party libraries. I fail to see this as part of a programming language. Am I wrong here ?

One good approach is to have separate "byte array" and "string" types, and say, "Strings are always UTF-8. Anything else is a bug. Deal with it."

Then you can have a nice, user-friendly string class for basic UTF-8 text, which is pretty easy. Ignore sorting and grapheme clusters (those probably belong in libraries, and they require fairly large tables). Consider providing a library function to iterate over UTF-8 "characters" (as unpacked 32 bit Unicode code points).

This is one of the sweet spots in language design, and it provides enough structure for third-party libraries to work well together, without everyone reinventing their own string type.

For another good alternative to this approach, see Ruby.

Re: Unicode is harder than you think

#32

Post by skitter » Tue, Jul 25, 2023, 9:55 PM UTC

Annoyingly, Java, JavaScript, Windows file paths and more don't quite use UTF-16 (well, even if they did, that would be annoying) — they allow unpaired surrogates, which don't represent any Unicode character. So if you want to represent e.g. an arbitrary Windows file path in UTF-8, you can't; you have to use WTF-8 (wobbly transformation format) instead.

Re: Unicode is harder than you think

#33

Post by Lammy » Tue, Jul 25, 2023, 10:01 PM UTC

nightpool wrote: post #26

Thank you for being the first article I've ever actually read to explain the difference between NFC, NFD, NFKD and NFKC in a way that I actually understood. I was a little bored through the whole UCS/UTF* history lesson because I knew a lot of it already, but the normalization and collation examples were definitely worth it

Agreed, and it would be even better if it mentioned some real-world normalization issues like it does for the UCS encodings. I learned about it the hard way when dealing with Apple filesystems: https://eclecticlight.co/2021/05/08/explainer-unicode-normal...

Re: Unicode is harder than you think

#34

Post by kergonath » Tue, Jul 25, 2023, 10:08 PM UTC

Longhanks wrote: post #11

dmitrygr wrote: post #3

Favourite unicode fact: properly rendering unicode requires understanding of the current geopolitical situation (Depending on whom you accept as a country and whom you do not, two country-code-letters may or may not render as a flag. This changes sometimes in today's world.). https://esham.io/2014/06/unicode-flags

Imho, unicode should stay out of politics. Country flags, vaccine syringes and pregnant men should have nothing to do with how computers handle text and writing systems.

What does a syringe has to do with this, exactly?

Besides, why do you care what funny symbols people use in discussions that don’t involve you?

Re: Unicode is harder than you think

#35

Post by kergonath » Tue, Jul 25, 2023, 10:10 PM UTC

naniwaduni wrote: post #18

makeworld wrote: post #16

Earlier quoted context omitted.

There is no way to avoid it. It is very obvious that deciding whether "vaccine syringes" are political (and therefore excluded) or not is itself a political decision.

There's a certain kind of extremist who claims that their contentious positions aren't political, but the fact that there's an argument, and that you can point to mainstream coverage of it, strongly suggests that they're full of shit and everyone can see it.

What a strange position. The fact that an argument exists just shows that some people want to argue. Anyone can start an argument about anything. It’s hardly a good base to make a decision.

Re: Unicode is harder than you think

#36

Post by deadbeeves » Tue, Jul 25, 2023, 10:14 PM UTC

skitter wrote: post #32

Annoyingly, Java, JavaScript, Windows file paths and more don't quite use UTF-16 (well, even if they did, that would be annoying) — they allow unpaired surrogates, which don't represent any Unicode character. So if you want to represent e.g. an arbitrary Windows file path in UTF-8, you can't; you have to use WTF-8 (wobbly transformation format) instead.

But UTF-8 is just a way to encode a number as a variable-length string of octets. Why would you be unable to encode, say, a terminating U+D800 as a string of three bytes at the end of a UTF-8 stream?

Re: Unicode is harder than you think

#37

Post by sedatk » Tue, Jul 25, 2023, 10:15 PM UTC

skitter wrote: post #32

Annoyingly, Java, JavaScript, Windows file paths and more don't quite use UTF-16 (well, even if they did, that would be annoying) — they allow unpaired surrogates, which don't represent any Unicode character. So if you want to represent e.g. an arbitrary Windows file path in UTF-8, you can't; you have to use WTF-8 (wobbly transformation format) instead.

Certainly not true for Windows. Windows uses UTF-16; e.g. it uses proper surrogate pairs.

https://learn.microsoft.com/en-us/windows/win32/intl/surroga...

Re: Unicode is harder than you think

#38

Post by skitter » Tue, Jul 25, 2023, 10:28 PM UTC

deadbeeves wrote: post #36

skitter wrote: post #32

Annoyingly, Java, JavaScript, Windows file paths and more don't quite use UTF-16 (well, even if they did, that would be annoying) — they allow unpaired surrogates, which don't represent any Unicode character. So if you want to represent e.g. an arbitrary Windows file path in UTF-8, you can't; you have to use WTF-8 (wobbly transformation format) instead.

But UTF-8 is just a way to encode a number as a variable-length string of octets. Why would you be unable to encode, say, a terminating U+D800 as a string of three bytes at the end of a UTF-8 stream?

Because that's how UTF-8 is defined[1]. WTF-8 lifts that restriction.

[1] https://simonsapin.github.io/wtf-8/#utf-8

Re: Unicode is harder than you think

#39

Post by m_0x » Tue, Jul 25, 2023, 10:29 PM UTC

o1y32 wrote: post #30

Regarding the title -- anecdotally, everyone I know is sacred of encoding issues, and I don't know anyone who claims they have a great understanding of Unicode or think it is easy (including myself). It is often overlooked for sure -- people don't realize there is a problem in the code until they run into a bug, ane it turns out they are treating strings wrong from the very beginning.

sacred or scared? ;)

Re: Unicode is harder than you think

#40

Post by Knee_Pain » Tue, Jul 25, 2023, 10:31 PM UTC

skitter wrote: post #32

Annoyingly, Java, JavaScript, Windows file paths and more don't quite use UTF-16 (well, even if they did, that would be annoying) — they allow unpaired surrogates, which don't represent any Unicode character. So if you want to represent e.g. an arbitrary Windows file path in UTF-8, you can't; you have to use WTF-8 (wobbly transformation format) instead.

>WTF-8

truly an appropriate name