Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unicode is harder than you think

mcilloni.ovh

Post reply on HN
Print view View original

41–50 of 121 posts

Re: Unicode is harder than you think

#41

Post by [deleted] »

deadbeeves wrote: post #36
skitter wrote: post #32

Annoyingly, Java, JavaScript, Windows file paths and more don't quite use UTF-16 (well, even if they did, that would be annoying) — they allow unpaired surrogates, which don't represent any Unicode character. So if you want to represent e.g. an arbitrary Windows file path in UTF-8, you can't; you have to use WTF-8 (wobbly transformation format) instead.

But UTF-8 is just a way to encode a number as a variable-length string of octets. Why would you be unable to encode, say, a terminating U+D800 as a string of three bytes at the end of a UTF-8 stream?

[deleted]

Re: Unicode is harder than you think

#42

Post by zadokshi »

alcover wrote: post #29

Currently working on a language, I feel dizzy after reading this. My stdlib will provide a (byte) Buffer class with basic low-level methods but I feel like iterating through it in fancy ways should be the concern of the user or 3rd-party libraries. I fail to see this as part of a programming language. Am I wrong here ?

You’re definitely wrong. You’re designing a language that only works for “Americans” by default.

Imagine how you would feel about a language that supports Arabic by default and needs special foo to work with American English?

You need to start thinking of characters as a type. Characters do not fit in bytes unless you’re American. And even if you’re American, people will still throw emdashes and Unicode quote symbols and emojis into your text. I’d suggest that if a language doesn’t have a proper character type, it’s not a useful language for the majority of the world.

UTF-8 is a reasonable compromise though.

Re: Unicode is harder than you think

#43

Post by deadbeeves »

skitter wrote: post #38
deadbeeves wrote: post #36

Earlier quoted context omitted.

But UTF-8 is just a way to encode a number as a variable-length string of octets. Why would you be unable to encode, say, a terminating U+D800 as a string of three bytes at the end of a UTF-8 stream?

Because that's how UTF-8 is defined[1]. WTF-8 lifts that restriction. [1] https://simonsapin.github.io/wtf-8/#utf-8

It doesn't sound very annoying, then. You use the exact same encoding scheme, but skip a verification step. Actually it sounds more convenient.

Re: Unicode is harder than you think

#44

Post by skitter »

sedatk wrote: post #37
skitter wrote: post #32

Annoyingly, Java, JavaScript, Windows file paths and more don't quite use UTF-16 (well, even if they did, that would be annoying) — they allow unpaired surrogates, which don't represent any Unicode character. So if you want to represent e.g. an arbitrary Windows file path in UTF-8, you can't; you have to use WTF-8 (wobbly transformation format) instead.

Certainly not true for Windows. Windows uses UTF-16; e.g. it uses proper surrogate pairs. https://learn.microsoft.com/en-us/windows/win32/intl/surroga...

That would be great, but that article is about recommendations for applications running on Windows, not about what valid file names applications may encounter. Here's a counter-example: https://github.com/golang/go/issues/32334

Re: Unicode is harder than you think

#45

Post by not2b »

svachalek wrote: post #9
dmitrygr wrote: post #3

Favourite unicode fact: properly rendering unicode requires understanding of the current geopolitical situation (Depending on whom you accept as a country and whom you do not, two country-code-letters may or may not render as a flag. This changes sometimes in today's world.). https://esham.io/2014/06/unicode-flags

Interesting. They pushed all the politics onto the font designers.

The font designer has to include a flag for each supported country. Often a given font is missing lots of flags for reasons that have nothing to do with whether the designer recognizes a given country or not, just a question of priorities; perhaps only 100 out of 200 flags are supported.

Re: Unicode is harder than you think

#46

Post by jmclnx »

qalmakka wrote: post #10
jmclnx wrote: post #4

No kidding, you have not lived until you try and explain UTF-8 to people who only believes in what they called "doublebyte". You think they get it, but surprise happens when a database load fails when loading Chinese Character "string" into a field sized calculated based upon 2 bytes per character.

It's terrible, and we IMHO owe that to some introductory university courses to Java (plus some Win32 veterans). I got very close to being rejected by a professor that was obstinately convinced that Unicode "characters" were 2 bytes because it drunk the Kool Aid in the '90s about Java's `char` type representing a Unicode character. Ugh. I still get angry by thinking back at that sometimes

I can relate, I remember a teacher stating "you never have to worry about the amount of memory". This was in the late 90s, I then asked "So I can load a 20 gig data file into memory", he said yes.

Re: Unicode is harder than you think

#47

Post by sedatk »

skitter wrote: post #44
sedatk wrote: post #37

Earlier quoted context omitted.

Certainly not true for Windows. Windows uses UTF-16; e.g. it uses proper surrogate pairs. https://learn.microsoft.com/en-us/windows/win32/intl/surroga...

That would be great, but that article is about recommendations for applications running on Windows, not about what valid file names applications may encounter. Here's a counter-example: https://github.com/golang/go/issues/32334

No, I mean Windows API honors UTF-16 surrogate pairs, and processes them correctly. It doesn't produce invalid UTF-16 strings either. Apps may not support UTF-16 properly, and that's not on Windows, is it?

NTFS, on the other hand, has no dictated format for filename encoding. It just stores raw bytes as filenames, so anything can be a filename on NTFS, including invalid strings if the caller decides to do so. That's not on Windows either, otherwise, we should add Linux to the list too as ext4 and most other file systems also don't care about filename encoding.

Re: Unicode is harder than you think

#48

Post by jraph »

deadbeeves wrote: post #43
skitter wrote: post #38

Earlier quoted context omitted.

Because that's how UTF-8 is defined[1]. WTF-8 lifts that restriction. [1] https://simonsapin.github.io/wtf-8/#utf-8

It doesn't sound very annoying, then. You use the exact same encoding scheme, but skip a verification step. Actually it sounds more convenient.

Still potentially annoying if you deal with some other code that expects UTF-8 proper and you pass it a wtf-8 string that fails the lifted verification.

Re: Unicode is harder than you think

#49

Post by Roark66 »

Indeed it is. One use of Unicode I do is for icons that can be used by console programs like (neo)vim. I was quite happy that xterm supports Unicode these days so I can use a fast terminal that supports OSC52 system clipboard integration(none of the newer gnome/KDE terminals do).

I was rather disappointed when I noticed my pretty Unicode icons would sometimes end up cut in half :-(

Re: Unicode is harder than you think

#50

Post by scatters »

alcover wrote: post #29

Currently working on a language, I feel dizzy after reading this. My stdlib will provide a (byte) Buffer class with basic low-level methods but I feel like iterating through it in fancy ways should be the concern of the user or 3rd-party libraries. I fail to see this as part of a programming language. Am I wrong here ?

It depends on the level and domain of your language. A low-level language can get by with just byte arrays. A mid-level language should probably handle at least some encodings, and provide codepoint access. A high-level language should handle locale-sensitive casing and collating, and grapheme-cluster access (note that this depends on the font!).
Post reply on HN