Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unicode is harder than you think

mcilloni.ovh

Post reply on HN
Print view View original

111–120 of 121 posts

Re: Unicode is harder than you think

#111

Post by torstenvl »

theamk wrote: post #109
torstenvl wrote: post #108

Earlier quoted context omitted.

Yeah. This is objectively terrible advice. Ideology is not an excuse for data loss.

Do you have any specific examples, or are you just saying general statements? Back in the day, I worked quite a bit various encodings (my language had 2 primary one and 2 secondary one, and it was a guess which one the text was), and the data loss usually happened from programs that tried to support encodings. When program would not touch encodings, there might be some mojibake and unreadable text, but you could gene…

> Unless you deal with very old data, just use UTF-8

> I had to write some scripts which changed encoding of filenames and fixup random database or five, but there were no data loss.

If you do the first, it is impossible to do the second.

Are you just disagreeing for the sake of being disagreeable? Why would you do that?

Re: Unicode is harder than you think

#112

Post by sedatk »

kzrdude wrote: post #80
sedatk wrote: post #68

Earlier quoted context omitted.

Windows used to use UCS-2 because UTF-16 hadn't been invented yet. Any Windows after Windows 2000 supports UTF-16. You can't just dismiss the whole operating system API space just because NTFS doesn't enforce filename encoding. Yes, you might encounter file names with invalid UTF sequences because some app might be allowing them to be created. That can also happen in Linux too. Then add Linux, and probably others to…

On Linux it's at least explicit that file paths can be any string of bytes (any or "no" encoding)

I don't understand why this gets brought up. You still have to assume the filenames are encoded to something in order to display them, otherwise there would be no way to display a filename other than in hexadecimal. Windows always uses UTF-16 while Linux uses the locale (which defaults to UTF-8). It's not like working with filenames on Linux is somewhat easier or more straightforward, it's certainly not.

Re: Unicode is harder than you think

#113

Post by zamadatix »

eviks wrote: post #96
MadVikingGod wrote: post #63

Earlier quoted context omitted.

The number of bytes is still hard to do right. For example if I have a payload that can only be 4000 bytes, how do I take an arbitrary utf-8 string and get one that is <= 4000 bytes and doesn't cut off a graphmeme cluster?

Count by grapheme clusters and stop right before it exceeds the limit?

Isn't that the length of the string? Also there are various things about Unicode that make doing this "right" for an input much harder than it sounds.

Re: Unicode is harder than you think

#114

Post by eviks »

zamadatix wrote: post #113
eviks wrote: post #96

Earlier quoted context omitted.

Count by grapheme clusters and stop right before it exceeds the limit?

Isn't that the length of the string? Also there are various things about Unicode that make doing this "right" for an input much harder than it sounds.

string length is a poorly defined concept, but yes, counting real characters (grapheme clusters) counts

And what "various things" do you mean that are on top the char split?

Re: Unicode is harder than you think

#115

Post by qalmakka »

zadokshi wrote: post #65
alcover wrote: post #51

Earlier quoted context omitted.

I mean the reserved words in my lang are in English, and the user-created identifiers must be ASCII at the moment. I could extend the latter to some wider range but my question was more about providing built-in Unicode string handling. My Buffer class lets you allocate bytes and it's up to your code to interpret those bytes. I get your concern though and realize parsing let foo = " " would be a problem.

Apologies if I come off a bit strong on this one. I deal mostly with non “American” text so it’s a pain point. Unicode is so messy, trying to push handling it on to each individual programmer is not an ideal situation. Let’s just say, no one in Greece, Iceland, or Korea is sitting around wondering if their new programming language should support their alphabet natively. Imagine being from a country where a language’s…

To be fair, that has never stopped people from using C and C++ all around the world...

EDIT: and most languages made before 1990 too

Re: Unicode is harder than you think

#116

Post by dhosek »

MadVikingGod wrote: post #63
jameshart wrote: post #57

Earlier quoted context omitted.

Why are people so interested in ‘lengths’ of strings? The only thing anyone should actually care about is either the number of bytes it takes to store the string, or the number of pixels wide it is when rendered. Both of which are only loosely related to how many ‘characters’ or ‘grapheme clusters’ they contain, and which are themselves only vaguely correlated. 𒈙 (CUNEIFORM SIGN LUGAL OPPOSING LUGAL) is four bytes o…

The number of bytes is still hard to do right. For example if I have a payload that can only be 4000 bytes, how do I take an arbitrary utf-8 string and get one that is <= 4000 bytes and doesn't cut off a graphmeme cluster?

The rules for grapheme clusters are well-defined and documented. I’ve only implemented a forward iterator on graphemes in the finl_unicode rust crate, but backwards would not be that difficult either. If I’m looking at a byte stream, I have to first make sure that I’m not in the middle of a multi-byte sequence (easily enough accomplished), you need to do some forwards and backwards iteration to determine where the grapheme boundaries are, and likely you will want a library to do this for you, but it’s something that should take a reasonably skilled programmer in about a day or two.

Re: Unicode is harder than you think

#117

Post by dhosek »

gumby wrote: post #102
qalmakka wrote: post #95

Earlier quoted context omitted.

I don't understand why the felt the need to add precomposed characters for Hangul. Why? Why couldn't they just let the system compose them instead?

There was probably an existing character set that had precomposed characters and Unicode always includes existing character sets for round trip idempotency. Imagine if hanzi had been encoded as radicals…

Exactly this. When Unicode 1.0 came out, there was no standardized way to indicate that f i should compose into fi, let alone that ᄋ ㅡ ᄀ should compose into 윽. IIRC, composing of East Asian scripts into han or hangul happened at the firmware level, not at the app or OS level. If you didn’t have the hardware built into your keyboard, you simply could not type anything in East Asian scripts. A lot of the inconsistencies in Unicode (like the difference in how vowels are handled in Thai vs Indic scripts) come down to how the original 8-bit encodings worked pre-Unicode.

Re: Unicode is harder than you think

#118

Post by qalmakka »

dhosek wrote: post #117
gumby wrote: post #102

Earlier quoted context omitted.

There was probably an existing character set that had precomposed characters and Unicode always includes existing character sets for round trip idempotency. Imagine if hanzi had been encoded as radicals…

Exactly this. When Unicode 1.0 came out, there was no standardized way to indicate that f i should compose into fi, let alone that ᄋ ㅡ ᄀ should compose into 윽. IIRC, composing of East Asian scripts into han or hangul happened at the firmware level, not at the app or OS level. If you didn’t have the hardware built into your keyboard, you simply could not type anything in East Asian scripts. A lot of the inconsistencies…

Very interesting, I guess that legacy really is the root cause of everything. Do you have any links or such about the history of Unicode?

Re: Unicode is harder than you think

#119

Post by seoulbigchris »

Lammy wrote: post #33
nightpool wrote: post #26

Thank you for being the first article I've ever actually read to explain the difference between NFC, NFD, NFKD and NFKC in a way that I actually understood. I was a little bored through the whole UCS/UTF* history lesson because I knew a lot of it already, but the normalization and collation examples were definitely worth it

Agreed, and it would be even better if it mentioned some real-world normalization issues like it does for the UCS encodings. I learned about it the hard way when dealing with Apple filesystems: https://eclecticlight.co/2021/05/08/explainer-unicode-normal...

Man, this gave me such a big headache when using a NAS with MacOS. I have a bunch of file names that use Hangul, and it resulted in a mess that I never fully sorted out.
Post reply on HN