Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unicode is harder than you think

mcilloni.ovh

Post reply on HN
Print view View original

51–60 of 121 posts

Re: Unicode is harder than you think

#51

Post by alcover »

zadokshi wrote: post #42
alcover wrote: post #29

Currently working on a language, I feel dizzy after reading this. My stdlib will provide a (byte) Buffer class with basic low-level methods but I feel like iterating through it in fancy ways should be the concern of the user or 3rd-party libraries. I fail to see this as part of a programming language. Am I wrong here ?

You’re definitely wrong. You’re designing a language that only works for “Americans” by default. Imagine how you would feel about a language that supports Arabic by default and needs special foo to work with American English? You need to start thinking of characters as a type. Characters do not fit in bytes unless you’re American. And even if you’re American, people will still throw emdashes and Unicode quote symbols…

I mean the reserved words in my lang are in English, and the user-created identifiers must be ASCII at the moment. I could extend the latter to some wider range but my question was more about providing built-in Unicode string handling.

My Buffer class lets you allocate bytes and it's up to your code to interpret those bytes.

I get your concern though and realize parsing

  let foo = ""
would be a problem.

Re: Unicode is harder than you think

#52

Post by chungy »

sedatk wrote: post #37
skitter wrote: post #32

Annoyingly, Java, JavaScript, Windows file paths and more don't quite use UTF-16 (well, even if they did, that would be annoying) — they allow unpaired surrogates, which don't represent any Unicode character. So if you want to represent e.g. an arbitrary Windows file path in UTF-8, you can't; you have to use WTF-8 (wobbly transformation format) instead.

Certainly not true for Windows. Windows uses UTF-16; e.g. it uses proper surrogate pairs. https://learn.microsoft.com/en-us/windows/win32/intl/surroga...

Windows was designed around UCS-2, while it can be upwards compatible with storing UTF-16 data, the Windows APIs don't stop you from inserting almost any sequence of 16-bit words as file names, including unpaired surrogates (because Windows doesn't actually care about the contents of the 16-bit words).

It just happens that most Windows programs will treat it as UTF-16 and it'll mostly be fine. Until you run into the edge case where you do have file names that aren't UTF-16.

This is basically a result of Windows's file API being fixed in 1993, a good few years before Unicode decided that 16 bits wasn't enough for everyone, and hence a good few years before UTF-16 existed.

Re: Unicode is harder than you think

#53

Post by tialaramex »

alcover wrote: post #29

Currently working on a language, I feel dizzy after reading this. My stdlib will provide a (byte) Buffer class with basic low-level methods but I feel like iterating through it in fancy ways should be the concern of the user or 3rd-party libraries. I fail to see this as part of a programming language. Am I wrong here ?

People expect Strings. You don't have to provide strings, but you should be aware that people expect them and if you provide something else but don't clearly label it as not strings there's an excellent chance they'll go "Oh, strings" and then they'll complain that it doesn't work the way they expected.

If I can write "foo".contains('o') or similar in your language, but then you pretend you don't have strings, nobody will believe you and they're quite right to complain when, inevitably, lots of trickier stuff doesn't work because you didn't bother.

WUFFS doesn't have strings. But WUFFS isn't a general purpose language. Fortran has strings. C has strings (they're terrible but they're clearly strings).

Re: Unicode is harder than you think

#54

Post by dahfizz »

chungy wrote: post #52
sedatk wrote: post #37

Earlier quoted context omitted.

Certainly not true for Windows. Windows uses UTF-16; e.g. it uses proper surrogate pairs. https://learn.microsoft.com/en-us/windows/win32/intl/surroga...

Windows was designed around UCS-2, while it can be upwards compatible with storing UTF-16 data, the Windows APIs don't stop you from inserting almost any sequence of 16-bit words as file names, including unpaired surrogates (because Windows doesn't actually care about the contents of the 16-bit words). It just happens that most Windows programs will treat it as UTF-16 and it'll mostly be fine. Until you run into the…

Why is that a problem? That's how unix does it as well.

A file path is just a sequence of bytes. Don't try to validate it in any sort of encoding, it's just bytes that the OS uses to identify a file.

Re: Unicode is harder than you think

#55

Post by alcover »

scatters wrote: post #50
alcover wrote: post #29

Currently working on a language, I feel dizzy after reading this. My stdlib will provide a (byte) Buffer class with basic low-level methods but I feel like iterating through it in fancy ways should be the concern of the user or 3rd-party libraries. I fail to see this as part of a programming language. Am I wrong here ?

It depends on the level and domain of your language. A low-level language can get by with just byte arrays. A mid-level language should probably handle at least some encodings, and provide codepoint access. A high-level language should handle locale-sensitive casing and collating, and grapheme-cluster access (note that this depends on the font!).

It's a dialect of C, so somewhere between low and mid.

  > handle at least some encodings
What else could it be than Utf8 ?

  > locale-sensitive casing and collating, and grapheme-cluster access (note that this depends on the font!)
You made up my mind haha. It won't be high-level.

Re: Unicode is harder than you think

#56

Post by chungy »

dahfizz wrote: post #54
chungy wrote: post #52

Earlier quoted context omitted.

Windows was designed around UCS-2, while it can be upwards compatible with storing UTF-16 data, the Windows APIs don't stop you from inserting almost any sequence of 16-bit words as file names, including unpaired surrogates (because Windows doesn't actually care about the contents of the 16-bit words). It just happens that most Windows programs will treat it as UTF-16 and it'll mostly be fine. Until you run into the…

Why is that a problem? That's how unix does it as well. A file path is just a sequence of bytes. Don't try to validate it in any sort of encoding, it's just bytes that the OS uses to identify a file.

The problem is that if the operating system allows sequences that are not valid UTF-16, then you cannot just state "it uses UTF-16": anything that is expecting UTF-16 and only UTF-16 is going to break on valid file names.

"That's how unix does it as well" is the exact same thing. Most people these days expect file names in UTF-8, but not all file systems restrict the valid character sequences to be UTF-8 compliant. If you treat everything as if it must be UTF-8, your application may just break when it encounters a name that isn't UTF-8. (This detail gets messy because there are configurations of, eg, ext4 and ZFS that limit valid file names to be valid UTF-8.)

Re: Unicode is harder than you think

#57

Post by jameshart »

Pannoniae wrote: post #21

Most programs claim to support Unicode but they actually don't. They either miscount string lengths (you type a CJK character or an emoji in, string appears shorter than what the program thinks), separate them improperly or many other things. It doesn't help that by default, most programming languages also handle unicode poorly, with the default APIs producing wrong results. I'd take "we don't do unicode at all" or "…

Why are people so interested in ‘lengths’ of strings?

The only thing anyone should actually care about is either the number of bytes it takes to store the string, or the number of pixels wide it is when rendered. Both of which are only loosely related to how many ‘characters’ or ‘grapheme clusters’ they contain, and which are themselves only vaguely correlated.

𒈙 (CUNEIFORM SIGN LUGAL OPPOSING LUGAL) is four bytes of UTF8 and as wide as about 9 Latin characters. 󠁧󠁢󠁥󠁮󠁧󠁿 (an England flag emoji) is about a character wide, but takes 28 bytes to store in UTF8.

Re: Unicode is harder than you think

#58

Post by ggm »

if you see URLs which project the microsoft editors choice of "single quote mark" into a triad of %xx; values, you've seen how people wind up in unexpected unicode/utf land, because of the text editor swapping ASCII for an uplift into either distorted iso-latin1, or unicode/utf8 depending.

Re: Unicode is harder than you think

#59

Post by OkayPhysicist »

chungy wrote: post #56
dahfizz wrote: post #54

Earlier quoted context omitted.

Why is that a problem? That's how unix does it as well. A file path is just a sequence of bytes. Don't try to validate it in any sort of encoding, it's just bytes that the OS uses to identify a file.

The problem is that if the operating system allows sequences that are not valid UTF-16, then you cannot just state "it uses UTF-16": anything that is expecting UTF-16 and only UTF-16 is going to break on valid file names. "That's how unix does it as well" is the exact same thing. Most people these days expect file names in UTF-8, but not all file systems restrict the valid character sequences to be UTF-8 compliant. I…

Does this actually matter? Who is out there, in the wild, creating file paths that use incredibly cursed random sequences of bytes? Maybe it's just me, but I don't really see the need to accommodate people who do stupid things to see what breaks.

Re: Unicode is harder than you think

#60

Post by alcover »

tialaramex wrote: post #53
alcover wrote: post #29

Currently working on a language, I feel dizzy after reading this. My stdlib will provide a (byte) Buffer class with basic low-level methods but I feel like iterating through it in fancy ways should be the concern of the user or 3rd-party libraries. I fail to see this as part of a programming language. Am I wrong here ?

People expect Strings. You don't have to provide strings, but you should be aware that people expect them and if you provide something else but don't clearly label it as not strings there's an excellent chance they'll go "Oh, strings" and then they'll complain that it doesn't work the way they expected. If I can write "foo".contains('o') or similar in your language, but then you pretend you don't have strings, nobody…

> If I can write "foo".contains('o') [..] but then you pretend you don't have strings, [..] C has strings

By your logic, couldn't I pretend then to have strings insomuch as C when "foo".contains('o') can be transpiled by my lang as

  strchr("foo", 'o') != NULL
Post reply on HN