Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

domm.plix.at

Post reply on HN
Print view View original

61–66 of 66 posts

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#61

Post by CRConrad »

asabil wrote: post #58
gmueckl wrote: post #57

Earlier quoted context omitted.

Randomly truncating words can have the same effect in any language. It's outright trivial to find examples in English or German. I don't understand why one has to invoke Arab script for a good example.

Yes, but you don’t end up with different glyphs. Arabic script has letter shaping, that means a letter can have up to 4 shapes based on its position within the word. If you chop off the last letter, the previous one which used to have a “middle” position shape suddenly changes into “terminal” position shape.

I'm thinking even bog-standard European umlauts, cedillas, etc go multi-byte in Unicode? (Take a string of ÅÄÖåäöÜü and chop it off at various byte limits and see.)

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#62

Post by CRConrad »

magicalhippo wrote: post #22
heinrich5991 wrote: post #14

Earlier quoted context omitted.

No need to go to the kernel API to create filenames that are invalid UTF-16. The Win32 API will happily let you do it.

I was AFK so couldn't check, but yeah you're right. Just made two files named ä.txt which happily sat next to each other, one being NFC and other NFD. So yeah, don't mess with the normalization of filenames.

But ä.txt is a perfectly normal filename! WTF is there to "normalize" about that?!?

Äh, nu går vi och gör något annat.

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#64

Post by panzi »

CRConrad wrote: post #61
asabil wrote: post #58

Earlier quoted context omitted.

Yes, but you don’t end up with different glyphs. Arabic script has letter shaping, that means a letter can have up to 4 shapes based on its position within the word. If you chop off the last letter, the previous one which used to have a “middle” position shape suddenly changes into “terminal” position shape.

I'm thinking even bog-standard European umlauts, cedillas, etc go multi-byte in Unicode? (Take a string of ÅÄÖåäöÜü and chop it off at various byte limits and see.)

I don't remember the details, only that it was a bigger deal than with umlauts. I'll see if I can find the talk again.

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#65

Post by gmueckl »

CRConrad wrote: post #61
asabil wrote: post #58

Earlier quoted context omitted.

Yes, but you don’t end up with different glyphs. Arabic script has letter shaping, that means a letter can have up to 4 shapes based on its position within the word. If you chop off the last letter, the previous one which used to have a “middle” position shape suddenly changes into “terminal” position shape.

I'm thinking even bog-standard European umlauts, cedillas, etc go multi-byte in Unicode? (Take a string of ÅÄÖåäöÜü and chop it off at various byte limits and see.)

This is just the general behavior of truncating strings by code point when they contain decomposed glyphs. This can also impact accents etc.

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#66

Post by magicalhippo »

CRConrad wrote: post #62
magicalhippo wrote: post #22

Earlier quoted context omitted.

I was AFK so couldn't check, but yeah you're right. Just made two files named ä.txt which happily sat next to each other, one being NFC and other NFD. So yeah, don't mess with the normalization of filenames.

But ä.txt is a perfectly normal filename! WTF is there to "normalize" about that?!? Äh, nu går vi och gör något annat.

Either ä as a single code unit or a + ¨ as two code units.

Win32/NTFS treats them as two different filenames, as it doesn't normalize them before storing/comparing.

Post reply on HN