Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

domm.plix.at

Post reply on HN
Print view View original

21–30 of 66 posts

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#21

Post by Sharlin »

iforgotpassword wrote: post #4
re wrote: post #3

Further reading: * https://hoytech.github.io/truncate-presentation/ / https://metacpan.org/pod/Unicode::Truncate * https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundarie... Truncating at codepoint boundaries at least avoids generating invalid (non-UTF-8) strings, but can still result in confusing or incorrect displays for human readers, so for best results the truncation algorithm should take extended grapheme…

To avoid this and a bunch of other confusion, when accepting user input I recommend normalizing it to the composed form before writing to a DB or file. While unicode-aware tools and software should handle either form just fine, you probably want to avoid that there's something in the pipeline somewhere that treats the decomposed and composed form of the same string as different.

Even if you only support scripts for which Unicode has composed codepoints, these days you likely can’t get away without properly handling emoji, and there are no precomposed versions of all the numerous emojis that are made of multiple code points (eg. skin color and gender variants as well as flags).

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#22

Post by magicalhippo »

heinrich5991 wrote: post #14
magicalhippo wrote: post #13

Earlier quoted context omitted.

How does that affect filenames? IIRC the lower levels of Windows will happily work with filenames that are not valid Unicode strings, for example if you use the kernel API rather than Win32. But what about Win32? If you create a file before normalization and then open it using the normalized form, will it open the same file or return file not found? What about other systems? For example AWS' S3 allows UTF-8 keys, wit…

No need to go to the kernel API to create filenames that are invalid UTF-16. The Win32 API will happily let you do it.

I was AFK so couldn't check, but yeah you're right.

Just made two files named ä.txt which happily sat next to each other, one being NFC and other NFD.

So yeah, don't mess with the normalization of filenames.

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#23

Post by Aransentin »

There are some hard-to-handle edge cases when doing display length truncation in Unicode, e.g. the character U+FDFD or "﷽" is four bytes but can be very long depending on the typeface*, so "completely" solving it is quite hard and has to depend on feedback from your rasterization engine.

(*Rendered version on Wikipedia: https://commons.wikimedia.org/wiki/File:Lateef_unicode_U%2BF... )

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#25

Post by bux93 »

amelius wrote: post #20
LeonidasXIV wrote: post #12

This will probably fail if the thing being chopped off is a composed emoji, like the flag emoji (where it can chop off the second letter of the ISO code and just leave a bewildering to the user but completely valid first letter) or the ZWJ sequence emojis which will leave a color or half a family or other shenanigans, depending where it cuts.

Why is that a problem? If you cut off the country, at least you know that there was a flag. If you cut off the entire grapheme, then you know nothing!

Well, if the user entered a French flag, and then you show it back to them as a white flag, you may cause a bit of an international incident. Or worse, accusations of telling very old jokes.

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#26

Post by recursive »

mort96 wrote: post #19
iforgotpassword wrote: post #4

Earlier quoted context omitted.

To avoid this and a bunch of other confusion, when accepting user input I recommend normalizing it to the composed form before writing to a DB or file. While unicode-aware tools and software should handle either form just fine, you probably want to avoid that there's something in the pipeline somewhere that treats the decomposed and composed form of the same string as different.

The emoji "" can't be normalized further -- it's a "" followed by a "". If you just split on code points rather than grapheme clusters, even after normalizing, your naïve truncation algorithm will have accidentally changed the skin colors of emoji. Or turned the flag of Norway into an "". Or turned the rainbow flag into a white flag . EDIT: oh lord Hacker News strips emoji. You get the idea even though HN ruined the…

Presumably referring to Fitzpatrick modifiers.

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#27

Post by recursive »

amelius wrote: post #20
LeonidasXIV wrote: post #12

This will probably fail if the thing being chopped off is a composed emoji, like the flag emoji (where it can chop off the second letter of the ISO code and just leave a bewildering to the user but completely valid first letter) or the ZWJ sequence emojis which will leave a color or half a family or other shenanigans, depending where it cuts.

Why is that a problem? If you cut off the country, at least you know that there was a flag. If you cut off the entire grapheme, then you know nothing!

I think you'd just get the first letter of the country code and not a flag at all.

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#28

Post by moefh »

magicalhippo wrote: post #13
iforgotpassword wrote: post #4

Earlier quoted context omitted.

To avoid this and a bunch of other confusion, when accepting user input I recommend normalizing it to the composed form before writing to a DB or file. While unicode-aware tools and software should handle either form just fine, you probably want to avoid that there's something in the pipeline somewhere that treats the decomposed and composed form of the same string as different.

How does that affect filenames? IIRC the lower levels of Windows will happily work with filenames that are not valid Unicode strings, for example if you use the kernel API rather than Win32. But what about Win32? If you create a file before normalization and then open it using the normalized form, will it open the same file or return file not found? What about other systems? For example AWS' S3 allows UTF-8 keys, wit…

> What about other systems?

The Linux kernel doesn't validate filenames in any way, so a filename in Linux can contain any byte except 0x2F ('/', which is interpreted as directory separator) and 0x00 (which signals the end of the byte string).

ETA: of course some file systems have other limitations, for example '\' is not valid in FAT32.

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#29

Post by tingletech »

mort96 wrote: post #19
iforgotpassword wrote: post #4

Earlier quoted context omitted.

To avoid this and a bunch of other confusion, when accepting user input I recommend normalizing it to the composed form before writing to a DB or file. While unicode-aware tools and software should handle either form just fine, you probably want to avoid that there's something in the pipeline somewhere that treats the decomposed and composed form of the same string as different.

The emoji "" can't be normalized further -- it's a "" followed by a "". If you just split on code points rather than grapheme clusters, even after normalizing, your naïve truncation algorithm will have accidentally changed the skin colors of emoji. Or turned the flag of Norway into an "". Or turned the rainbow flag into a white flag . EDIT: oh lord Hacker News strips emoji. You get the idea even though HN ruined the…

HN is not broken, it's working as designed.

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#30

Post by tingletech »

quink wrote: post #8
pianohacker wrote: post #2

I cut my programmer teeth on Koha years and years ago. Still one of the warmest open-source communities I've ever been involved in, especially to a shy teenager with a lot of opinions. Great to see new faces in the community, sad to see the sheer insanity of MARC21 still causing chaos. MARCXML is gonna make it obsolete Any Day Now!

MARCXML is just a new format to encode what's the vast majority of the MARC 21 standard (or for that matter any other MARC variety). BIBFRAME is gonna make it obsolete Any Day Now! ("any day now" in this sector means that librarians have been talking about it for two decades and in about two decades something might actually happen)

I do wish the community had leaned into the MODS direction vs going over to RDF.
Post reply on HN