I wrote a JavaScript version of this using Intl: https://github.com/GoogleChrome/lighthouse/blob/9baac0ae9da7...
How to chop off bytes of an UTF-8 string to fit into a small slot and look nice
11–20 of 66 posts
Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice
#12This will probably fail if the thing being chopped off is a composed emoji, like the flag emoji (where it can chop off the second letter of the ISO code and just leave a bewildering to the user but completely valid first letter) or the ZWJ sequence emojis which will leave a color or half a family or other shenanigans, depending where it cuts.
Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice
#13Further reading: * https://hoytech.github.io/truncate-presentation/ / https://metacpan.org/pod/Unicode::Truncate * https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundarie... Truncating at codepoint boundaries at least avoids generating invalid (non-UTF-8) strings, but can still result in confusing or incorrect displays for human readers, so for best results the truncation algorithm should take extended grapheme…
To avoid this and a bunch of other confusion, when accepting user input I recommend normalizing it to the composed form before writing to a DB or file. While unicode-aware tools and software should handle either form just fine, you probably want to avoid that there's something in the pipeline somewhere that treats the decomposed and composed form of the same string as different.
How does that affect filenames?
IIRC the lower levels of Windows will happily work with filenames that are not valid Unicode strings, for example if you use the kernel API rather than Win32.
But what about Win32? If you create a file before normalization and then open it using the normalized form, will it open the same file or return file not found?
What about other systems? For example AWS' S3 allows UTF-8 keys, with no mention of normalization[1].
On the phone so can't try myself right now.
Anyway for general text I agree, but for identifiers, filenames and such I prefer to treat them as opaquely as possible.
[1]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object...
Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice
#14Earlier quoted context omitted.
To avoid this and a bunch of other confusion, when accepting user input I recommend normalizing it to the composed form before writing to a DB or file. While unicode-aware tools and software should handle either form just fine, you probably want to avoid that there's something in the pipeline somewhere that treats the decomposed and composed form of the same string as different.
How does that affect filenames? IIRC the lower levels of Windows will happily work with filenames that are not valid Unicode strings, for example if you use the kernel API rather than Win32. But what about Win32? If you create a file before normalization and then open it using the normalized form, will it open the same file or return file not found? What about other systems? For example AWS' S3 allows UTF-8 keys, wit…
No need to go to the kernel API to create filenames that are invalid UTF-16. The Win32 API will happily let you do it.
Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice
#15Further reading: * https://hoytech.github.io/truncate-presentation/ / https://metacpan.org/pod/Unicode::Truncate * https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundarie... Truncating at codepoint boundaries at least avoids generating invalid (non-UTF-8) strings, but can still result in confusing or incorrect displays for human readers, so for best results the truncation algorithm should take extended grapheme…
To avoid this and a bunch of other confusion, when accepting user input I recommend normalizing it to the composed form before writing to a DB or file. While unicode-aware tools and software should handle either form just fine, you probably want to avoid that there's something in the pipeline somewhere that treats the decomposed and composed form of the same string as different.
Not all grapheme clusters have composed forms so normalization doesn't actually gain you anything here.
Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice
#16Earlier quoted context omitted.
To avoid this and a bunch of other confusion, when accepting user input I recommend normalizing it to the composed form before writing to a DB or file. While unicode-aware tools and software should handle either form just fine, you probably want to avoid that there's something in the pipeline somewhere that treats the decomposed and composed form of the same string as different.
How does that affect filenames? IIRC the lower levels of Windows will happily work with filenames that are not valid Unicode strings, for example if you use the kernel API rather than Win32. But what about Win32? If you create a file before normalization and then open it using the normalized form, will it open the same file or return file not found? What about other systems? For example AWS' S3 allows UTF-8 keys, wit…
[deleted]
Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice
#17[flagged]
Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice
#18[flagged]
[deleted]
Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice
#19Further reading: * https://hoytech.github.io/truncate-presentation/ / https://metacpan.org/pod/Unicode::Truncate * https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundarie... Truncating at codepoint boundaries at least avoids generating invalid (non-UTF-8) strings, but can still result in confusing or incorrect displays for human readers, so for best results the truncation algorithm should take extended grapheme…
To avoid this and a bunch of other confusion, when accepting user input I recommend normalizing it to the composed form before writing to a DB or file. While unicode-aware tools and software should handle either form just fine, you probably want to avoid that there's something in the pipeline somewhere that treats the decomposed and composed form of the same string as different.
The emoji "" can't be normalized further -- it's a "" followed by a "". If you just split on code points rather than grapheme clusters, even after normalizing, your naïve truncation algorithm will have accidentally changed the skin colors of emoji. Or turned the flag of Norway into an "". Or turned the rainbow flag into a white flag .
EDIT: oh lord Hacker News strips emoji. You get the idea even though HN ruined the illustrations. Not my fault HN is broken.
Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice
#20This will probably fail if the thing being chopped off is a composed emoji, like the flag emoji (where it can chop off the second letter of the ISO code and just leave a bewildering to the user but completely valid first letter) or the ZWJ sequence emojis which will leave a color or half a family or other shenanigans, depending where it cuts.
Why is that a problem? If you cut off the country, at least you know that there was a flag. If you cut off the entire grapheme, then you know nothing!