How to chop off bytes of an UTF-8 string to fit into a small slot and look nice
1–10 of 66 posts
Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice
#2I cut my programmer teeth on Koha years and years ago. Still one of the warmest open-source communities I've ever been involved in, especially to a shy teenager with a lot of opinions.
Great to see new faces in the community, sad to see the sheer insanity of MARC21 still causing chaos. MARCXML is gonna make it obsolete Any Day Now!
Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice
#3Further reading:
* https://hoytech.github.io/truncate-presentation/ / https://metacpan.org/pod/Unicode::Truncate
* https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundarie...
Truncating at codepoint boundaries at least avoids generating invalid (non-UTF-8) strings, but can still result in confusing or incorrect displays for human readers, so for best results the truncation algorithm should take extended grapheme clusters into account, which are probably the closest thing that Unicode has to what most people think of as "characters".
Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice
#4Further reading: * https://hoytech.github.io/truncate-presentation/ / https://metacpan.org/pod/Unicode::Truncate * https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundarie... Truncating at codepoint boundaries at least avoids generating invalid (non-UTF-8) strings, but can still result in confusing or incorrect displays for human readers, so for best results the truncation algorithm should take extended grapheme…
To avoid this and a bunch of other confusion, when accepting user input I recommend normalizing it to the composed form before writing to a DB or file. While unicode-aware tools and software should handle either form just fine, you probably want to avoid that there's something in the pipeline somewhere that treats the decomposed and composed form of the same string as different.
Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice
#5Better use grapheme clusters than Unicode characters. After all, you don't want to chop the diaeresis off an ë.
Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice
#6Further reading: * https://hoytech.github.io/truncate-presentation/ / https://metacpan.org/pod/Unicode::Truncate * https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundarie... Truncating at codepoint boundaries at least avoids generating invalid (non-UTF-8) strings, but can still result in confusing or incorrect displays for human readers, so for best results the truncation algorithm should take extended grapheme…
To avoid this and a bunch of other confusion, when accepting user input I recommend normalizing it to the composed form before writing to a DB or file. While unicode-aware tools and software should handle either form just fine, you probably want to avoid that there's something in the pipeline somewhere that treats the decomposed and composed form of the same string as different.
It's good advice to normalise to pre-composed form, but that doesn't solve the problem the previous poster mentioned as not everything exists as a composed form. That said: most things do have a composed form, so you can probably get away with it – right up to when you can't.
Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice
#7Don't do this. Use a language (like C#) or library (like libunistring) that can do grapheme cluster segmentation. In .NET it's StringInfo.GetTextElementEnumerator(). In libunistring it's u8_grapheme_breaks(). In ICU4C it's icu::BreakIterator::createCharacterInstance(). In Ruby it's each_grapheme_cluster(). Other ecosystems with rich Unicode support should have similar functionality.
Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice
#8I cut my programmer teeth on Koha years and years ago. Still one of the warmest open-source communities I've ever been involved in, especially to a shy teenager with a lot of opinions. Great to see new faces in the community, sad to see the sheer insanity of MARC21 still causing chaos. MARCXML is gonna make it obsolete Any Day Now!
MARCXML is just a new format to encode what's the vast majority of the MARC 21 standard (or for that matter any other MARC variety).
BIBFRAME is gonna make it obsolete Any Day Now!
("any day now" in this sector means that librarians have been talking about it for two decades and in about two decades something might actually happen)
Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice
#9Earlier quoted context omitted.
To avoid this and a bunch of other confusion, when accepting user input I recommend normalizing it to the composed form before writing to a DB or file. While unicode-aware tools and software should handle either form just fine, you probably want to avoid that there's something in the pipeline somewhere that treats the decomposed and composed form of the same string as different.
It's good advice to normalise to pre-composed form, but that doesn't solve the problem the previous poster mentioned as not everything exists as a composed form. That said: most things do have a composed form, so you can probably get away with it – right up to when you can't.
Yeah, working an a library system our path was to compose everything (taking into account of course that the octet sizes specified in the directory may or may not actually be accurate depending on whatever system produced the record) and around the same time deprecate any pretense we had of supporting MARC-8.
Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice
#10Further reading: * https://hoytech.github.io/truncate-presentation/ / https://metacpan.org/pod/Unicode::Truncate * https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundarie... Truncating at codepoint boundaries at least avoids generating invalid (non-UTF-8) strings, but can still result in confusing or incorrect displays for human readers, so for best results the truncation algorithm should take extended grapheme…
For actual best results you’d probably want to truncate at the word or syllable boundary, and it should likely be language specific.