Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

domm.plix.at

Post reply on HN
Print view View original

41–50 of 66 posts

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#41

Post by AlotOfReading »

simonw wrote: post #38
electroly wrote: post #7

Don't do this. Use a language (like C#) or library (like libunistring) that can do grapheme cluster segmentation. In .NET it's StringInfo.GetTextElementEnumerator(). In libunistring it's u8_grapheme_breaks(). In ICU4C it's icu::BreakIterator::createCharacterInstance(). In Ruby it's each_grapheme_cluster(). Other ecosystems with rich Unicode support should have similar functionality.

I pasted your comment here into GPT-4o and asked for the Python equivalent, it suggested this which seems to work well: import regex as re def grapheme_clusters(text): # \X is the regex pattern that matches a grapheme cluster pattern = re.compile(r'\X') return [ match.group(0) for match in pattern.finditer(text) ] https://chatgpt.com/share/481c9c94-0431-4fcb-82aa-a44a4f3c21...

Note that regex is not the re module from the stdlib, it's a separate third party module that exposes the more powerful capabilities of PCRE like grapheme clustering directly.

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#42

Post by electroly »

memco wrote: post #36
electroly wrote: post #7

Don't do this. Use a language (like C#) or library (like libunistring) that can do grapheme cluster segmentation. In .NET it's StringInfo.GetTextElementEnumerator(). In libunistring it's u8_grapheme_breaks(). In ICU4C it's icu::BreakIterator::createCharacterInstance(). In Ruby it's each_grapheme_cluster(). Other ecosystems with rich Unicode support should have similar functionality.

That’s fine unless you are a language or library creator in which case knowing how to do it properly can’t be deferred to someone else. Perhaps porting someone else’s correct implementation is good but someone somewhere has to implement this. If they don’t share their knowledge this will always be esoteric knowledge locked away unless those who do that kind of work share their knowledge and experience. Most of us are…

As it turns out, I am writing my own language, and my language supports grapheme cluster segmentation. I just used libunistring (and before that, I used ICU). TFA is not doing this correctly at all; the Unicode specification provides the rules for grapheme cluster segmentation if you wish to implement it yourself[0]. There's nothing to be learned from TFA's hacky and fundamentally incorrect approach. OP's technique will freely chop combining code points that needed to be kept.

[0] https://unicode.org/reports/tr29/

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#43

Post by mort96 »

tingletech wrote: post #29
mort96 wrote: post #19

Earlier quoted context omitted.

The emoji "" can't be normalized further -- it's a "" followed by a "". If you just split on code points rather than grapheme clusters, even after normalizing, your naïve truncation algorithm will have accidentally changed the skin colors of emoji. Or turned the flag of Norway into an "". Or turned the rainbow flag into a white flag . EDIT: oh lord Hacker News strips emoji. You get the idea even though HN ruined the…

HN is not broken, it's working as designed.

It makes technical conversations about Unicode ridiculously annoying. It's working as designed and the design is broken.

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#44

Post by electroly »

daneel_w wrote: post #35
electroly wrote: post #7

Don't do this. Use a language (like C#) or library (like libunistring) that can do grapheme cluster segmentation. In .NET it's StringInfo.GetTextElementEnumerator(). In libunistring it's u8_grapheme_breaks(). In ICU4C it's icu::BreakIterator::createCharacterInstance(). In Ruby it's each_grapheme_cluster(). Other ecosystems with rich Unicode support should have similar functionality.

"I had a problem, and here's my working solution for my specific case." -"Don't do this. Instead use a completely different programming language."

In Perl (OP's chosen language) you can use the Unicode::Util package. That's why I was pretty clear that you can use a different language or a different library. This seems to be a pretty uncharitable reading of my post. Use the right tool for the job.

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#45

Post by account42 »

Aransentin wrote: post #23

There are some hard-to-handle edge cases when doing display length truncation in Unicode, e.g. the character U+FDFD or "﷽" is four bytes but can be very long depending on the typeface*, so "completely" solving it is quite hard and has to depend on feedback from your rasterization engine. (*Rendered version on Wikipedia: https://commons.wikimedia.org/wiki/File:Lateef_unicode_U%2BF... )

This is a completely unrelated problem since the article is quite clearly about limiting to a certain maximum byte length and not display length. For display length you don't even need Unicode for that to depend on the font and shaping engine.

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#46

Post by lisper »

mort96 wrote: post #43
tingletech wrote: post #29

Earlier quoted context omitted.

HN is not broken, it's working as designed.

It makes technical conversations about Unicode ridiculously annoying. It's working as designed and the design is broken.

Whether the absence of emojis on HN is a feature or a bug is arguable. But if you can't figure out a way to work around this constraint (e.g. put your example literally anywhere else on the web and post a link here) HN is probably not a good fit for you.

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#47

Post by mort96 »

lisper wrote: post #46
mort96 wrote: post #43

Earlier quoted context omitted.

It makes technical conversations about Unicode ridiculously annoying. It's working as designed and the design is broken.

Whether the absence of emojis on HN is a feature or a bug is arguable. But if you can't figure out a way to work around this constraint (e.g. put your example literally anywhere else on the web and post a link here) HN is probably not a good fit for you.

Reading a discussion thread where each message is just a link to some pastebin with the actual message isn't very nice. Besides, I wasn't going to write the message again after HN removed arbitrary parts of it, hence the edit; I think people got the gist. You may feel that discussion about Unicode doesn't belong on HN but I feel otherwise.

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#48

Post by panzi »

nordsieck wrote: post #34
account42 wrote: post #15

Earlier quoted context omitted.

Not all grapheme clusters have composed forms so normalization doesn't actually gain you anything here.

> Not all grapheme clusters have composed forms so normalization doesn't actually gain you anything here. Just because the worst case can't improve doesn't mean that making the average case better is worthless.

I saw something about Arabic text, where that naive truncation at codepoint boundaries turns one word into a different word! Like the sequence of codepoints generate something that is represented as a single glyph in fonts, but truncated its totally different glyphs. I don't remember more details, I don't know any Arabic, but grapheme clusters aren't just about adding diacritics to latin characters. In other languages it all might work quite differently. So truncating at word boundaries (at breakable white-space or punctuation) is probably best. Though of course that way you might truncate the string by a lot. shrug-emoji

(I don't think the talk where the stuff about Arabic was mentioned was Plain Text by Dylan Beattie, but I haven't re-watched it to confirm. So maybe it is. Can't remember the name of any other talk about the subject right now.)

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#49

Post by lisper »

mort96 wrote: post #47
lisper wrote: post #46

Earlier quoted context omitted.

Whether the absence of emojis on HN is a feature or a bug is arguable. But if you can't figure out a way to work around this constraint (e.g. put your example literally anywhere else on the web and post a link here) HN is probably not a good fit for you.

Reading a discussion thread where each message is just a link to some pastebin with the actual message isn't very nice. Besides, I wasn't going to write the message again after HN removed arbitrary parts of it, hence the edit; I think people got the gist. You may feel that discussion about Unicode doesn't belong on HN but I feel otherwise.

Reading discussion threads full of silly emojis isn't "very nice" either, at least for a certain kind of audience. It's a tradeoff, and the powers that be at HN have decided to optimize for sober discussion over expressivity. It's a defensible decision. Keeping HN from degenerating into Reddit is already hard enough.

Re: How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

#50

Post by AlienRobot »

mort96 wrote: post #19
iforgotpassword wrote: post #4

Earlier quoted context omitted.

To avoid this and a bunch of other confusion, when accepting user input I recommend normalizing it to the composed form before writing to a DB or file. While unicode-aware tools and software should handle either form just fine, you probably want to avoid that there's something in the pipeline somewhere that treats the decomposed and composed form of the same string as different.

The emoji "" can't be normalized further -- it's a "" followed by a "". If you just split on code points rather than grapheme clusters, even after normalizing, your naïve truncation algorithm will have accidentally changed the skin colors of emoji. Or turned the flag of Norway into an "". Or turned the rainbow flag into a white flag . EDIT: oh lord Hacker News strips emoji. You get the idea even though HN ruined the…

More sites should have this bug. Whenever I see a colored icon in the middle of black text I get inexplicably angry.
Post reply on HN