How to chop off bytes of an UTF-8 string to fit into a small slot and look nice

Post by postmodest » Wed, Jun 05, 2024, 1:53 PM UTC

tingletech wrote: post #29

mort96 wrote: post #19

Earlier quoted context omitted.

The emoji "" can't be normalized further -- it's a "" followed by a "". If you just split on code points rather than grapheme clusters, even after normalizing, your naïve truncation algorithm will have accidentally changed the skin colors of emoji. Or turned the flag of Norway into an "". Or turned the rainbow flag into a white flag . EDIT: oh lord Hacker News strips emoji. You get the idea even though HN ruined the…

HN is not broken, it's working as designed.

Post by nordsieck » Wed, Jun 05, 2024, 2:31 PM UTC

account42 wrote: post #15

iforgotpassword wrote: post #4

Earlier quoted context omitted.

To avoid this and a bunch of other confusion, when accepting user input I recommend normalizing it to the composed form before writing to a DB or file. While unicode-aware tools and software should handle either form just fine, you probably want to avoid that there's something in the pipeline somewhere that treats the decomposed and composed form of the same string as different.

Not all grapheme clusters have composed forms so normalization doesn't actually gain you anything here.

Post by memco » Wed, Jun 05, 2024, 2:32 PM UTC

electroly wrote: post #7

Don't do this. Use a language (like C#) or library (like libunistring) that can do grapheme cluster segmentation. In .NET it's StringInfo.GetTextElementEnumerator(). In libunistring it's u8_grapheme_breaks(). In ICU4C it's icu::BreakIterator::createCharacterInstance(). In Ruby it's each_grapheme_cluster(). Other ecosystems with rich Unicode support should have similar functionality.

Post by simonw » Wed, Jun 05, 2024, 2:45 PM UTC

electroly wrote: post #7

Don't do this. Use a language (like C#) or library (like libunistring) that can do grapheme cluster segmentation. In .NET it's StringInfo.GetTextElementEnumerator(). In libunistring it's u8_grapheme_breaks(). In ICU4C it's icu::BreakIterator::createCharacterInstance(). In Ruby it's each_grapheme_cluster(). Other ecosystems with rich Unicode support should have similar functionality.

    import regex as re
    
    def grapheme_clusters(text):
        # \X is the regex pattern that matches a grapheme cluster
        pattern = re.compile(r'\X')
        return [
            match.group(0)
            for match in
            pattern.finditer(text)
        ]

Post by neonsunset » Wed, Jun 05, 2024, 3:02 PM UTC

memco wrote: post #36

electroly wrote: post #7

Don't do this. Use a language (like C#) or library (like libunistring) that can do grapheme cluster segmentation. In .NET it's StringInfo.GetTextElementEnumerator(). In libunistring it's u8_grapheme_breaks(). In ICU4C it's icu::BreakIterator::createCharacterInstance(). In Ruby it's each_grapheme_cluster(). Other ecosystems with rich Unicode support should have similar functionality.

That’s fine unless you are a language or library creator in which case knowing how to do it properly can’t be deferred to someone else. Perhaps porting someone else’s correct implementation is good but someone somewhere has to implement this. If they don’t share their knowledge this will always be esoteric knowledge locked away unless those who do that kind of work share their knowledge and experience. Most of us are…