Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why can't you reverse a string with a flag emoji?

davidamos.dev

Post reply on HN
Print view View original

181–190 of 247 posts

Re: Why can't you reverse a string with a flag emoji?

#181

Post by emodendroket »

shalmanese wrote: post #163
emodendroket wrote: post #114

What I'd like to know is, given the explosion of the character set for emoji, does the rationale for Han unification still make sense? The case for not allowing national variants seems less and less compelling with every emoji they add. This is a bit of a hobby horse, but imagine if every time you read an article in English on your phone some of the letters were replaced with "equivalent" Greek or Cyrillic one and yo…

It doesn't make sense but there's also no way to fix it now. Once the Han characters were unified, there's no non-trivial way to ununify them.

To an extent that's true, but introducing national variant characters in addition to the unified ones would at least allow careful writers to avoid the problem.

Re: Why can't you reverse a string with a flag emoji?

#182

Post by kevin_thibedeau »

masklinn wrote: post #94
jug wrote: post #74

Earlier quoted context omitted.

I think that somewhere in this answer lies a reason why Windows still doesn't support flag emoji. I don't count Microsoft Edge as "Windows" in this case, but as Chromium. Windows doesn't support flag emoji in its native text boxes, but it does support even colorized emoji. But then again, flags seem to be not only Unicode-hard but post-Unicode-hard.

> But then again, flags seem to be not only Unicode-hard but post-Unicode-hard. Flags are not that hard, they're a very specific block combining in very predictable way. They're little more than ligatures. Family emoji are much harder. And this is not "post-Unicode" in any way.

Consider you have to split a string with 20 flags in sequence at a given offset. That's 40 codepoints with no readily discernible boundaries. To parse that you have to scan backwards to find the first non-flag codepoint. Otherwise you could split the middle of a flag pair. You also have to handle rendering invalid combinations as two glyphs and unpaired codes. For normal codepoints with combining characters you can scan forwards until you reach a non-combining character.

Re: Why can't you reverse a string with a flag emoji?

#183

Post by fomine3 »

lmm wrote: post #176
saltminer wrote: post #157

Earlier quoted context omitted.

> WTF business do emojis have in Unicode? Unicode didn't invent emoji, they incorporated it because they were already popular in Japan, and if they didn't incorporate it, it would greatly reduce Japanese adoption. Keep in mind that Unicode was intended to unify all the disparate encodings that had been brewed up to support different languages and which made exchanging documents between non-English speaking countries…

> The term "mojibake" comes to mind [0] - Japan alone had so many encodings that a slang term for text encoded with something different than what your device expected (and subsequently got rendered as nonsensical/garbled text) came about. Mojibake was not a "Japan has too many encodings" problem. It was a "western developers assume everyone is using CP1252" problem. > Unicode wasn't intended to be pretty. It was inte…

Mojibake is a universal problem when multiple charset is used and there are no charset specification on metadata. Software guess charset but it's just a guess. Japanese locale software occasionally confuses Latin-1 vs SJIS but often confuses SJIS vs EUC-JP or UTF-8.

Unicode/UTF-8 is widely adopted/recommended in Japan and there are no widely used alternative. Japanese company tend to still use SJIS but it's just laziness. Han unification isn't a problem to handle only Japanese text: just use Japanese font everywhere. To handle multiple language text, it's pain but anyway there are no alternatives.

Re: Why can't you reverse a string with a flag emoji?

#184

Post by coreyp_1 »

DecoPerson wrote: post #168
coreyp_1 wrote: post #53

If you think the Unicode flag emoji take a lot of bytes, then consider the family emoji! ( https://unicode.org/emoji/charts/full-emoji-list.html#family ) I'm in the process of designing a scripting language and implementing it in C++. I plan to put together a YouTube series about it. (Doesn't everyone want to see Bison and Flex mixed with proper unit tests and C++20 code?) Due to my future intended use case, I needed…

Do you have a YouTube for people to subscribe to in anticipation of you releasing your YouTube series about your work? The development processes of new languages is so intriguing.

I'll post about it here on HN after I have a few episodes up.

Re: Why can't you reverse a string with a flag emoji?

#186

Post by lmm »

fomine3 wrote: post #183
lmm wrote: post #176

Earlier quoted context omitted.

> The term "mojibake" comes to mind [0] - Japan alone had so many encodings that a slang term for text encoded with something different than what your device expected (and subsequently got rendered as nonsensical/garbled text) came about. Mojibake was not a "Japan has too many encodings" problem. It was a "western developers assume everyone is using CP1252" problem. > Unicode wasn't intended to be pretty. It was inte…

Mojibake is a universal problem when multiple charset is used and there are no charset specification on metadata. Software guess charset but it's just a guess. Japanese locale software occasionally confuses Latin-1 vs SJIS but often confuses SJIS vs EUC-JP or UTF-8. Unicode/UTF-8 is widely adopted/recommended in Japan and there are no widely used alternative. Japanese company tend to still use SJIS but it's just lazi…

> Mojibake is a universal problem when multiple charset is used and there are no charset specification on metadata. Software guess charset but it's just a guess. Japanese locale software occasionally confuses Latin-1 vs SJIS but often confuses SJIS vs EUC-JP or UTF-8.

In theory it can happen with any combination of character sets, sure, but in practice every example of mojibake I've seen has been SJIS (or UTF-8) encoded text being decoded as CP1252 ("Latin-1" but that's an ambiguous term) by software that assumed the whole world used that particular western codepage. If you've got examples of SJIS vs EUC-JP confusion in the wild I'd be vaguely interested to see them (is there even anywhere that still uses EUC-JP?)

> Japanese company tend to still use SJIS but it's just laziness.

It's not just laziness; switching to unicode is a downgrade, because in practice it means you're going to get your characters rendered wrongly (Chinese style) for a certain percentage of your customers, for little clear benefit.

> To handle multiple language text, it's pain but anyway there are no alternatives.

Given that you have to build a structure that looks like a sequence of spans with language metadata attached to each one, there's not much benefit to using unicode versus letting each span specify its own encoding.

Re: Why can't you reverse a string with a flag emoji?

#187

Post by johndough »

amelius wrote: post #156
coreyp_1 wrote: post #53

If you think the Unicode flag emoji take a lot of bytes, then consider the family emoji! ( https://unicode.org/emoji/charts/full-emoji-list.html#family ) I'm in the process of designing a scripting language and implementing it in C++. I plan to put together a YouTube series about it. (Doesn't everyone want to see Bison and Flex mixed with proper unit tests and C++20 code?) Due to my future intended use case, I needed…

Why is this stuff even reinvented for every programming language? Isn't it about time that we have some common language that every other language builds on?

> Isn't it about time that we have some common language that every other language builds on?

That language is C. It is debatable whether it was a good choice, but at least this is how it turned.

Re: Why can't you reverse a string with a flag emoji?

#188

Post by emodendroket »

fomine3 wrote: post #174
emodendroket wrote: post #114

What I'd like to know is, given the explosion of the character set for emoji, does the rationale for Han unification still make sense? The case for not allowing national variants seems less and less compelling with every emoji they add. This is a bit of a hobby horse, but imagine if every time you read an article in English on your phone some of the letters were replaced with "equivalent" Greek or Cyrillic one and yo…

Han unification was a try to fit CJK characters into 16bit BMP. Finally BMP is failed so meaningless but reverting it also produces huge compatibility issue.

Of course, the old characters must be left alone. But I'm not seeing what stops them from introducing new ones.

Re: Why can't you reverse a string with a flag emoji?

#189

Post by nitely »

xmprt wrote: post #12

This is a cool article about Unicode encoding however I still feel like it should be possible to reverse strings with Flag emojis. I don't see why computers can't handle multi rune symbols in the same way that they handle multi byte runes. We could combine all the runes that should be a single symbol and make sure that we're maintaining the ordering of those runes in the reversed string. Of course that means that nai…

Of course it's possible, the Unicode standard even has a table[0] you can use to build a DFA (Deterministic Finite Automata) to break up a string into grapheme clusters. You can reverse the DFA to match and yield the graphemes backwards as well, which will give you the reversed unicode string.

[0]: http://www.unicode.org/reports/tr29/#Table_Combining_Char_Se...

Re: Why can't you reverse a string with a flag emoji?

#190

Post by nitely »

happytoexplain wrote: post #24
xmprt wrote: post #12

This is a cool article about Unicode encoding however I still feel like it should be possible to reverse strings with Flag emojis. I don't see why computers can't handle multi rune symbols in the same way that they handle multi byte runes. We could combine all the runes that should be a single symbol and make sure that we're maintaining the ordering of those runes in the reversed string. Of course that means that nai…

Swift, for example, does what you're saying. I thought that the reason many languages don't do it that way is that part of the definition of an array (or at least expected-by-convention) is constant-time operations. If you treat a string as an array, then having to deal with variable-length units breaks that rule. That's why, when there is an API for dealing with grapheme clusters, it is usually a special case that d…

CPython 3 does use UTF-32 under the hood for strings (there is bytes for plain sequence of bytes). As you say, it's the worst of both worlds. High memory usage, and not really useful if you are dealing with unicode characters (grapheme clusters).

My impression is most modern languages that bother with unicode (swift, rust, nim) are using utf-8, and doing linear time operations to handle unicode. I think that's the right approach, as I don't recall ever needing random access on a unicode string.

Post reply on HN