Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why can't you reverse a string with a flag emoji?

davidamos.dev

Post reply on HN
Print view View original

211–220 of 247 posts

Re: Why can't you reverse a string with a flag emoji?

#211

Post by zarzavat »

zarzavat wrote: post #175

This reminds me of an interesting bug I saw where I was seeing a strange flag in some Arabic text. However when I copied the string and pasted it into a text editor, the flag of Saudi Arabia appeared instead (which made much more sense). After some vexillologic research on Wikipedia I identified the original flag as American Samoa and it suddenly all made sense. Turns out some broken RTL support was flipping the SA i…

After writing this comment I did some more research. Apparently this is actually a bug in Chrome itself (!).

https://bugs.chromium.org/p/chromium/issues/detail?id=127243...

Re: Why can't you reverse a string with a flag emoji?

#212

Post by account42 »

dagmx wrote: post #204
coreyp_1 wrote: post #53

If you think the Unicode flag emoji take a lot of bytes, then consider the family emoji! ( https://unicode.org/emoji/charts/full-emoji-list.html#family ) I'm in the process of designing a scripting language and implementing it in C++. I plan to put together a YouTube series about it. (Doesn't everyone want to see Bison and Flex mixed with proper unit tests and C++20 code?) Due to my future intended use case, I needed…

It would actually be pretty interesting to see how you use Bison and Flex with utf-8. Most resources say to not bother due to lack of support for Unicode, but they're so ubiquitous

Do they need special support for UTF-8? One of the nice things about UTF-8 is that you can treat it as an 8-bit encoding in many cases if you only care about substrings and don't need to decode individual non-ASCII characters.

Re: Why can't you reverse a string with a flag emoji?

#213

Post by numpad0 »

lmm wrote: post #186
fomine3 wrote: post #183

Earlier quoted context omitted.

Mojibake is a universal problem when multiple charset is used and there are no charset specification on metadata. Software guess charset but it's just a guess. Japanese locale software occasionally confuses Latin-1 vs SJIS but often confuses SJIS vs EUC-JP or UTF-8. Unicode/UTF-8 is widely adopted/recommended in Japan and there are no widely used alternative. Japanese company tend to still use SJIS but it's just lazi…

> Mojibake is a universal problem when multiple charset is used and there are no charset specification on metadata. Software guess charset but it's just a guess. Japanese locale software occasionally confuses Latin-1 vs SJIS but often confuses SJIS vs EUC-JP or UTF-8. In theory it can happen with any combination of character sets, sure, but in practice every example of mojibake I've seen has been SJIS (or UTF-8) enco…

I think by far the largest contributor that coined mojibake was E-mail MTA. Some E-mail implementations assumed 7-bit ASCII for all text and dropped MSB on 8-bit SJIS/Unicode/etc, ending up as corrupt text at the receiving end. Next up was texts written in EUC(Extended UNIX Code)-JP probably by someone either running a real Unix(likely a Solaris) or early GNU/Linux, and floppies from a classic MacOS computer. Those must have defined it and various edge cases on web like header-encoding mismatch popularized it.

"Zhonghua fonts" issue is not necessarily linked to encoding, it's an issue about assuming or guessing locales - that has to be solved by adding a language identifier or by ending han unification.

Re: Why can't you reverse a string with a flag emoji?

#214

Post by account42 »

lmm wrote: post #176
saltminer wrote: post #157

Earlier quoted context omitted.

> WTF business do emojis have in Unicode? Unicode didn't invent emoji, they incorporated it because they were already popular in Japan, and if they didn't incorporate it, it would greatly reduce Japanese adoption. Keep in mind that Unicode was intended to unify all the disparate encodings that had been brewed up to support different languages and which made exchanging documents between non-English speaking countries…

> The term "mojibake" comes to mind [0] - Japan alone had so many encodings that a slang term for text encoded with something different than what your device expected (and subsequently got rendered as nonsensical/garbled text) came about. Mojibake was not a "Japan has too many encodings" problem. It was a "western developers assume everyone is using CP1252" problem. > Unicode wasn't intended to be pretty. It was inte…

> Unfortunately they undermined all that with Han Unification, with the result that it's never going to be adopted in Japan.

This is an absolute shame and there is no excuse for fixing it so that variations for unified characters can be encoded before adding unimportant things like skin tones.

Re: Why can't you reverse a string with a flag emoji?

#215

Post by account42 »

lmm wrote: post #177
Findecanor wrote: post #160

Earlier quoted context omitted.

I don't think that argument holds water. Emoji could just as well have been encoded as markup. There were for instance long-established conventions of using strings starting with : and ; . Bulletin boards extended that to a convention using letters delimited by : for example :rolleyes: . Not to mention that those codes can be typed more efficiently than browsing in an Emoji Picker box. Because emoji became characters…

> Emoji could just as well have been encoded as markup. They could have, but they were already being encoded as character codepoints in existing charactersets. So any character encoding scheme that wanted to replace all use cases for existing charactersets needed to match that. If switching charactersets meant you lost the ability to use emoji until you upgraded all your applications to support some markup format, pe…

> If switching charactersets meant you lost the ability to use emoji until you upgraded all your applications to support some markup format, people would just not switch.

You need to upgrade those applications to support Unicode too.

Re: Why can't you reverse a string with a flag emoji?

#216

Post by numpad0 »

Gigachad wrote: post #150
mike_hock wrote: post #135

Earlier quoted context omitted.

WTF business do emojis have in Unicode? The BMP is all there ever should have been. Standardize the actual writing systems of the world, so everyone can write in their language. And once that is done, the standard doesn't need to change for a hundred years. What we need now is a standardized, sane subset of Unicode that implementations can support while rejecting the insane scope creep that got added on top of that.…

This argument was lost the moment Unicode was created. Japanese carriers had created their own standard for emoji encoding for sms. And they would not switch to Unicode unless the emoji were ported over. It’s a tricky situation. Maybe allowing an arbitrary bitmap char to represent any emoji would have been better but then we could have ended up in a situation where normal text or meaningful punctuation or perhaps eve…

I might think the important part was Japanese carriers were weaponizing flip phone culture to gatekeep "PCs" and open standard smartphones out of their microtransaction ecosystem. Emoji was one of the keys to disprove the FUD that iPhone can't be equal to flip phones and establish first class citizen status.

Re: Why can't you reverse a string with a flag emoji?

#217

Post by account42 »

AlanYx wrote: post #127
emodendroket wrote: post #114

What I'd like to know is, given the explosion of the character set for emoji, does the rationale for Han unification still make sense? The case for not allowing national variants seems less and less compelling with every emoji they add. This is a bit of a hobby horse, but imagine if every time you read an article in English on your phone some of the letters were replaced with "equivalent" Greek or Cyrillic one and yo…

I agree that Han unification was an unfortunate design decision, but I'd argue that the consortium is following a consistent approach to the Han unification with emoji. For example, they treat "regional" vendor variations in emoji as a font issue. If you get a message with the gun emoji, unless you have out-of-band information regarding which vendor variant is intended, there's no way in software to know if it should…

Have skin tone variants (which is somethine Unicode chose to add rather than added because of existing use) is consistent with not have distinct variants for glyphs from different languages?

Re: Why can't you reverse a string with a flag emoji?

#218

Post by account42 »

fomine3 wrote: post #192
emodendroket wrote: post #188

Earlier quoted context omitted.

Of course, the old characters must be left alone. But I'm not seeing what stops them from introducing new ones.

New characters have same glyph as old characters. It's the nightmare. For example, I can't find old one by searching new one. It's hard to know the reason for normal people. Should all software support searching by both characters? I don't expect all western developer take care. Equality comparison also fails without special support.

That is a bad exuse since it would preclude adding any new characters for existing languages. Would you have made the same objection for U+1E9E "ẞ", which was added in 2008?

Also, equality comparison already requires special support, e.g. normalization before comparison.

Sure, there would be an period where software support is incomplete but that is a bad reason to keep things broken forever.

Re: Why can't you reverse a string with a flag emoji?

#219

Post by lmm »

account42 wrote: post #215
lmm wrote: post #177

Earlier quoted context omitted.

> Emoji could just as well have been encoded as markup. They could have, but they were already being encoded as character codepoints in existing charactersets. So any character encoding scheme that wanted to replace all use cases for existing charactersets needed to match that. If switching charactersets meant you lost the ability to use emoji until you upgraded all your applications to support some markup format, pe…

> If switching charactersets meant you lost the ability to use emoji until you upgraded all your applications to support some markup format, people would just not switch. You need to upgrade those applications to support Unicode too.

Not necessarily, most applications already supported multiple encodings, having the OS implement one of the unicode encodings was often all that was needed.

Re: Why can't you reverse a string with a flag emoji?

#220

Post by account42 »

digisign wrote: post #149
emodendroket wrote: post #114

What I'd like to know is, given the explosion of the character set for emoji, does the rationale for Han unification still make sense? The case for not allowing national variants seems less and less compelling with every emoji they add. This is a bit of a hobby horse, but imagine if every time you read an article in English on your phone some of the letters were replaced with "equivalent" Greek or Cyrillic one and yo…

> were replaced with "equivalent" Greek or Cyrillic one The subset of equivalent letters, or different ones? If they looked the same, it wouldn't bother me if the letters in the center were a single codepoint between European languages: https://upload.wikimedia.org/wikipedia/commons/8/84/Venn_dia...

I am disappointed that that diagram omits ꙮ [0]

[0] https://en.wikipedia.org/wiki/Multiocular_O

Post reply on HN