Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why can't you reverse a string with a flag emoji?

davidamos.dev

Post reply on HN
Print view View original

191–200 of 247 posts

Re: Why can't you reverse a string with a flag emoji?

#191

Post by fomine3 »

lmm wrote: post #186
fomine3 wrote: post #183

Earlier quoted context omitted.

Mojibake is a universal problem when multiple charset is used and there are no charset specification on metadata. Software guess charset but it's just a guess. Japanese locale software occasionally confuses Latin-1 vs SJIS but often confuses SJIS vs EUC-JP or UTF-8. Unicode/UTF-8 is widely adopted/recommended in Japan and there are no widely used alternative. Japanese company tend to still use SJIS but it's just lazi…

> Mojibake is a universal problem when multiple charset is used and there are no charset specification on metadata. Software guess charset but it's just a guess. Japanese locale software occasionally confuses Latin-1 vs SJIS but often confuses SJIS vs EUC-JP or UTF-8. In theory it can happen with any combination of character sets, sure, but in practice every example of mojibake I've seen has been SJIS (or UTF-8) enco…

Maybe the guess order depends on locale reasonably. GP is my experience mainly on old days ja-JP localed Windows software. IIRC Unix software tend to not good at guess so maybe you referring them.

Nowadays I rarely see new EUC-JP contents (or I just not recognized) but still sometimes I encounter mojibake on Chrome while visiting old homepage (like once per month). For web page, anyway most modern pages (including SJIS) don't rely on guess but have tag so mojibake very rarely happen. For plaintext files, I still see UTF-8 file shown as SJIS on Windows Chrome.

Viewing Japanese only UTF-8 text is totally fine for Japanese localed Windows/Mac/(Linux but YMMV). So your case is to view the text on non-Japanese locale. It possibly have a problem but how SJIS solved the issue? What software switches font if it opens SJIS file? Is the app/format don't support specifying font/lang like HTML/Word?

I believe no developer want to treat foreign charset like GBK/Big-5/whatever. There are very few information. If developer can switch reading charset on a file, then they can also switch font.

Re: Why can't you reverse a string with a flag emoji?

#192

Post by fomine3 »

emodendroket wrote: post #188
fomine3 wrote: post #174

Earlier quoted context omitted.

Han unification was a try to fit CJK characters into 16bit BMP. Finally BMP is failed so meaningless but reverting it also produces huge compatibility issue.

Of course, the old characters must be left alone. But I'm not seeing what stops them from introducing new ones.

New characters have same glyph as old characters. It's the nightmare. For example, I can't find old one by searching new one. It's hard to know the reason for normal people. Should all software support searching by both characters? I don't expect all western developer take care. Equality comparison also fails without special support.

Re: Why can't you reverse a string with a flag emoji?

#193

Post by emodendroket »

fomine3 wrote: post #192
emodendroket wrote: post #188

Earlier quoted context omitted.

Of course, the old characters must be left alone. But I'm not seeing what stops them from introducing new ones.

New characters have same glyph as old characters. It's the nightmare. For example, I can't find old one by searching new one. It's hard to know the reason for normal people. Should all software support searching by both characters? I don't expect all western developer take care. Equality comparison also fails without special support.

It doesn't seem unfeasible to make a search that would support both.

Re: Why can't you reverse a string with a flag emoji?

#194

Post by fomine3 »

emodendroket wrote: post #193
fomine3 wrote: post #192

Earlier quoted context omitted.

New characters have same glyph as old characters. It's the nightmare. For example, I can't find old one by searching new one. It's hard to know the reason for normal people. Should all software support searching by both characters? I don't expect all western developer take care. Equality comparison also fails without special support.

It doesn't seem unfeasible to make a search that would support both.

It's possible to implement, but it makes confusion than benefit until all existing software support it.

Re: Why can't you reverse a string with a flag emoji?

#195

Post by easrng »

ineedasername wrote: post #170

It's an emoji... Are there any emojis with only one character? My assumption going in would be that any emoji is > 1 character. Admittedly, despite lots of string processing, I never have to deal with emojis so I guess I'm not sure. An interesting exercise would be emoji detection during string reversal to preserve the original emoji. I though something like that would be the crux of the article. Am I wrong about sin…

It depends what you mean by character, there are lots of single codepoint emojis though.

Re: Why can't you reverse a string with a flag emoji?

#196

Post by easrng »

pilsetnieks wrote: post #68
Beldin wrote: post #44

Interestingly, on my phone the so-called flag is not a flag at all, but "US" in outline. So python behaves as expected: the 2 character string, when reversed, becomes "SU". Similar stuff happens with the other "flag" strings. I'm sure emojis in my phone are outdated. I'm not sure how that affects whether I see a flag or letters.

Thankfully, there isn't an assigned ISO 3166-1 2-letter country code for SU currently; people may have interesting reactions seeing what happens when reversing a US flag emoji if there were.

If this was 1990 (and we somehow had the current emoji standard) SU would be the USSR flag.

Re: Why can't you reverse a string with a flag emoji?

#197

Post by easrng »

faebi wrote: post #134

Why reverse them if one barely can implement, display and edit them correctly. I never could make them work perfectly in VIM. Also I had to open a bug in Firefox recently: Flag emojis and others are displayed in double the size on Windows 10 using Firefox Nightly https://bugzilla.mozilla.org/show_bug.cgi?id=1746795

Windows doesn't even have flag emojis, they just show up as the country code.

Edit: Actually Firefox ships a copy of twemoji for fallback purposes, so flags will still render.

Re: Why can't you reverse a string with a flag emoji?

#198

Post by jahewson »

jonas21 wrote: post #165
Gigachad wrote: post #73

Earlier quoted context omitted.

It always feels like the most amount of work goes to the least used emoji. So many revisions and additions to the family emoji and yet it’s one of the ones I don’t recall anyone ever using. I think the trap Unicode got in to is technically they can have infinite emoji so they just don’t ever have a way to say no to new proposals.

This work wasn't done for emoji. They use the same zero-width joiner character [1] that exists to support Indic scripts like Devanagari, and any system that properly handles these languages will also properly handle the emoji. Yes, this adds a lot of complexity, but it's really a question of whether that complexity is justified in order to support all of the world's languages. And I think many would argue that it is.…

Emoji has turned out to be a great way to enhance complex script support in much real-world software and they get my full support for that reason too.

Re: Why can't you reverse a string with a flag emoji?

#200

Post by lmm »

fomine3 wrote: post #191
lmm wrote: post #186

Earlier quoted context omitted.

> Mojibake is a universal problem when multiple charset is used and there are no charset specification on metadata. Software guess charset but it's just a guess. Japanese locale software occasionally confuses Latin-1 vs SJIS but often confuses SJIS vs EUC-JP or UTF-8. In theory it can happen with any combination of character sets, sure, but in practice every example of mojibake I've seen has been SJIS (or UTF-8) enco…

Maybe the guess order depends on locale reasonably. GP is my experience mainly on old days ja-JP localed Windows software. IIRC Unix software tend to not good at guess so maybe you referring them. Nowadays I rarely see new EUC-JP contents (or I just not recognized) but still sometimes I encounter mojibake on Chrome while visiting old homepage (like once per month). For web page, anyway most modern pages (including SJ…

> Viewing Japanese only UTF-8 text is totally fine for Japanese localed Windows/Mac/(Linux but YMMV). So your case is to view the text on non-Japanese locale.

The issue is that realistically a certain proportion of customers are going to have the wrong locale setting or wrong default font set.

> It possibly have a problem but how SJIS solved the issue? What software switches font if it opens SJIS file? Is the app/format don't support specifying font/lang like HTML/Word?

Certainly Firefox will use a Japanese font by default for SJIS whereas it will use a generic (i.e. Chinese) font by default for UTF-8. I would expect most encoding-aware programs would do the same?

> If developer can switch reading charset on a file, then they can also switch font.

Sure, but it works both ways. And it's actually much easier for a lazy developer to ignore the font case because it's essentially only an issue for Japan. Whereas if you make a completely encoding-unaware program it will cause issues in much of Europe and all of Asia (well, it did pre-UTF8 anyway).

Post reply on HN