Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unicode is harder than you think

mcilloni.ovh

Post reply on HN
Print view View original

101–110 of 121 posts

Re: Unicode is harder than you think

#101

Post by torstenvl »

theamk wrote: post #87
torstenvl wrote: post #77

Earlier quoted context omitted.

How many bytes should be removed from your gap buffer when the user presses backspace? Does it matter if it's Hangul?

Your app, unless it's a text editor or UI toolkit, should not care. The easiest way to support Unicode is to avoid supportimg it. Leave text editing to UI widgets. Leave truncation to web browsers. Avoid fancy marquee effect on the terminal. Your program may look less fancy but it will automatically support unicode, even future extensions.

This is not good advice.

Most software is not an "app," and even if it is, you at least need to be able to tell the difference between, e.g., ISO 8859-1, ISO 8859-15, and UTF-8. Otherwise you'll get text talking about cutting doorways 8œ feet tall or submitting a résumé.

Any software that needs to process information effectively needs to understand character sets, including Unicode.

Re: Unicode is harder than you think

#102

Post by gumby »

qalmakka wrote: post #95
torstenvl wrote: post #85

Earlier quoted context omitted.

> What is that case? ... Hangul. What happens when the user presses backspace when the carat is positioned just past the graphical representation of U+AC01? It should by default result in U+AC01 being replaced by U+1100 and U+1161. (In some IME implementations the user might have to configure it, e.g., "Delete by jaso unit" in Windows 7.)

I don't understand why the felt the need to add precomposed characters for Hangul. Why? Why couldn't they just let the system compose them instead?

There was probably an existing character set that had precomposed characters and Unicode always includes existing character sets for round trip idempotency.

Imagine if hanzi had been encoded as radicals…

Re: Unicode is harder than you think

#103

Post by throwawaymobule »

theamk wrote: post #8
jmclnx wrote: post #4

No kidding, you have not lived until you try and explain UTF-8 to people who only believes in what they called "doublebyte". You think they get it, but surprise happens when a database load fails when loading Chinese Character "string" into a field sized calculated based upon 2 bytes per character.

Thank god for emojis! Those people would say, "No one in our org would use chinese" and refuse to fix things... but now I just point them to latest message from upper management which contain emoji or two. (And emoji are such a fine example - once they ate on the table, you need support for combining characters, characters outside of BMP, ligatures.. a large part of Unicode spec)

It's a very strange series of events that led us here, with various stages of 'oh no' and 'oh yes' events happening.

Like Apple trying to region-lock emojis, or whoever decided merging CJK together was a good idea, or that plan9 dev who scribbled out UTF8 on a napkin.

Re: Unicode is harder than you think

#104

Post by qalmakka »

gumby wrote: post #102
qalmakka wrote: post #95

Earlier quoted context omitted.

I don't understand why the felt the need to add precomposed characters for Hangul. Why? Why couldn't they just let the system compose them instead?

There was probably an existing character set that had precomposed characters and Unicode always includes existing character sets for round trip idempotency. Imagine if hanzi had been encoded as radicals…

Yeah, Chinese characters radicals are not an alphabet per se, while Hangul has separate, standalone letters. It's literally a matter of composing them in a fixed way, compared to Chinese characters which tend to go bananas on how many thousands of ways they can be composed.

Re: Unicode is harder than you think

#105

Post by theamk »

torstenvl wrote: post #101
theamk wrote: post #87

Earlier quoted context omitted.

Your app, unless it's a text editor or UI toolkit, should not care. The easiest way to support Unicode is to avoid supportimg it. Leave text editing to UI widgets. Leave truncation to web browsers. Avoid fancy marquee effect on the terminal. Your program may look less fancy but it will automatically support unicode, even future extensions.

This is not good advice. Most software is not an "app," and even if it is, you at least need to be able to tell the difference between, e.g., ISO 8859-1, ISO 8859-15, and UTF-8. Otherwise you'll get text talking about cutting doorways 8œ feet tall or submitting a résumé. Any software that needs to process information effectively needs to understand character sets, including Unicode.

Unless you deal with very old data, just use UTF-8 [0]. Mac terminal uses UTF-8 by default, most Linuxes use UTF-8 by default, and if you are stuck with Windows, there are libraries which let you use UTF-8 everywhere.

And if you do deal with old data, still prefer to use UTF-8, and make sure you only translate character sets if user asks / they are explicitly specified. And for the love of god, _please_ don't just hardcode two common encodings (like 8859-1 and 8859-15)... I used to deal with encodings a lot, and the only time I actually had data loss from encoding problems was from misguided apps which assume 8859-1 (and tried to convert to ASCII by stripping accents). Please either assume UTF-8 or assume nothing and keep data as-is.

And that's it, _most_ software does not need to know more. Let's look at some examples:

- For file archiver, assume utf-8 everywhere and don't worry about encoding at all (unless you planning to deal with very old archives, in this case you'd need encoding support. But for something made from scratch, don't bother.) _Especially_ don't add any encoding support for "extract to console" function -- if I need a different encoding I can pipe into iconv myself, thank you very much.

- For API client, you don't need to worry about encoding either.. A very old HTTP server might return data in something other than utf-8 but (1) your HTTP library likely handles it already (2) how many such servers are left, anyway?

- For XML analyzer, use proper XML library, they will handle encodings for you.

- For web dashboard or a database, keep everything in utf-8 and things will just work.

- For ETL-like applications, many modern data sources are already in UTF-8.. and if they are not, convert everything to utf-8 ASAP.

In other words: while it's useful to know that encodings exist, unless you are working with very old data or legacy systems, utf-8 is the only thing you need.

[0] http://utf8everywhere.org/

Re: Unicode is harder than you think

#106

Post by throwawaymobule »

veave wrote: post #15
Longhanks wrote: post #11

Earlier quoted context omitted.

Imho, unicode should stay out of politics. Country flags, vaccine syringes and pregnant men should have nothing to do with how computers handle text and writing systems.

Or when big tech banded together to change the pistol emoji to some scifi zapper.

I think that was a case of apple leading and everyone else following. They went with a water pistol in their font.

Re: Unicode is harder than you think

#107

Post by gumby »

qalmakka wrote: post #104
gumby wrote: post #102

Earlier quoted context omitted.

There was probably an existing character set that had precomposed characters and Unicode always includes existing character sets for round trip idempotency. Imagine if hanzi had been encoded as radicals…

Yeah, Chinese characters radicals are not an alphabet per se, while Hangul has separate, standalone letters. It's literally a matter of composing them in a fixed way, compared to Chinese characters which tend to go bananas on how many thousands of ways they can be composed.

Yeah, in case my message wasn't clear: my comment about the radicals was in support of encoding precomposed characters.

Re: Unicode is harder than you think

#108

Post by torstenvl »

theamk wrote: post #105
torstenvl wrote: post #101

Earlier quoted context omitted.

This is not good advice. Most software is not an "app," and even if it is, you at least need to be able to tell the difference between, e.g., ISO 8859-1, ISO 8859-15, and UTF-8. Otherwise you'll get text talking about cutting doorways 8œ feet tall or submitting a résumé. Any software that needs to process information effectively needs to understand character sets, including Unicode.

Unless you deal with very old data, just use UTF-8 [0]. Mac terminal uses UTF-8 by default, most Linuxes use UTF-8 by default, and if you are stuck with Windows, there are libraries which let you use UTF-8 everywhere. And if you do deal with old data, still prefer to use UTF-8, and make sure you only translate character sets if user asks / they are explicitly specified. And for the love of god, _please_ don't just ha…

Yeah. This is objectively terrible advice. Ideology is not an excuse for data loss.

Re: Unicode is harder than you think

#109

Post by theamk »

torstenvl wrote: post #108
theamk wrote: post #105

Earlier quoted context omitted.

Unless you deal with very old data, just use UTF-8 [0]. Mac terminal uses UTF-8 by default, most Linuxes use UTF-8 by default, and if you are stuck with Windows, there are libraries which let you use UTF-8 everywhere. And if you do deal with old data, still prefer to use UTF-8, and make sure you only translate character sets if user asks / they are explicitly specified. And for the love of god, _please_ don't just ha…

Yeah. This is objectively terrible advice. Ideology is not an excuse for data loss.

Do you have any specific examples, or are you just saying general statements?

Back in the day, I worked quite a bit various encodings (my language had 2 primary one and 2 secondary one, and it was a guess which one the text was), and the data loss usually happened from programs that tried to support encodings.

When program would not touch encodings, there might be some mojibake and unreadable text, but you could generally fix things.. I had to write some scripts which changed encoding of filenames and fixup random database or five, but there were no data loss.

It's the programs which were "able to tell the difference between, e.g., ISO 8859-1, ISO 8859-15, and UTF-8" which caused data loss. There were so many cases when I ended up with directory full of "?????????" or "aoeoaoao" files and that was it, there is no way to recover.

So please don't add Unicode support unless you have to, it is not ideology but rather the results of hard practical experience.

Re: Unicode is harder than you think

#110

Post by _nalply »

Another important subject are confusables. "A" looks same for Latin A, Greek Alpha and Cyrillic A. If you need users to identify names by sight, you need to do something about confusables. It is even a security problem, for example for hostnames, imagine a hacker build your bank's website with a similar looking URL.

https://news.ycombinator.com/item?id=32497414

Post reply on HN