Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UTF-8 Everywhere

utf8everywhere.org

Post reply on HN
Print view View original

171–180 of 289 posts

Re: UTF-8 Everywhere

#171

Post by ksec »

cryptonector wrote: post #118
shpx wrote: post #93

What I never see mentioned about Unicode is Han Unification https://en.m.wikipedia.org/wiki/Han_unification As I understand it, it's impossible to have a txt file that uses Japanese and Chinese characters at the same time. The file will either use the Chinese or Japanese forms of the characters, depending on your font. I would think this is a big gotcha people must run into all the time, but I never hear anyone talk…

First of all, there is no new unification work ongoing. The Unicode Consortium moved on from that by moving on from UCS-2. UCS-2 drove unification as a way to preserve precious codespace. There used to be language tag codepoints for this, but they've been deprecated. Han unification is an accident of history: a result of UTF-8 not having existed until it was too late! There's not going to be a different new Unicode f…

The same could be said whether è é should be the same as e with different fonts. People who cares about it would complain. To those who only uses English it is only the same e.

Re: UTF-8 Everywhere

#172

Post by akiselev »

mhh__ wrote: post #115
DannyB2 wrote: post #78

Earlier quoted context omitted.

> Rust is one of the few programming languages that correctly doesn’t treat file paths as strings. Imagine if languages allowed subtypes of strings which are not directly assignment compatible. HtmlString SqlString String A String could be converted to HtmlString not by assignment, but through a function call, which escapes characters that the browser would recognize as markup. Similarly a String would be converted t…

Allowed you to? You could do that in C++ quite happily, it's just not useful enough. To bother implementing, at least.

They're not worth the effort in C++ because it doesn't have strictly enforced affine/dependent types. The GP is invisioning a language that does.

Re: UTF-8 Everywhere

#173

Post by dtech »

ignoramous wrote: post #159
TheCoelacanth wrote: post #120

Earlier quoted context omitted.

You only need one sentence to explain why ASCII isn't sufficient: There are languages other than English.

> You only need one sentence to explain why ASCII isn't sufficient Nitpick: ASCII is sufficient when you consider that Base64, despite its 33% overhead from representing 6 bits with 8 bits, makes life easier for certain classes of software.

You still need an encoding to represent non-ASCII characters like ë or 木. Base64 is no help at all there

Re: UTF-8 Everywhere

#174

Post by dtech »

anderspitman wrote: post #94

Trying to figure out how to express this without making people mad at me. I think the conflation of Unicode with "plain text" might be a mistake. Don't get me wrong, Unicode serves an important purpose. But bumping the version from plain text 1.0 (ASCII) to plain text 2.0 (Unicode) introduced a ton of complexity, and there are cases where the abstractions start leaking (iterating characters etc). With things like dat…

Of course ASCII is simpler than Unicode, it handles only 127 characters. If you restrict yourself to those characters ASCII is binary equivalent to UTF-8.

So yeah, maybe you shouldn't use characters 128+ for data archival, I doubt that's a good idea, but that's irrelevant to whether UTF-8 is plain text or not.

Re: UTF-8 Everywhere

#175

Post by tingletech »

anderspitman wrote: post #94

Trying to figure out how to express this without making people mad at me. I think the conflation of Unicode with "plain text" might be a mistake. Don't get me wrong, Unicode serves an important purpose. But bumping the version from plain text 1.0 (ASCII) to plain text 2.0 (Unicode) introduced a ton of complexity, and there are cases where the abstractions start leaking (iterating characters etc). With things like dat…

LET'S GO BACK TO 6-BITS

Re: UTF-8 Everywhere

#176

Post by kazinator »

DagAgren wrote: post #158
kazinator wrote: post #96

Earlier quoted context omitted.

You can't write proper snooty English in ASCII, with diaereses and whatnot.

ASCII doesn't have have all the punctuation regularly used in English.

ASCII doesn't have a direct representation of all the punctuation used in English print, like 66 99 quotes, and different kinds of dashes (distinct from minus). For non-print, it's entirely fine.

Typesetting should be handled by a markup language anyway. Adding a few characters to Notepad doesn't create a typesetting system. A typesetting system needs to be able to do kerning, ligatures, justification. Not to mention bold, italics, and different fonts.

Re: UTF-8 Everywhere

#177

Post by Animats »

I'd argue for some standard tests for UTF-8 strings:

- Basic - UTF-8 byte syntax correct.

- Unambiguous - similar to the rules for Unicode domain names. The rules are complicated, but basically they prohibit homoglyphs, mixing glyphs from different character sets, forwards and backwards modifiers in the same string, no emoji or modifiers, etc. Use where people have to visually compare two things for identity or retype them, such as file names.

- Unambiguous, light version - as above, but allow emoji and modifiers. Normal form for documents.

Re: UTF-8 Everywhere

#178

Post by tachyonbeam »

dtech wrote: post #174
anderspitman wrote: post #94

Trying to figure out how to express this without making people mad at me. I think the conflation of Unicode with "plain text" might be a mistake. Don't get me wrong, Unicode serves an important purpose. But bumping the version from plain text 1.0 (ASCII) to plain text 2.0 (Unicode) introduced a ton of complexity, and there are cases where the abstractions start leaking (iterating characters etc). With things like dat…

Of course ASCII is simpler than Unicode, it handles only 127 characters. If you restrict yourself to those characters ASCII is binary equivalent to UTF-8. So yeah, maybe you shouldn't use characters 128+ for data archival, I doubt that's a good idea, but that's irrelevant to whether UTF-8 is plain text or not.

I think that sometimes it makes sense to enforce strict limitations early on (eg: overly strict input validation). You can then remove such limitations in later versions of your software, after careful consideration and after inserting the necessary tests. The reverse usually doesn't work. If you didn't have those limitations early on, and your database is full of strings with characters that should never have been allowed in there, you will have a hard time cleaning up the mess.

This seems especially true to me in the design of programming languages. If you have useless, badly thought out features in your programming language, people will begin to rely on them, and you will never be able to get rid of them... So start with a small language, and make it strict. Grow it gradually.

Re: UTF-8 Everywhere

#179

Post by ksec »

klodolph wrote: post #111
shpx wrote: post #93

What I never see mentioned about Unicode is Han Unification https://en.m.wikipedia.org/wiki/Han_unification As I understand it, it's impossible to have a txt file that uses Japanese and Chinese characters at the same time. The file will either use the Chinese or Japanese forms of the characters, depending on your font. I would think this is a big gotcha people must run into all the time, but I never hear anyone talk…

I’m not going to try and minimize the problem, here. Han unification was pushed through by western interests, by my understanding. However, most Unicode characters are identical or nearly identical in Chinese and Japanese. Characters with “significant” visual differences got encoded as different Unicode characters. The same thing applies to simplified and traditional Chinese characters. So for a given “Han character”…

Yes, the real problem is when you start mixing All Four ( or Five ) of them together Chinese Traditional, Simplified Korean, Japanese things becomes extremely problematic.

I think it is by luck, All four writings has significant usage within their own region, imagine if one of them were significantly smaller and over time were forced ( or by ease of use or what ever reason ) to switch to a different style without knowing it.

Re: UTF-8 Everywhere

#180

Post by tester89 »

lisper wrote: post #142
legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

> Rust is one of the few programming languages that correctly doesn’t treat file paths as strings. Common Lisp too.

I’ve never actually understood how pathnames work in CL actually.
Post reply on HN