Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UTF-8 Everywhere

utf8everywhere.org

Post reply on HN
Print view View original

221–230 of 289 posts

Re: UTF-8 Everywhere

#221

Post by xeeeeeeeeeeenu »

gpvos wrote: post #202
xeeeeeeeeeeenu wrote: post #10

Earlier quoted context omitted.

Bugs like WriteFile() reporting the wrong number of bytes written with 65001 codepage were fixed years ago.

Ah, that's surprising, Microsoft was very stubbornly not doing that for at least a decade and a half. In fact, the FAQ in TFA (questions 9 and 20) mentions that there are still problems with CP_UTF8 (65001). Is the article out of date? Can someone respond to those statements?

The article is outdated, it's from 2012. Not only they fixed the problems but in Windows 10 1803 they also added an option to globally and permanently set both OEM and ANSI(!) codepages to 65001.

It can be enabled by checking "Beta: Use Unicode UTF-8 for worldwide language support" checkbox in region settings.

Re: UTF-8 Everywhere

#222

Post by tester89 »

lisper wrote: post #192
tester89 wrote: post #180

Earlier quoted context omitted.

I’ve never actually understood how pathnames work in CL actually.

That makes two of us. But they aren't strings :-) (Seriously though, is it pathnames you don't understand or logical hosts? Because CL pathnames are actually pretty straightforward. Logical hosts, on the other hand, are a hot mess.)

I don’t really understand how the #P”” “strings” aren’t différent from strings.

Re: UTF-8 Everywhere

#223

Post by tester89 »

gumby wrote: post #199
tester89 wrote: post #180

Earlier quoted context omitted.

I’ve never actually understood how pathnames work in CL actually.

They are pretty straightforward: they are just path structures rather than path names that may turn into single strings when supplied to your kernel. Or, depending on the OS maybe only part of the name is turned into a string and part determines which device or syntax applies. All of which is abstracted away by the path objects. Back in the 1970s when thins first appeared on lisp machines is was not uncommon to use r…

Ohh, so they’re kinda like date objects?

Re: UTF-8 Everywhere

#224

Post by naniwaduni »

legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

> Rust is one of the few programming languages that correctly doesn’t treat file paths as strings.

Rust is one of a few programming languages that incorrectly treat strings as if it were a coherent concept distinct from byte buffers.

Among those, it has the distinction of not forcing file paths into this inherently incorrect model.

(In practice, if you have a type system that can distinguish arbitrary byte buffers from ones with a known encoding, that is far from the most useful thing to distinguish about them anyway.)

Re: UTF-8 Everywhere

#225

Post by nybble41 »

alkonaut wrote: post #168
jodrellblank wrote: post #50

> Q: What do you think about Byte Order Marks? A: According to the Unicode Standard (v6.2, p.30): "Use of a BOM is neither required nor recommended for UTF-8". [...] Using BOMs would require all existing code to be aware of them, even in simple scenarios as file concatenation. This is unacceptable. Then your site "UTF-8 everywhere" is misnamed, because standards-following UTF-8 can have a BOM. It's not required or re…

100% agree. > using BOMs would require all existing code to be aware of them, even in simple scenarios as file concatenation Absolutely! Any app that writes UTF-files can (and probably should) avoid writing them. But any program that reads UTF files must handle a BOM. A lot of apps write UTF-8 including the BOM by default, for example Visual Studio. You can NOT concatenate two UTF-8 streams and expect that the result…

> You can NOT concatenate two UTF-8 streams and expect that the resulting stream is also a valid UTF-8 stream.

Actually you can; the ability to concatenate UTF-8 streams is an intentionally part of the design of UTF-8. The BOM is an ordinary Unicode code point and can occur in the middle of a valid UTF-8 stream, where it should be treated as either a zero-width non-breaking space or an unsupported character (which only affects rendering). So concatenating two UTF-8 streams with leading BOMs still results in a valid UTF-8 stream, albeit with an extra zero-width space.

The bigger problem with the BOM is that it breaks transparent compatibility with ASCII. Absent a leading BOM character, a UTF-8 steam containing only codepoints 0-127 is binary-identical to an ASCII-encoded text stream and can be handled with tools that are not UTF-8 aware. This was an explicit design consideration for both Unicode and UTF-8. Add the BOM, however, and your file is no longer plain text, which can lead to syntax errors or other issues that are difficult to diagnose because the BOM is invisible in UTF-8 aware text editors.

I think the BOM was a mistake—along with the variable-length multi-byte encodings it was created to support—but unfortunately at this point we're stuck with it. (Actually the BOM is prohibited in the multi-byte formats with an explicit byte order, like UTF-16BE; it would have been really nice if the same policy had been applied to UTF-8 where byte order is irrelevant.) The best we can do is recommend that new programs omit the BOM when outputting UTF-8 and either skip it at the beginning or convert it to U+2060 WORD JOINER anywhere else when it appears in the input.

Re: UTF-8 Everywhere

#226

Post by naniwaduni »

msla wrote: post #58
ngrnjp wrote: post #51

Earlier quoted context omitted.

That's a fundamental flaw of UNIX.

It's a reflection of the fact people aren't going to throw out existing filesystems because they aren't in a specific character encoding. There's nothing the OS can do about that, there's nothing programmers in general can do about that, and the only way to fix it is with a time machine and enough persuasion to force everyone to implement Unicode and UTF-8 to the exclusion of any other character encoding schemes.

And it would still be wrong, because the rules of what constitutes valid unicode have changed (what's a surrogate?), and also why would that be a good idea to bake into your filesystem??

Re: UTF-8 Everywhere

#227

Post by samatman »

smasher164 wrote: post #194
samatman wrote: post #184

Earlier quoted context omitted.

Avoiding invalid UTF-8 is easy, almost trivial: just make sure you don't truncate in the middle of a code point. The latter is fiendishly difficult to get right in all cases, the ugliest case being emoji flags. Being all-or-nothing on both sides of a ZWJ will get you most of the way there, however.

It's not though. Replacing invalid byte sequences is not terribly difficult. https://golang.org/src/strings/strings.go?s=15854:15900#L627 .

We are agreeing, the part I was indicating is difficult is 'displaying gibberish'.

Knowing what constitutes a grapheme cluster is detailed and frequently changes.

Re: UTF-8 Everywhere

#228

Post by naniwaduni »

cygx wrote: post #152
klodolph wrote: post #111

Earlier quoted context omitted.

I’m not going to try and minimize the problem, here. Han unification was pushed through by western interests, by my understanding. However, most Unicode characters are identical or nearly identical in Chinese and Japanese. Characters with “significant” visual differences got encoded as different Unicode characters. The same thing applies to simplified and traditional Chinese characters. So for a given “Han character”…

Han unification was pushed through by western interests, by my understanding. Note that as far as I'm aware, the interest in question was the initial 16-bit limit of the character set and later on the non-proliferation of competing standards. Also note that while Han unification is the most prominent example, there are technically similar cases, which just aren't as charged culturally. For one, Unicode doesn't encode…

Let's not excuse the utter irresponsibility of deciding on 16 bits: the initial 16-bit limit of the character set is instantly invalidated by looking at any comprehensive Chinese character dictionary, no reasonable choice of which will give you an estimate of under about 30k characters, even excluding graphical variants.

Even assuming that we discount 80k+ estimates by collapsing graphical variants, that's over half of your code space right off the bat. For this to seem like a seem like a good idea, you'd need to assume that Chinese is a uniquely bad one-off case. Not a good bet to stake your character set on.

Re: UTF-8 Everywhere

#229

Post by mmm_grayons »

Symbiote wrote: post #218
tachyonbeam wrote: post #182

Earlier quoted context omitted.

Playing the devil's advocate here. I am not a native English speaker, I'm a French speaker, but I'm happy that English is kind of the default international language. It's a relatively simple language. I actually make less grammar mistakes in English than I do in my native language. I suppose it's probably not a politically correct thing to say, the English are the colonists, the invaders, the oppressors, but eh, mayb…

English isn't even ASCII anyway. Some loanwords like façade or café retain their accents. Units like ° £ € and symbols like © ® × ÷ ½ aren't ASCII. It doesn't take much to need one of these cases in a project.

I know almost no one who actually types the accented e, let alone the c with the cedilla. I scarcely ever see the degree symbol typed. Rather, I see facade, cafe, and "degrees".

That aside, the big problem with unicode is not those characters; they're a simple two-byte extension. They obey the simple bijective mapping of binary character character on screen. Unicode doesn't. You have to deal with multiple code points representing one on-screen grapheme, which in turn may or may not translate into a single on-screen glyph. Also bi-directional text, or even vertical text (see the recent post about Mongolian script). Unicode is still probably one of the better solutions possible, but there's a reason you don't see it everywhere: it means not just updating to wide chars but having to deal with a text shaper, re-do your interfaces, and tons of other messy stuff. It's very easy for most people to look at that and ask why they'd bother if only a tiny percentage of users use, say, vertical text.

Re: UTF-8 Everywhere

#230

Post by naniwaduni »

xg15 wrote: post #162

> When writing a UTF-8 string to a file, it is the length in bytes which is important. Counting any other type of ‘characters’ is, on the other hand, not very helpful. So, suppose I have a UTF-8 string of n code units (bytes) length. Unfortunately my data structure only permits strings of length m How do I correctly truncate the string so it doesn't become invalid UTF-8 and won't show any unexpected gibberish when re…

What do you think distinguishes the truncated string containing "glyphs or grapheme clusters" that weren't in the original string from the truncated string containing words that weren't in the original string? Is the latter somehow more acceptable? How about missing necessary context from the end of a sentence?
Post reply on HN