Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UTF-8 Everywhere

utf8everywhere.org

Post reply on HN
Print view View original

191–200 of 289 posts

Re: UTF-8 Everywhere

#191

Post by Thrymr »

jcranmer wrote: post #89
legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

> It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) A decent fraction of software can impose rules on the portion of the filesystem within their control. A tool like mv or vim has to be prepared to handle any filepath encoding. But something like a VCS…

Sure, as long as you don't have to be compatible with anything else, you can assume whatever encoding you want. That doesn't change the point that general programs can't make that assumption.

Re: UTF-8 Everywhere

#192

Post by lisper »

tester89 wrote: post #180
lisper wrote: post #142

Earlier quoted context omitted.

> Rust is one of the few programming languages that correctly doesn’t treat file paths as strings. Common Lisp too.

I’ve never actually understood how pathnames work in CL actually.

That makes two of us. But they aren't strings :-)

(Seriously though, is it pathnames you don't understand or logical hosts? Because CL pathnames are actually pretty straightforward. Logical hosts, on the other hand, are a hot mess.)

Re: UTF-8 Everywhere

#193

Post by jcranmer »

camgunz wrote: post #166
jcranmer wrote: post #104

Earlier quoted context omitted.

> I'll choose whatever encoding I like, thanks. If everyone chooses whatever encoding they like, then the charset being used has to be encoded somewhere. The problem is, there are lots of places where charset isn't encoded (such as your filesystem). That this is a problem can be missed, because almost all charsets are a strict superset of ASCII (UTF-{7,16} are the only such charsets to be found in the top 99.99% of u…

> If everyone chooses whatever encoding they like, then the charset being used has to be encoded somewhere. This is gonna be the case for the foreseeable future, as you point out. Settling on one encoding only fixes this like, 100 years from now. I'd prefer to build encoding-aware software that solves this problem now. > given its compatibility with ASCII, UTF-8 is the most reasonable one to pick This only makes sens…

> But if I were to write software that had a hope of being broadly useful, UTF-8 everywhere doesn't get me there.

Actually, it does.

Right now, in 2020, if you're writing a new programming language, you can insist that the input files must be valid UTF-8 or it's a compiler error. If you're writing a localization tool, you can insist that the localization files be valid UTF-8 or it's an error. Even if you're writing a compiler for an existing language (e.g., C), it would not be unreasonable to say that the source file must be valid UTF-8 or it's an error--and let those not using UTF-8 right now handle it by converting their source code to use UTF-8. And this has been the case for a decade or so.

That's the point of UTF-8 everywhere: if you don't have legacy concerns [someone actively using a non-ASCII, non-UTF-8 charset that you have to support], force UTF-8 and be done with it. And if you do have legacy concerns, try to push people to using UTF-8 anyways (e.g., default to UTF-8).

Re: UTF-8 Everywhere

#194

Post by smasher164 »

samatman wrote: post #184
xg15 wrote: post #162

> When writing a UTF-8 string to a file, it is the length in bytes which is important. Counting any other type of ‘characters’ is, on the other hand, not very helpful. So, suppose I have a UTF-8 string of n code units (bytes) length. Unfortunately my data structure only permits strings of length m How do I correctly truncate the string so it doesn't become invalid UTF-8 and won't show any unexpected gibberish when re…

Avoiding invalid UTF-8 is easy, almost trivial: just make sure you don't truncate in the middle of a code point. The latter is fiendishly difficult to get right in all cases, the ugliest case being emoji flags. Being all-or-nothing on both sides of a ZWJ will get you most of the way there, however.

It's not though. Replacing invalid byte sequences is not terribly difficult.

https://golang.org/src/strings/strings.go?s=15854:15900#L627.

Re: UTF-8 Everywhere

#195

Post by carapace »

pjscott wrote: post #101
anderspitman wrote: post #94

Trying to figure out how to express this without making people mad at me. I think the conflation of Unicode with "plain text" might be a mistake. Don't get me wrong, Unicode serves an important purpose. But bumping the version from plain text 1.0 (ASCII) to plain text 2.0 (Unicode) introduced a ton of complexity, and there are cases where the abstractions start leaking (iterating characters etc). With things like dat…

Unicode is complicated because the languages it needs to handle are, alas, complicated. UTF-8 is super simple. It's a variable-length encoding for 21-bit unsigned integers. Wikipedia gives a handy table showing how it works: https://en.wikipedia.org/wiki/UTF-8#Description

Yeah, this. I have a pat "Unicode Rant" that boils down to this essentially.

Having a catalog of standard numbers-to-glyphs (or symbols or whatever, little pictures humans use to communicate with) is awesome and useful (and all ASCII ever was) but trying to digitalize all of human language is much much more challenging.

Re: UTF-8 Everywhere

#196

Post by TheCoelacanth »

ignoramous wrote: post #159
TheCoelacanth wrote: post #120

Earlier quoted context omitted.

You only need one sentence to explain why ASCII isn't sufficient: There are languages other than English.

> You only need one sentence to explain why ASCII isn't sufficient Nitpick: ASCII is sufficient when you consider that Base64, despite its 33% overhead from representing 6 bits with 8 bits, makes life easier for certain classes of software.

Base64 is an encoding for representing bytes[0] in ASCII.

That doesn't help you represent text unless you already have an encoding for representing text in bytes (e.g. UTF8).

[0] Octets if you want to be pedantic

Re: UTF-8 Everywhere

#197

Post by camgunz »

jcranmer wrote: post #193
camgunz wrote: post #166

Earlier quoted context omitted.

> If everyone chooses whatever encoding they like, then the charset being used has to be encoded somewhere. This is gonna be the case for the foreseeable future, as you point out. Settling on one encoding only fixes this like, 100 years from now. I'd prefer to build encoding-aware software that solves this problem now. > given its compatibility with ASCII, UTF-8 is the most reasonable one to pick This only makes sens…

> But if I were to write software that had a hope of being broadly useful, UTF-8 everywhere doesn't get me there. Actually, it does. Right now, in 2020, if you're writing a new programming language, you can insist that the input files must be valid UTF-8 or it's a compiler error. If you're writing a localization tool, you can insist that the localization files be valid UTF-8 or it's an error. Even if you're writing a…

I can't insist that other systems send your program UTF-8, or that the users' OS use UTF-8 for filenames and file contents, or that data in databases uses UTF-8, or that the UTF-8 you might get is always valid. The end result of all these things you're raising is "you can't assume, you have to check always, UTF-8 everywhere buys you nothing". Even if we did somehow get there, you'd still have to validate it.

Re: UTF-8 Everywhere

#198

Post by TheCoelacanth »

ftvy wrote: post #151
pjscott wrote: post #101

Earlier quoted context omitted.

Unicode is complicated because the languages it needs to handle are, alas, complicated. UTF-8 is super simple. It's a variable-length encoding for 21-bit unsigned integers. Wikipedia gives a handy table showing how it works: https://en.wikipedia.org/wiki/UTF-8#Description

When I wrote a very primitive UTF-8 library, I really began to appreciate UTF-8's design. For example; the first byte says how many bytes the character requires. At first it was daunting, but when I put 2 and 2 together, it really opened up. I am sure there are many aspects I am missing about UTF-8, but it is all reasonable in its design and implementation. For reference, I was converting between code points and actu…

The self-synchronizing property is also very clever. If you start at an arbitrary byte, you can find the start of the next character by scanning forward a maximum of 3 bytes.

Re: UTF-8 Everywhere

#199

Post by gumby »

tester89 wrote: post #180
lisper wrote: post #142

Earlier quoted context omitted.

> Rust is one of the few programming languages that correctly doesn’t treat file paths as strings. Common Lisp too.

I’ve never actually understood how pathnames work in CL actually.

They are pretty straightforward: they are just path structures rather than path names that may turn into single strings when supplied to your kernel. Or, depending on the OS maybe only part of the name is turned into a string and part determines which device or syntax applies. All of which is abstracted away by the path objects.

Back in the 1970s when thins first appeared on lisp machines is was not uncommon to use remote file systems transparently, and those remote file systems could be on quite different OSes like ITS, TOPS10 or -20, VMS, one of the lisp machine file systems and even Unix (though Networking came quite late to Unix). “MC:GUMBY; FOO >” and “OZ:FOO.TXT;0” were perfectly reasonable filenames. Some of those systems had file versioning built into them. So if the world likes like Unix to you some of that additional expressive power could be confusing.

C++17 path support is a neutered version of Common Lisp’s.

Re: UTF-8 Everywhere

#200

Post by acdha »

jcranmer wrote: post #89
legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

> It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) A decent fraction of software can impose rules on the portion of the filesystem within their control. A tool like mv or vim has to be prepared to handle any filepath encoding. But something like a VCS…

The history of Git and Subversion handling filenames makes me think that the opposite is true: A VCS which doesn't handle arbitrary byte-strings will have weird edge cases which prevent users from adding files or accessing them, possibly even “losing” data in a local checkout. This is especially tedious because it'll appear to work for a while until someone first tries to commit an unusual file or checks it out with a previously-unused client.
Post reply on HN