Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UTF-8 Everywhere

utf8everywhere.org

Post reply on HN
Print view View original

41–50 of 289 posts

Re: UTF-8 Everywhere

#41

Post by leosarev »

mormegil wrote: post #18
nathanaldensr wrote: post #3

Earlier quoted context omitted.

Do you have a source for this? AFAIK the .NET Framework CLR and CoreCLR both still store strings internally as UTF-16.

AFAICT, it's not only "internal representation". .NET strings are defined as a sequence of UTF-16 units, including the definition of the Char type representing a single UTF-16 code unit. I can't imagine how such a change could be implemented (other than changing the internal representation but converting on all accesses which would be nonsense, I think).

Current plan is: https://github.com/dotnet/corefxlab/issues/2350

Re: UTF-8 Everywhere

#42

Post by pilsetnieks »

dpc_pw wrote: post #20

> For instance, ‘ch’ is two letters in English and Latin, but considered to be one letter in Czech and Slovak. Is "ch" really considered one _character_ in Czech and Slovak? I'm Polish and we do have "ch" and consider it one ... sound... represented by two letters? I mean... if you asked anyone to count letters/characters in a word, they would count "ch" as two. So I wonder if that's different in Slovakia or Chech Re…

At first I though they simply mean the letter "č" but no, it turns out that "ch" (and also "dz") is a digraph with a separate place in Czech and Slovak alphabets.

Re: UTF-8 Everywhere

#43

Post by JdeBP »

jdsully wrote: post #38
jfkebwjsbx wrote: post #2

Even Microsoft is finally giving up UTF-16! They recommend now to use the UTF-8 "code page" in new code.

Are you sure? That will result in a conversion every time a string is passed to the kernel. Windows can handle utf-8 but it is not the native character set for the platform.

There's a conversion in every ...A() function. Conversion between UTF-8 and WTF-16 is just more of the same, but without codepage lookup tables. (-:

Re: UTF-8 Everywhere

#45

Post by pilsetnieks »

enedil wrote: post #37
dpc_pw wrote: post #20

> For instance, ‘ch’ is two letters in English and Latin, but considered to be one letter in Czech and Slovak. Is "ch" really considered one _character_ in Czech and Slovak? I'm Polish and we do have "ch" and consider it one ... sound... represented by two letters? I mean... if you asked anyone to count letters/characters in a word, they would count "ch" as two. So I wonder if that's different in Slovakia or Chech Re…

Yeah, but in Czech it's "č".

No. Č is something else, ch is a digraph that's pronounced differently. Take a look at Czech and Slovak alphabets specifically:

https://en.wikipedia.org/wiki/Czech_orthography

https://en.wikipedia.org/wiki/Slovak_orthography

Re: UTF-8 Everywhere

#46

Post by jandrese »

heyplanet wrote: post #25

I think UTF-8 was a mistake. It is a pain in the ass to have a variable number of bytes per char. In Ascii, you could easily know every character personally. No strange surprises. Also no surprises while reading black on white text and suddenly being confronted with clors [1]. [1] Also no surprises when writing a comment on HN like this one and having some characters stripped. I put in a smiley as the firs "o" in col…

> It is a pain in the ass to have a variable number of bytes per char.

Maybe, but nobody can stomach the wasted space you get with UTF-32 in almost every situation. The encoding time tradeoff was considered less objectionable than making most of your text twice or four times larger.

Re: UTF-8 Everywhere

#47

Post by mlj45 »

dpc_pw wrote: post #20

> For instance, ‘ch’ is two letters in English and Latin, but considered to be one letter in Czech and Slovak. Is "ch" really considered one _character_ in Czech and Slovak? I'm Polish and we do have "ch" and consider it one ... sound... represented by two letters? I mean... if you asked anyone to count letters/characters in a word, they would count "ch" as two. So I wonder if that's different in Slovakia or Chech Re…

This depends on your definition of informal terms like "letter", "character" etc.

The typographic term for combinations like this is "digraph". (Wikipedia's definition: "A digraph [...] is a pair of characters used in the orthography of a language to write either a single phoneme [...] or a sequence of phonemes that does not correspond to the normal values of the two characters combined".)

Whether digraphs have separate keys on a keyboard, are treated as distinct for the purposes of alphabetisation, whether speakers of the language think of them as separate "letters" when spelling out a word and so on, are all separate issues and varies between languages (or, more precisely, between the conventions for writing a certain language).

Re: UTF-8 Everywhere

#48

Post by sitzkrieg »

legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

git will also do this, so on a fs that allowa arbitrarily byte named files, you end up with tree objects of same name which makes digging them out later "fun"

Re: UTF-8 Everywhere

#49

Post by orf »

ken wrote: post #28
legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

> one of the few programming languages that correctly doesn’t treat file paths as strings I hear: one of those few programming languages that, despite its vaunted type-safety, makes it possible to accidentally create a file with a completely bogus name that I won't be able to view or open correctly with half the programs on my computer. Languages which allow arbitrary byte sequences in paths are the cause of, and sol…

No, it’s impossible to do that accidentally. Due to its type safety. You have to be pretty explicit about passing a non-string in (all rust strings are valid utf8).

Re: UTF-8 Everywhere

#50

Post by jodrellblank »

> Q: What do you think about Byte Order Marks? A: According to the Unicode Standard (v6.2, p.30): "Use of a BOM is neither required nor recommended for UTF-8". [...] Using BOMs would require all existing code to be aware of them, even in simple scenarios as file concatenation. This is unacceptable.

Then your site "UTF-8 everywhere" is misnamed, because standards-following UTF-8 can have a BOM. It's not required or recommended, but it is possible and allowable, so you might see them and if you follow the standard you have to deal with them. It's not a matter of "this would require all existing code to handle them" - that is not hypothetical, that is the current world, to be standards-compliant all existing code does already need to be aware of them. It isn't, which means it's broken. Declaring it "unacceptable" is meaningless, except to say you're rejecting the standard and doing something incompatible and broken because it's easier.

Which is a position one can take and defend, but it's not a good position for a site claiming to be pushing for people to follow the standard. What it is, is yet another non-standard ad-hoc variant defined by what some subset of tools the authors use can/can't handle in April 2020.

> "the UTF-8 BOM exists only to manifest that this is a UTF-8 stream"

Throwing the word "only" in there doesn't make it go away. It exists as a standards-compliant way to distinguish UTF-8 from ASCII, not recommended but not forbidden.

> "A: Are you serious about not supporting all of Unicode in your software design? And, if you are going to support it anyway, how does the fact that non-BMP characters are rare practically change anything"

Well in the same way, how does the fact that UTF8+BOM is rare practically change anything? At some level you're either pushing for everyone to follow standards even if it's inconvenient because that makes life better for everyone overall, like you are with surrogate pairs and indexing, or you're creating another ad-hoc incompatible variation of UTF-8 which you prefer to the standard and trying to strong-arm everyone else into using it with threats of being incompatible with all the code which already does it wrong.

Being wary of Chesterton's Fence, presumably there's some company or system which got UTF-8+BOM added to the standard because they wanted it, or needed it.

Post reply on HN