Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UTF-8 Everywhere

utf8everywhere.org

Post reply on HN
Print view View original

31–40 of 289 posts

Re: UTF-8 Everywhere

#31

Post by flohofwoe »

camgunz wrote: post #21

This pops up every so often, and is wrong on several fronts (UNIX is UTF-8, UTF-8/32 lexicographically sort, etc.) There's not really a good reason to support UTF-8 over UTF-16; you can quibble over byte order (just pick one) and you can try and make an argument about everything being markup (it's not), but the fact is that UTF-16 is a more efficient encoding for the languages a plurality of people use natively. But…

I think it's quite obvious that UTF-8 is the better choice over UTF-16 or UTF-32 for exchanging data (if just for the little/big endian mess alone, and that UTF-16 isn't a fixed-length encoding either).

From that perspective, keeping the data in UTF-8 for most of its lifetime also when loaded into a program, and only convert "at the last minute" when talking to underlying operating system APIs makes a lot of sense, except for some very specific application types which do heavy text processing.

Re: UTF-8 Everywhere

#32

Post by bartwe »

Svip wrote: post #26
dpc_pw wrote: post #20

> For instance, ‘ch’ is two letters in English and Latin, but considered to be one letter in Czech and Slovak. Is "ch" really considered one _character_ in Czech and Slovak? I'm Polish and we do have "ch" and consider it one ... sound... represented by two letters? I mean... if you asked anyone to count letters/characters in a word, they would count "ch" as two. So I wonder if that's different in Slovakia or Chech Re…

A better example would probably be "ij" in Dutch. That's definitely considered a single letter, as words starting with ij in Dutch are capitalised IJ. Though there are glyphs for IJ /ij already in unicode.

Nobody has that as a letter on the keyboard here though, so it doesn't matter. Normally typed as a digraph. Would be nice if we just switched over to using y at this point. Makes me wonder, is the use of diacritics reducing since ascii keyboards became the norm ?

Re: UTF-8 Everywhere

#33

Post by nayuki »

I love the typesetting on the page. It is content-first, clean, and simple.

It lacks all the usual noise like modal dialogs, headers and footers, social media icons, colorful sidebars, newsletter sign-ups, cookie warnings, etc.

Re: UTF-8 Everywhere

#34

Post by bartwe »

goatinaboat wrote: post #27
heyplanet wrote: post #25

I think UTF-8 was a mistake. It is a pain in the ass to have a variable number of bytes per char. In Ascii, you could easily know every character personally. No strange surprises. Also no surprises while reading black on white text and suddenly being confronted with clors [1]. [1] Also no surprises when writing a comment on HN like this one and having some characters stripped. I put in a smiley as the firs "o" in col…

Certain things such as DNS, email addresses and so on should be restricted to ASCII, it’s a security nightmare otherwise.

I assume you mean a limited subset of 7bit ascii ? 33-126

Re: UTF-8 Everywhere

#35

Post by mercer »

Svip wrote: post #26
dpc_pw wrote: post #20

> For instance, ‘ch’ is two letters in English and Latin, but considered to be one letter in Czech and Slovak. Is "ch" really considered one _character_ in Czech and Slovak? I'm Polish and we do have "ch" and consider it one ... sound... represented by two letters? I mean... if you asked anyone to count letters/characters in a word, they would count "ch" as two. So I wonder if that's different in Slovakia or Chech Re…

A better example would probably be "ij" in Dutch. That's definitely considered a single letter, as words starting with ij in Dutch are capitalised IJ. Though there are glyphs for IJ /ij already in unicode.

"Ij" is also one sounds represented bij two letters, and I think capitalizing just the 'I' is pretty standard. As a Dutch person myself, I didn't even know that there's a glyph for it!

We also have "ei", which sounds the same and was invented to annoy people learning Dutch. Then there's "oe", "eu", "ui". And just to fuck even more with people learning the language, we have "au" and "ou" which also sound the same. Oh, and "ch" and "g".

Hans Brinker, the inventor of the Dutch language, famously would toss a florijn to decide between using ei/ij and au/ou, as he was not fond of foreigners. He's mostly known for saving our country though when he plugged a hole in a dyke with his finger (yes, I know what you're thinking, and no, we do not appreciate your dirty minds making light of this heroic act).

Re: UTF-8 Everywhere

#36

Post by crazygringo »

camgunz wrote: post #21

This pops up every so often, and is wrong on several fronts (UNIX is UTF-8, UTF-8/32 lexicographically sort, etc.) There's not really a good reason to support UTF-8 over UTF-16; you can quibble over byte order (just pick one) and you can try and make an argument about everything being markup (it's not), but the fact is that UTF-16 is a more efficient encoding for the languages a plurality of people use natively. But…

> not really a good reason to support UTF-8 over UTF-16

Of course there is, the fact that if you're dealing only with ASCII characters then it's backwards-compatible. Which is a nice convenience in a great number of situations programmers encounter.

The minor details of efficiency of an encoding these days isn't particularly relevant -- sure UTF-16 is better for Chinese, but the average webpage usually does have way more markup, CSS and JavaScript than text, and gzip-ing it on delivery will result in a similar payload totally independent of the encoding you choose.

Re: UTF-8 Everywhere

#37

Post by enedil »

dpc_pw wrote: post #20

> For instance, ‘ch’ is two letters in English and Latin, but considered to be one letter in Czech and Slovak. Is "ch" really considered one _character_ in Czech and Slovak? I'm Polish and we do have "ch" and consider it one ... sound... represented by two letters? I mean... if you asked anyone to count letters/characters in a word, they would count "ch" as two. So I wonder if that's different in Slovakia or Chech Re…

Yeah, but in Czech it's "č".

Re: UTF-8 Everywhere

#38

Post by jdsully »

jfkebwjsbx wrote: post #2

Even Microsoft is finally giving up UTF-16! They recommend now to use the UTF-8 "code page" in new code.

Are you sure? That will result in a conversion every time a string is passed to the kernel.

Windows can handle utf-8 but it is not the native character set for the platform.

Re: UTF-8 Everywhere

#39

Post by leosarev »

nathanaldensr wrote: post #3
jfkebwjsbx wrote: post #2

Even Microsoft is finally giving up UTF-16! They recommend now to use the UTF-8 "code page" in new code.

Do you have a source for this? AFAIK the .NET Framework CLR and CoreCLR both still store strings internally as UTF-16.

CoreCLR actively discussing introducing Utf8String type. https://github.com/dotnet/corefxlab/issues/2350

Re: UTF-8 Everywhere

#40

Post by jcranmer »

camgunz wrote: post #21

This pops up every so often, and is wrong on several fronts (UNIX is UTF-8, UTF-8/32 lexicographically sort, etc.) There's not really a good reason to support UTF-8 over UTF-16; you can quibble over byte order (just pick one) and you can try and make an argument about everything being markup (it's not), but the fact is that UTF-16 is a more efficient encoding for the languages a plurality of people use natively. But…

> There's not really a good reason to support UTF-8 over UTF-16

Two big reasons:

1. All legal ASCII text is UTF-8. That means upgrading ASCII to UTF-8 to support i18n doesn't require you to convert all your files that were in ASCII.

2. UTF-16 gives people the mistaken impression that characters are fixed-width instead of variable-width, and this causes things to break horribly on non-BMP data. I've seen amusing examples of this.

> Write your programs/whatevers allowing your users to be aware of and configure encodings.

Internally, your program should be using UTF-8 (or UTF-16 if you have to for legacy reasons), and you should convert from non-Unicode charsets as soon as possible. But if you're emitting stuff... you should try hard to make sure that UTF-8 is the only output charset you have to support. Letting people select non-UTF-8 charsets for output adds lots of complication (now you have to have error paths for characters that can't be emitted), and you need to have strong justification for why your code needs that complication.

Post reply on HN