Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UTF-8 Everywhere

utf8everywhere.org

Post reply on HN
Print view View original

61–70 of 289 posts

Re: UTF-8 Everywhere

#61

Post by kosievdmerwe »

bartwe wrote: post #32
Svip wrote: post #26

Earlier quoted context omitted.

A better example would probably be "ij" in Dutch. That's definitely considered a single letter, as words starting with ij in Dutch are capitalised IJ. Though there are glyphs for IJ /ij already in unicode.

Nobody has that as a letter on the keyboard here though, so it doesn't matter. Normally typed as a digraph. Would be nice if we just switched over to using y at this point. Makes me wonder, is the use of diacritics reducing since ascii keyboards became the norm ?

Afrikaans did this. We use "y" instead of "ij".

Re: UTF-8 Everywhere

#62

Post by projektfu »

When I used to do a lot of windows programming in the late 90s, I wish that I had a sensible guide like this for handling strings. TCHAR was always a source of subtle bugs.

I suppose, though, that the underlying problem was that Microsoft was so late to implement a compatibility solution for Windows 9x. Most software of the time ended up implementing on "ANSI" multibyte character set (MBCS) just because otherwise you would need to either deploy 2 executables or do your own thunking. This solution would be a double thunk on 9x because you'd be thunking your UTF-8 to unicode and then thunking that back to MBCS.

Re: UTF-8 Everywhere

#63

Post by projektfu »

snazz wrote: post #6
jfkebwjsbx wrote: post #2

Even Microsoft is finally giving up UTF-16! They recommend now to use the UTF-8 "code page" in new code.

Is java.lang.String still UTF-16? Is there any plan to fix that? Once Windows and Java take care of it, I can't think of any other major UTF-16 uses left. Are there any that I've forgotten about? Edit: Still looks like UTF-16, according to the Oracle documentation page: https://docs.oracle.com/en/java/javase/14/docs/api/java.base... Edit 2: JavaScript too. See my reply to someone else below.

I don't think it's a big deal for Java because it's always easy to transfer in from and out to UTF-8. Very few Java programs use UTF-16 as a persistence format, and Java-native applications can directly marshal strings around as they are a first-class datatype.

Re: UTF-8 Everywhere

#64

Post by camgunz »

flohofwoe wrote: post #31
camgunz wrote: post #21

This pops up every so often, and is wrong on several fronts (UNIX is UTF-8, UTF-8/32 lexicographically sort, etc.) There's not really a good reason to support UTF-8 over UTF-16; you can quibble over byte order (just pick one) and you can try and make an argument about everything being markup (it's not), but the fact is that UTF-16 is a more efficient encoding for the languages a plurality of people use natively. But…

I think it's quite obvious that UTF-8 is the better choice over UTF-16 or UTF-32 for exchanging data (if just for the little/big endian mess alone, and that UTF-16 isn't a fixed-length encoding either). From that perspective, keeping the data in UTF-8 for most of its lifetime also when loaded into a program, and only convert "at the last minute" when talking to underlying operating system APIs makes a lot of sense, e…

I'm gonna do little quotes but, I don't mean to be passive aggressive. It's just that this stuff comes up all the time

> I think it's quite obvious that UTF-8 is the better choice over UTF-16 or UTF-32 for exchanging data (if just for the little/big endian mess alone...

This should be the responsibility of a string library internally, and if you're saving data to disk or sending it over the network, you should be serializing to a specific format. That format can be UTF-8, or it can be whatever, depending on your application's needs.

> and that UTF-16 isn't a fixed-length encoding either)

We should stop assuming any string data is a fixed-length encoding. This is a major disadvantage of UTF-8, because it allows for this conflation.

> keeping the data in UTF-8 for most of its lifetime also when loaded into a program, and only convert "at the last minute" when talking to underlying operating system APIs makes a lot of sense, except for some very specific application types which do heavy text processing.

Well, you're essentially saying "I know about your use case better than you do". It might be important to me to not blow space on UTF-8. But if my platform/libraries have bought into "UTF-8 everywhere" and don't give me knobs to configure the encoding, I have no recourse.

And that's the entire basis for this. It's "having to mess with encodings is worse than the application-specific benefits of being able to choose an encoding". I think that's... at best an impossible claim and at worst pretty arrogant. Again here I don't mean you, but this "UTF-8 everywhere" thing.

Re: UTF-8 Everywhere

#65

Post by downerending »

ngrnjp wrote: post #51
legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

That's a fundamental flaw of UNIX.

As flaws go, it's pretty awesome. Wish we had more such.

Re: UTF-8 Everywhere

#66

Post by Shebanator »

JdeBP wrote: post #43
jdsully wrote: post #38

Earlier quoted context omitted.

Are you sure? That will result in a conversion every time a string is passed to the kernel. Windows can handle utf-8 but it is not the native character set for the platform.

There's a conversion in every ...A() function. Conversion between UTF-8 and WTF-16 is just more of the same, but without codepage lookup tables. (-:

WTF-16? I like it...

Re: UTF-8 Everywhere

#67

Post by andy_wrote »

dpc_pw wrote: post #20

> For instance, ‘ch’ is two letters in English and Latin, but considered to be one letter in Czech and Slovak. Is "ch" really considered one _character_ in Czech and Slovak? I'm Polish and we do have "ch" and consider it one ... sound... represented by two letters? I mean... if you asked anyone to count letters/characters in a word, they would count "ch" as two. So I wonder if that's different in Slovakia or Chech Re…

Based on my experience learning Czech (not native at all, just interested):

- it's typically listed as a separate letter when writing out the alphabet

- but in practice it's typed out as "c h" and not as a single character

- it occupies its own place in Czech standard alphabetical order, my English-Czech dictionary has all the "ch" words after "h" (so interestingly in order to do a proper sort programmatically you need to possibly look 2 characters ahead)

Re: UTF-8 Everywhere

#68

Post by ChrisSD »

mark-r wrote: post #60
ChrisSD wrote: post #24

Earlier quoted context omitted.

Sure we could declare that but then what? Non-unicode filenames won't suddenly disappear. Operating systems won't suddenly enforce unicode. Filesystems will still allow non-unicode names. Simply declaring it doesn't help anybody. In the meantime your application still needs to handle non-unicode filenames otherwise those malicious ones are free to be malicious.

Once you lose the expectation of being able to work with non-unicode filenames, those files will quickly get renamed and cease to be a problem.

How can you rename them if you can only use unicode paths?

Re: UTF-8 Everywhere

#70

Post by magicalhippo »

heyplanet wrote: post #25

I think UTF-8 was a mistake. It is a pain in the ass to have a variable number of bytes per char. In Ascii, you could easily know every character personally. No strange surprises. Also no surprises while reading black on white text and suddenly being confronted with clors [1]. [1] Also no surprises when writing a comment on HN like this one and having some characters stripped. I put in a smiley as the firs "o" in col…

> It is a pain in the ass to have a variable number of bytes per char.

In the same vein it's a pain in the ass to write everything in assembler. Which is why we don't do that, we use high-level languages instead.

Post reply on HN