Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UTF-8 Everywhere

utf8everywhere.org

Post reply on HN
Print view View original

161–170 of 289 posts

Re: UTF-8 Everywhere

#161

Post by anoncake »

cygx wrote: post #152
klodolph wrote: post #111

Earlier quoted context omitted.

I’m not going to try and minimize the problem, here. Han unification was pushed through by western interests, by my understanding. However, most Unicode characters are identical or nearly identical in Chinese and Japanese. Characters with “significant” visual differences got encoded as different Unicode characters. The same thing applies to simplified and traditional Chinese characters. So for a given “Han character”…

Han unification was pushed through by western interests, by my understanding. Note that as far as I'm aware, the interest in question was the initial 16-bit limit of the character set and later on the non-proliferation of competing standards. Also note that while Han unification is the most prominent example, there are technically similar cases, which just aren't as charged culturally. For one, Unicode doesn't encode…

That's not the same thing. Fraktur is just a style of fonts, antiqua and fraktur letters are semantically the same.

Re: UTF-8 Everywhere

#162

Post by xg15 »

> When writing a UTF-8 string to a file, it is the length in bytes which is important. Counting any other type of ‘characters’ is, on the other hand, not very helpful.

So, suppose I have a UTF-8 string of n code units (bytes) length. Unfortunately my data structure only permits strings of length m How do I correctly truncate the string so it doesn't become invalid UTF-8 and won't show any unexpected gibberish when rendered? (E.g., the truncated string doesn't suddenly contain any glyphs or grapheme clusters that weren't in the original string)

Re: UTF-8 Everywhere

#163

Post by oconnor663 »

Spivak wrote: post #140
oconnor663 wrote: post #114

Earlier quoted context omitted.

> It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 Yes but , most programs expect to be able to print filepaths at least under some circumstances, like printing error messages. Even if a program is fully correct and doesn't assume an encoding in normal operation, it still has to assume one for printing. Filepaths that aren't utf-8 lead to a bunch of ����� in your output (at best). So I t…

I mean it doesn't have to assume an encoding for printing, it just has to have a sane way of turning the path into something human readable. Look you're right that this ship has sailed but ideally we would have decided on a way to display and encode binary for file paths.

I dunno. That sounds like proposing to render "foo.txt" as "Zm9vLnR4dA==" or "[102, 111, 111, 46, 116, 120, 116]" or something. I think you probably meant something like "print the regular characters if the string is UTF-8, or a lossless fallback representation of the bytes otherwise." That's a good idea, and I think a lot of programs do that, but at the same time "if the string is UTF-8" is problematic. There's no reliable way for us to know what strings are or are not intended to be decoded as UTF-8, because non-UTF-8 encodings can coincidentally produce valid UTF-8 bytes. For example, the two characters "&!" are the same bytes in UTF-8 as the character "Ω" is in UTF-16. This works in Python:

    assert "&!".encode("UTF-8").decode("UTF-16le") == "Ω"
So I think I want to claim something a bit stronger:

1) Users demand, quite rightly, to be able to read paths as text. 2) There is no reliable way to determine the encoding of a string, just by looking at its bytes. And Unix doesn't provide any other metadata. 3) Therefore, useful Unix programs must assume that any path that could be UTF-8, is UTF-8, for the purpose of displaying it to the user.

Maybe in an alternate reality, the system locale could've been the reliable source of truth for string encodings? But of course if we were starting from scratch today, we'd just mandate UTF-8 and be done with it :)

Re: UTF-8 Everywhere

#164

Post by kps »

kazinator wrote: post #96
DagAgren wrote: post #54

Earlier quoted context omitted.

You can't even write proper English in ASCII. ASCII is an absolute dead end. It's history. Actually representing human language is HARD. It is also absolutely necessary. Whatever solution you choose is going to be complicated, because it is solving a very complicated problem. Throwing your hands up and going "oh this is too hard, I don't like it" will get you nowhere.

You can't write proper snooty English in ASCII, with diaereses and whatnot.

1967 ASCII anticipated that, with dual-use character shapes so you could type o BS " → ö

But then people invented video terminals that didn't overstrike.

Re: UTF-8 Everywhere

#165

Post by cygx »

anoncake wrote: post #161
cygx wrote: post #152

Earlier quoted context omitted.

Han unification was pushed through by western interests, by my understanding. Note that as far as I'm aware, the interest in question was the initial 16-bit limit of the character set and later on the non-proliferation of competing standards. Also note that while Han unification is the most prominent example, there are technically similar cases, which just aren't as charged culturally. For one, Unicode doesn't encode…

That's not the same thing. Fraktur is just a style of fonts, antiqua and fraktur letters are semantically the same.

There are differences as well as similarities. I'm no expert, but shouldn't, say, U+4ECA still translate to 'now' no matter if you draw a particular line horizontally or diagonally? There are also some mandatory[1] ligatures in Fraktur unavailable in Unicode. What if I wanted to preserve that distinction in historic writing?

edit:

[1] I think the mandatory ones are actually there (just not in Fraktur), it's some optional ones like ſch that are missing.

Re: UTF-8 Everywhere

#166

Post by camgunz »

jcranmer wrote: post #104
camgunz wrote: post #71

Earlier quoted context omitted.

> 1. All legal ASCII text is UTF-8. That means upgrading ASCII to UTF-8 to support i18n doesn't require you to convert all your files that were in ASCII. Eh, realistically if you're doing this, you should be validating it like converting from one encoding to another anyway. I get that people won't and haven't, but that's because UTF-8 has this anti-feature where ASCII is compatible with it, and that's led to a lot of…

> I'll choose whatever encoding I like, thanks. If everyone chooses whatever encoding they like, then the charset being used has to be encoded somewhere. The problem is, there are lots of places where charset isn't encoded (such as your filesystem). That this is a problem can be missed, because almost all charsets are a strict superset of ASCII (UTF-{7,16} are the only such charsets to be found in the top 99.99% of u…

> If everyone chooses whatever encoding they like, then the charset being used has to be encoded somewhere.

This is gonna be the case for the foreseeable future, as you point out. Settling on one encoding only fixes this like, 100 years from now. I'd prefer to build encoding-aware software that solves this problem now.

> given its compatibility with ASCII, UTF-8 is the most reasonable one to pick

This only makes sense of your system is ASCII in the first place, and if you can't build encoding-aware software. I think we can both agree that's essentially legacy ASCII software, so you don't get to choose anything anyway. And any system that interacts with it should be encoding-aware and still validate the encoding anyway, as though it might be BIG5 or whatever. Assuming ASCII/UTF-8 is a bad idea, always and forever.

> If you want to insist on using KOI-8, or ISO-2022-JP, or ISO-8859-1, you're implicitly saying "fuck you" to 2/3 of the world's population since you can't support tasks as basic as "let me write my name" for them.

I'm not obligated to write software for every possible user at every point in time. It's perfectly acceptable for me to say, "I'm writing this program for my 1 friend who speaks Spanish" and have that be my requirements. But if I were to write software that had a hope of being broadly useful, UTF-8 everywhere doesn't get me there. I'd have to build it to be encoding-aware, and let my users configure the encoding(s) it uses.

Re: UTF-8 Everywhere

#167

Post by camgunz »

mark-r wrote: post #133
camgunz wrote: post #119

Earlier quoted context omitted.

I mean, I think we're both in the realm of [citation needed] here. I would argue that people index into strings quite a lot--whether that's because we thought UCS-2 would be enough for anybody or UTF-8 == ASCII and "it's probably fine" is academic. The solution is the same though: don't index into strings, don't assume an encoding until you've validated. That makes any "advantage" UTF-8 has disappear. If you really t…

The difference is that with UTF-8 you're much more likely to trip over those bugs in random testing. With UTF-16 you're likely to pass all your test cases if you didn't think to include a non-BMP character somewhere. Then someone feeds you an emoji character and you blow up.

Which is why you should be using a library for all this, that uses fuzzing and other robustness checks.

Re: UTF-8 Everywhere

#168

Post by alkonaut »

jodrellblank wrote: post #50

> Q: What do you think about Byte Order Marks? A: According to the Unicode Standard (v6.2, p.30): "Use of a BOM is neither required nor recommended for UTF-8". [...] Using BOMs would require all existing code to be aware of them, even in simple scenarios as file concatenation. This is unacceptable. Then your site "UTF-8 everywhere" is misnamed, because standards-following UTF-8 can have a BOM. It's not required or re…

100% agree.

> using BOMs would require all existing code to be aware of them, even in simple scenarios as file concatenation

Absolutely! Any app that writes UTF-files can (and probably should) avoid writing them. But any program that reads UTF files must handle a BOM. A lot of apps write UTF-8 including the BOM by default, for example Visual Studio.

You can NOT concatenate two UTF-8 streams and expect that the resulting stream is also a valid UTF-8 stream. NO tool should assume that, ever.

Re: UTF-8 Everywhere

#169

Post by Tyr42 »

xg15 wrote: post #162

> When writing a UTF-8 string to a file, it is the length in bytes which is important. Counting any other type of ‘characters’ is, on the other hand, not very helpful. So, suppose I have a UTF-8 string of n code units (bytes) length. Unfortunately my data structure only permits strings of length m How do I correctly truncate the string so it doesn't become invalid UTF-8 and won't show any unexpected gibberish when re…

https://play.rust-lang.org/?version=stable&mode=debug&editio...

Something like this? Check if each character pushes the byte total over the limit?

I think this might fail for combining characters though.

Re: UTF-8 Everywhere

#170

Post by camgunz »

crazygringo wrote: post #130
camgunz wrote: post #127

Earlier quoted context omitted.

That huge asset has become a liability. We always needed to become encoding-aware, but UTF-8's ASCII compatibility has let us delay it for decades, and caused exactly the confusion causing us to debate right now. So many engineers have been foiled by putting off learning about encodings. Joel Spolsky wrote an article, Atwood wrote an article, Python made a backwards incompatible change, etc. etc. etc. To be honest, I…

I just don't know what you're talking about. You can't rewrite all existing legacy software to support encodings. You just can't. A backwards-compatible format was a huge catalyst for widely supporting Unicode in the first place. What exactly are we delaying for decades? Engineers everywhere use Unicode today for new software. The battle has been won, moving forwards. And the vast majority of text isn't in computer c…

> You can't rewrite all existing legacy software to support encodings. You just can't. A backwards-compatible format was a huge catalyst for widely supporting Unicode in the first place.

Totally agree.

> What exactly are we delaying for decades?

Learning how encodings work and using that knowledge to write encoding-aware software.

> Engineers everywhere use Unicode today for new software. The battle has been won, moving forwards.

They do, but they're frequently foiled by on-disk encodings, filenames, internal string formats, network data, etc. etc. etc. All this stuff is outlined in TFA.

> And the vast majority of text isn't in computer code or even books. It's in the seemingly endless stream of content produced by journalists and social media each and every day

I concede I'm not likely to convince you here, but like, do you think Twitter is storing markup in their persistence layer? I doubt it. And even if there is some formatting, we're talking about here, not huge amounts of angle brackets.

But think about any car display. That's probably not markup. Think about ATMs. Log files. Bank records. Court records. Label makers. Airport signage. Road signage. University presses.

Post reply on HN