Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UTF-8 Everywhere

utf8everywhere.org

Post reply on HN
Print view View original

251–260 of 289 posts

Re: UTF-8 Everywhere

#251

Post by cygx »

anoncake wrote: post #246
cygx wrote: post #165

Earlier quoted context omitted.

There are differences as well as similarities. I'm no expert, but shouldn't, say, U+4ECA still translate to 'now' no matter if you draw a particular line horizontally or diagonally? There are also some mandatory[1] ligatures in Fraktur unavailable in Unicode. What if I wanted to preserve that distinction in historic writing? edit: [1] I think the mandatory ones are actually there (just not in Fraktur), it's some opti…

> There are differences as well as similarities. I'm no expert, but shouldn't, say, U+4ECA still translate to 'now' no matter if you draw a particular line horizontally or diagonally? No, since "now" is an English word, not a Japanese or Chinese one. > There are also some mandatory[1] ligatures in Fraktur unavailable in Unicode. Unicode doesn't encode ligatures except for backwards compatibility.

Unicode doesn't encode ligatures except for backwards compatibility.

And it doesn't encode separate variants for unified Han characters. As in, that's not an argument, just a description of the status quo.

Re: UTF-8 Everywhere

#252

Post by innocenat »

cryptonector wrote: post #239
innocenat wrote: post #131

Earlier quoted context omitted.

It's only stylistic issue if you also consider a and α (alpha) to also be just stylistic different. I have learned to live with it, but it is very annoying.

Do you want to see all Han texts in your preferred style, or do you want to see them in the style corresponding to the language they were written in?

In the original language, of course. Why is that even a question. That is like asking if Geek people would want to read Latin script using Greek alphabet or not.

You keep using the word 'style', so you agree that α is a style of a? Then I have no more comment. It's not 'style' at all.

Re: UTF-8 Everywhere

#253

Post by alkonaut »

nybble41 wrote: post #225
alkonaut wrote: post #168

Earlier quoted context omitted.

100% agree. > using BOMs would require all existing code to be aware of them, even in simple scenarios as file concatenation Absolutely! Any app that writes UTF-files can (and probably should) avoid writing them. But any program that reads UTF files must handle a BOM. A lot of apps write UTF-8 including the BOM by default, for example Visual Studio. You can NOT concatenate two UTF-8 streams and expect that the result…

> You can NOT concatenate two UTF-8 streams and expect that the resulting stream is also a valid UTF-8 stream. Actually you can; the ability to concatenate UTF-8 streams is an intentionally part of the design of UTF-8. The BOM is an ordinary Unicode code point and can occur in the middle of a valid UTF-8 stream, where it should be treated as either a zero-width non-breaking space or an unsupported character (which on…

Interesting, I thought a BOM-in-the-middle was invalid. I know apps are even more likely to choke on that than a leading BOM though.

In any case, you need to handle it in every app that claims to read UTF. The loss of compatibility is indeed the biggest problem and I agree the BOM should be omitted when possible, but that doesn’t change that it’s part of the spec and millions of UTF files have a BOM.

Even if 100% of all apps stopped using a BOM today you couldn’t ignore it in a parser.

Re: UTF-8 Everywhere

#254

Post by ordu »

arendtio wrote: post #135
marcosdumay wrote: post #103

Earlier quoted context omitted.

> Unix paths don’t need to be valid UTF-8 Yet, your shell will treat them like UTF-8 just as well. As will the standard library of almost every programming language, as you noticed. If you open one such file in most text editors, they will render whatever is in it as UTF-8. If you use text manipulating utilities, they will work with it as if it was encoded in UTF-8. It's mostly the Linux kernel that disagrees. Everyt…

Doesn't it depend on your locales? At least for source-based Linux distributions (Gentoo, Exherbo) I remember that you have to define the locales you want to use and which ones should be the default. And when I build a system without UTF-8 locales, I doubt that the shell will treat paths as UTF-8.

Shell is like most of the programs doesn't need to bother about encoding of filenames. Mostly doesn't. I could use LANG=C and then TAB autocompletes filenames even cyrillic ones, because bash wouldn't mind encodings: terminal uses utf8, so it could output utf8 without any help from bash. Though it nevertheless pain to work with this sometimes, because readline fails to count visible characters (counting bytes instead). You type chars into command line, fill it to the end, then cursor jumps to the left side of the terminal and continues, placing characters over other characters. It is like \r used instead of \n.

`LANG=C ls` tries to be smarter and uses escape-syntax for everything except printable ASCII characters. But other utilities from coreutils work even with a locale that doesn't match file name encoding. cp, mv, grep, ...

The point is: it doesn't matter what encoding strings use until you tried to render string on a screen.

Re: UTF-8 Everywhere

#255

Post by anoncake »

tialaramex wrote: post #248
anoncake wrote: post #245

Earlier quoted context omitted.

Chinese characters represent the Chinese words or parts thereof, Japanese ones represent Japanese words and parts thereof. That is a semantic difference.

So what you're saying is that because 'chat' in English and 'chat' in French are quite different words with very different meanings, you believe there should be a separate letter 'c' for English and French to enable us to tell those words apart?

The Latin alphabet is not logographic.

Re: UTF-8 Everywhere

#256

Post by anoncake »

cygx wrote: post #251
anoncake wrote: post #246

Earlier quoted context omitted.

> There are differences as well as similarities. I'm no expert, but shouldn't, say, U+4ECA still translate to 'now' no matter if you draw a particular line horizontally or diagonally? No, since "now" is an English word, not a Japanese or Chinese one. > There are also some mandatory[1] ligatures in Fraktur unavailable in Unicode. Unicode doesn't encode ligatures except for backwards compatibility.

Unicode doesn't encode ligatures except for backwards compatibility. And it doesn't encode separate variants for unified Han characters. As in, that's not an argument, just a description of the status quo.

Of course it is. Ligatures aren't characters, they're glyphs that represent multiple characters. Unicode does not encode glyphs, that's simply not its job. No more than encoding what font to use or when to render text in italic.

Re: UTF-8 Everywhere

#257

Post by cygx »

anoncake wrote: post #256
cygx wrote: post #251

Earlier quoted context omitted.

Unicode doesn't encode ligatures except for backwards compatibility. And it doesn't encode separate variants for unified Han characters. As in, that's not an argument, just a description of the status quo.

Of course it is. Ligatures aren't characters, they're glyphs that represent multiple characters. Unicode does not encode glyphs, that's simply not its job. No more than encoding what font to use or when to render text in italic.

Which is the whole point of Han unification, the argument being that whether or not a particular line in U+4ECA is horizontal or diagonal is just like that. What's the difference?

Re: UTF-8 Everywhere

#258

Post by DagAgren »

kazinator wrote: post #176
DagAgren wrote: post #158

Earlier quoted context omitted.

ASCII doesn't have have all the punctuation regularly used in English.

ASCII doesn't have a direct representation of all the punctuation used in English print , like 66 99 quotes, and different kinds of dashes (distinct from minus). For non-print, it's entirely fine. Typesetting should be handled by a markup language anyway. Adding a few characters to Notepad doesn't create a typesetting system. A typesetting system needs to be able to do kerning, ligatures, justification. Not to mentio…

Why would print be different here? A screen is as much "print" as a paper is these days.

Choosing correct punctuation is not typesetting, either.

Re: UTF-8 Everywhere

#259

Post by asiachick »

legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

Maybe Linux (and other OSes) should deprecate non utf8 filenames and start disallowing creating filenames that aren't valid utf-8?

It seems silly that directory entries are just binary blobs and yet 99.99% of all software I know of passes around paths as strings. We could ask all software to stop that (boil the ocean) or we could just ask the OSes to stop it (many less OSes than all the other software)

Re: UTF-8 Everywhere

#260

Post by stubish »

legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

Read that quote again: 'considered UTF-8 by default almost everywhere'. It is absolutely the truth. While you can stuff non-UTF8 in, almost all of your tools will handle it badly. Even Rust programs wanting to log the file name. It is the same as considering email addresses case sensitive; technically correct, practically shooting yourself in the food.
Post reply on HN