Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UTF-8 Everywhere

utf8everywhere.org

Post reply on HN
Print view View original

271–280 of 289 posts

Re: UTF-8 Everywhere

#271

Post by anderspitman »

tialaramex wrote: post #219
carapace wrote: post #195

Earlier quoted context omitted.

Yeah, this. I have a pat "Unicode Rant" that boils down to this essentially. Having a catalog of standard numbers-to-glyphs (or symbols or whatever, little pictures humans use to communicate with) is awesome and useful (and all ASCII ever was) but trying to digitalize all of human language is much much more challenging.

But human language doesn't stop being "much much more challenging" if you decide not to engage. Sometimes (and this can even be an admirable choice) in some specialist applications it's acceptable to decide you won't embrace the complexity of human language. But in a lot of places where that's fine we already did this with the decimal digits such as in telephone numbers, or UPC/EAN product codes, so we don't need ASC…

Your sister's name doesn't render in my browser (stable Firefox on Linux 5.6). I'm sure I'm missing a fontpack or something. Again, I'm not saying ASCII is the solution, I'm saying Unicode is much more difficult to get right, and maybe we should call it something other than "plain text", since we already had a generally accepted meaning for that for many years. I'm usually in favor of making a new name for a thing rather than overloading an old name.

Re: UTF-8 Everywhere

#272

Post by account42 »

kllrnohj wrote: post #157
heyplanet wrote: post #25

I think UTF-8 was a mistake. It is a pain in the ass to have a variable number of bytes per char. In Ascii, you could easily know every character personally. No strange surprises. Also no surprises while reading black on white text and suddenly being confronted with clors [1]. [1] Also no surprises when writing a comment on HN like this one and having some characters stripped. I put in a smiley as the firs "o" in col…

> It is a pain in the ass to have a variable number of bytes per char. This is from API & language mistakes more than an issue with UTF-8 itself. If you actually design your API & system around being UTF-8, like Rust did, then there's really no issue for the programmer. The API enforces the rules, and still gives you things like a simple character iterator (with characters being 32-bit, so that it actually fits: http…

Character (code point) iterators are useless.

For parsing text-based formats, UTF-8 has the nice property that the encoded byte sequence of a character is not a subsequence of the encoding for any other chracter or sequence of other characters. This means splitting on byte sequences of UTF-8 works just as well as spliting on code points.

And for text editing you need to deal with grapheme clusers anyway, which can be made up of a variable number of code points - so having these be made up of a variable number of bytes doesn't make anything worse.

Re: UTF-8 Everywhere

#273

Post by Symbiote »

mmm_grayons wrote: post #229
Symbiote wrote: post #218

Earlier quoted context omitted.

English isn't even ASCII anyway. Some loanwords like façade or café retain their accents. Units like ° £ € and symbols like © ® × ÷ ½ aren't ASCII. It doesn't take much to need one of these cases in a project.

I know almost no one who actually types the accented e, let alone the c with the cedilla. I scarcely ever see the degree symbol typed. Rather, I see facade, cafe, and "degrees". That aside, the big problem with unicode is not those characters; they're a simple two-byte extension. They obey the simple bijective mapping of binary character character on screen. Unicode doesn't. You have to deal with multiple code points…

The first point is just because of the keys on a keyboard.

I see many uses of "pounds" or "GBP" on HN. Anyone with the symbol on the keyboard (British and Irish obviously, plus several other European countries) types £. When people use a phone keyboard, and a long-press or symbol view shows $, £ and €, they can choose £.

Danish people use ½ and § (and £). These keys are labelled on the standard Danish Windows keyboard.

There's plenty of scope for implementing enough Unicode to support most Latin-like languages without going as far as supporting vertical or RTL text.

Re: UTF-8 Everywhere

#274

Post by sunyc »

xg15 wrote: post #162

> When writing a UTF-8 string to a file, it is the length in bytes which is important. Counting any other type of ‘characters’ is, on the other hand, not very helpful. So, suppose I have a UTF-8 string of n code units (bytes) length. Unfortunately my data structure only permits strings of length m How do I correctly truncate the string so it doesn't become invalid UTF-8 and won't show any unexpected gibberish when re…

use U8_TRUNCATE macros in icu4c

Re: UTF-8 Everywhere

#275

Post by tialaramex »

anderspitman wrote: post #271
tialaramex wrote: post #219

Earlier quoted context omitted.

But human language doesn't stop being "much much more challenging" if you decide not to engage. Sometimes (and this can even be an admirable choice) in some specialist applications it's acceptable to decide you won't embrace the complexity of human language. But in a lot of places where that's fine we already did this with the decimal digits such as in telephone numbers, or UPC/EAN product codes, so we don't need ASC…

Your sister's name doesn't render in my browser (stable Firefox on Linux 5.6). I'm sure I'm missing a fontpack or something. Again, I'm not saying ASCII is the solution, I'm saying Unicode is much more difficult to get right, and maybe we should call it something other than "plain text", since we already had a generally accepted meaning for that for many years. I'm usually in favor of making a new name for a thing ra…

Firefox does full font fallback. So this means your system just isn't capable of rendering her name (which yes you might be able to fix if you wanted to by installing font packages). If you don't understand Han characters that's an acceptable situation, the dotted boxes (which I assume rendered instead) alert you that there is something here you can't display properly but if you know you can't understand it even if it's displayed there's no need to bother.

It really is just plain text. Human writing systems were always this hard, and "for many years" what you had were separate independent understandings of what "plain text" means in different environments, which makes interoperability impossible. Unicode is mostly about having only one "plain text" rather than dozens.

It is not mandatory that your 80x25 terminal learn how to display Linear B, you can't read Linear B and you probably have no desire to learn how and no interest in any text written in it. But Unicode means your computer agrees with everybody else's computer that it's Linear B, and not a bunch of symbols for drawing Space Invaders, or the manufacturer's logo, if you fix a typo in a document I wrote that has some Linear B in it, your computer doesn't replace the Linear B with question marks, or erase the document, since it knows what that is even if you can't read it and it doesn't know how to display it.

Re: UTF-8 Everywhere

#276

Post by vortico »

What is this C++ `narrow()/widen()` function mentioned in the Windows section? At the risk of asking to be spoonfed, can someone give the source code of a function that takes a UTF-8 `std::string` and gives a UTF-16 `std::wstring`?

Re: UTF-8 Everywhere

#277

Post by cryptonector »

innocenat wrote: post #252
cryptonector wrote: post #239

Earlier quoted context omitted.

Do you want to see all Han texts in your preferred style, or do you want to see them in the style corresponding to the language they were written in?

In the original language, of course. Why is that even a question. That is like asking if Geek people would want to read Latin script using Greek alphabet or not. You keep using the word 'style', so you agree that α is a style of a? Then I have no more comment. It's not 'style' at all.

> Why is that even a question.

Why wouldn't it be. As I'm not a user of CJK scripts it seemed like a fair question to ask.

> You keep using the word 'style', so you agree that α is a style of a?

I didn't have a better word for it. Oddly alpha is getting different renderings in the text input box than in your comment.

So Han unification makes CJK as hard to read for you as mixing Greek and Latin scripts would for me. That's what I wanted to understand. Thanks.

Re: UTF-8 Everywhere

#278

Post by carapace »

perilunar wrote: post #261
carapace wrote: post #195

Earlier quoted context omitted.

Yeah, this. I have a pat "Unicode Rant" that boils down to this essentially. Having a catalog of standard numbers-to-glyphs (or symbols or whatever, little pictures humans use to communicate with) is awesome and useful (and all ASCII ever was) but trying to digitalize all of human language is much much more challenging.

> and all ASCII ever was Except that's not true. The ASCII control codes were never glyphs, but were used to control the hardware.

Sorry, what I mean was ASCII wasn't an encoding of the English language, just an encoding of the English alphabet and some other symbols.

You're quite right that, uh, meta-linguistic symbols are also in there and that does kind of complicate my argument.

Re: UTF-8 Everywhere

#279

Post by sitzkrieg »

benibela wrote: post #206
sitzkrieg wrote: post #48

Earlier quoted context omitted.

git will also do this, so on a fs that allowa arbitrarily byte named files, you end up with tree objects of same name which makes digging them out later "fun"

I have a repository full of such files: https://github.com/benibela/nasty-files You can clone the repository, and then you cannot delete it with tools that expect utf-8 names (like KDE's Dolphin)

haha pretty handy!

Re: UTF-8 Everywhere

#280

Post by benibela »

robocat wrote: post #203
mika9090 wrote: post #146

Earlier quoted context omitted.

Try Pascal (free pascal or Delphi)

AFAIK they just provide type name aliases, which do not enforce or warn of you if you mix the “types”.

They have changed it

Now the string types have an encoding and the string themselves, too. When you assign a string to a string variable with a type of a different encoding, the string is automatically converted.

But it is causing a huge mess. Especially with existing code. When you have a library using utf-8 and one library using the default codepage, that is not valid anymore. Although you can manually override the encoding for each string, so any string might have any encoding regardless of its type.

Post reply on HN