Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UTF-8 Everywhere

utf8everywhere.org

Post reply on HN
Print view View original

211–220 of 289 posts

Re: UTF-8 Everywhere

#211

Post by tialaramex »

anoncake wrote: post #161
cygx wrote: post #152

Earlier quoted context omitted.

Han unification was pushed through by western interests, by my understanding. Note that as far as I'm aware, the interest in question was the initial 16-bit limit of the character set and later on the non-proliferation of competing standards. Also note that while Han unification is the most prominent example, there are technically similar cases, which just aren't as charged culturally. For one, Unicode doesn't encode…

That's not the same thing. Fraktur is just a style of fonts, antiqua and fraktur letters are semantically the same.

It's actually exactly the same thing. The Han Unification didn't smash together unrelated squiggles that just happened to look similar, they were semantically the same - scholars of the Han writing system spent a bunch of time deciding what is or is not the same squiggle just drawn differently, like Fraktur, and today people are annoyed because, as you'd expect some of them believed that "style of fonts" was integral to the meaning anyway.

Re: UTF-8 Everywhere

#212

Post by gramakri »

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere

I think in unix world, null terminated strings are the default. It doesn't need to be valid UTF-8 even. For display purposes, the shell uses the locale setting

Re: UTF-8 Everywhere

#213

Post by gpderetta »

eska wrote: post #181
mhh__ wrote: post #115

Earlier quoted context omitted.

Allowed you to? You could do that in C++ quite happily, it's just not useful enough. To bother implementing, at least.

It's absolutely useful enough, it's just that it's awful in C++ due to language limitations as opposed to other languages such as Haskell, where it is standard.

How would be awful in c++? It seems trivial to do, basic_string is already templated and distinct instantiations are not mutually compatible by default. In fact wstring, u8string, u16string, u32string exist today in the language simply as distinct instatiantions of basic_string. You can crate your own by picking a new char type. Algorithms can be and are, generic and work on any string type.

Re: UTF-8 Everywhere

#214

Post by malkia »

qiqitori wrote: post #22
malkia wrote: post #8

Still doesn't solve the fact that filesystems across different OS's allow invalid UTF8 sequences in the filenames. Maybe 99% of apps do not care, but even a simple "cp" tool should care. Filenames (and maybe other named resoureces) should be treated completely differently, and not blindly assumed that they are utf8 compatible.

Are you saying that operating systems (i.e. the kernel) should check and enforce encodings in filenames? 1) Why? 2) Bye bye backward compatibility and interoperability

1) Nope. 2) Yes, we need to keep backward compatibility.

What I'm saying is that promoting UTF8 everywhere, without specifically stressing the fact that filesystems (in general) do no observe UFT8, leads to API/LIB designs that lack good support there.

Path/filename/dirname/whatever should be a different kind of "string".

Re: UTF-8 Everywhere

#215

Post by seppel »

xg15 wrote: post #162

> When writing a UTF-8 string to a file, it is the length in bytes which is important. Counting any other type of ‘characters’ is, on the other hand, not very helpful. So, suppose I have a UTF-8 string of n code units (bytes) length. Unfortunately my data structure only permits strings of length m How do I correctly truncate the string so it doesn't become invalid UTF-8 and won't show any unexpected gibberish when re…

> How do I correctly truncate the string so it doesn't become invalid UTF-8 and won't show any unexpected gibberish when rendered? (E.g., the truncated string doesn't suddenly contain any glyphs or grapheme clusters that weren't in the original string)

Cropping strings is a hard problem for ASCII strings as well. It can even be a security problem if the cropped part contains important information that alters the meaning of the first part (Something like "DELETE FROM table_name [WHERE condition]" or natural language where the cropped part is the condition or the negation).

But even if you dont care about this: If you care about cropping visually nicely, you want some ellipsis at the end, you dont want to crop in the middle of the word (if possible), etc. In the end, you need some nice text processing anyway.

Or you reject strings that are too long.

Re: UTF-8 Everywhere

#216

Post by user5994461 »

Spivak wrote: post #137
marcosdumay wrote: post #103

Earlier quoted context omitted.

> Unix paths don’t need to be valid UTF-8 Yet, your shell will treat them like UTF-8 just as well. As will the standard library of almost every programming language, as you noticed. If you open one such file in most text editors, they will render whatever is in it as UTF-8. If you use text manipulating utilities, they will work with it as if it was encoded in UTF-8. It's mostly the Linux kernel that disagrees. Everyt…

Which is a silly position since the kernel is the only thing that matters. You're right that not too many people will complain if your program crashes on non-UTF-8 paths. Same with spaces in group names. 100% valid and accepted. Breaks a ridiculous amount of software if you actually do it. But that doesn't mean it's right. It just means that we have a calcified convention.

What's right is that the software will be basically unusable outside of the US if it's crashing on non-ASCII characters in paths.

ñ é ß characters appear in every text in Spanish, French, German, etc and they go in filenames too.

Imagine if a table was called a tâble in English? Surely it would be outrageous to have software crash when you try to use the word tâble.

Re: UTF-8 Everywhere

#217

Post by malkia »

GnarfGnarf wrote: post #153

I came to the same conclusion years ago. My app is Win32, but I never defined UNICODE or used the TCHAR abomination. All strings are stored as UTF8 until they are passed to Win32 APIs, whereupon they are converted to UCS-2. I explicitly call the wchar version of functions (ex: TextOutW). This strategy enabled me to transition easily and safely from single-byte ASCII (Windows 3.1) to Unicode. The database is also UTF8…

Calling the "A", instead of "W" functions might be some small perf hit (don't know if it matters), but for some functionality you need to call the "W" functions, for example to break the limit of 256 or was it 260 characters, up to 32768 (or was it 16384).

:)

Re: UTF-8 Everywhere

#218

Post by Symbiote »

tachyonbeam wrote: post #182
crazygringo wrote: post #156

Earlier quoted context omitted.

So your argument is... it's easier to teach billions of people fluent English... than for software to support UTF-8? You are aware that a majority of the world's population speaks no English whatsoever?

Playing the devil's advocate here. I am not a native English speaker, I'm a French speaker, but I'm happy that English is kind of the default international language. It's a relatively simple language. I actually make less grammar mistakes in English than I do in my native language. I suppose it's probably not a politically correct thing to say, the English are the colonists, the invaders, the oppressors, but eh, mayb…

English isn't even ASCII anyway.

Some loanwords like façade or café retain their accents.

Units like ° £ € and symbols like © ® × ÷ ½ aren't ASCII.

It doesn't take much to need one of these cases in a project.

Re: UTF-8 Everywhere

#219

Post by tialaramex »

carapace wrote: post #195
pjscott wrote: post #101

Earlier quoted context omitted.

Unicode is complicated because the languages it needs to handle are, alas, complicated. UTF-8 is super simple. It's a variable-length encoding for 21-bit unsigned integers. Wikipedia gives a handy table showing how it works: https://en.wikipedia.org/wiki/UTF-8#Description

Yeah, this. I have a pat "Unicode Rant" that boils down to this essentially. Having a catalog of standard numbers-to-glyphs (or symbols or whatever, little pictures humans use to communicate with) is awesome and useful (and all ASCII ever was) but trying to digitalize all of human language is much much more challenging.

But human language doesn't stop being "much much more challenging" if you decide not to engage.

Sometimes (and this can even be an admirable choice) in some specialist applications it's acceptable to decide you won't embrace the complexity of human language. But in a lot of places where that's fine we already did this with the decimal digits such as in telephone numbers, or UPC/EAN product codes, so we don't need ASCII.

In most other places insisting upon ASCII is just an annoying limitation, it's annoying not being able to write your sister's name in the name of the JPEG file, regardless of whether her name is 林鳳嬌 or Jenny Smith, and it jumps out at you if the product you're using is OK with Jenny Smith but not 林鳳嬌.

You might think well, OK, but there weren't problems in ASCII. The complexity is Unicode's fault. Think about Sarah O'Connor? That apostrophe will often break people's software without any help from Unicode.

Re: UTF-8 Everywhere

#220

Post by justinfrankel »

I would leave this on their facebook page, but fuck facebook so I'm posting it here hoping that someone will find it useful.

We have a zlib-licensed wrapper header for some commonly-used win32 APIs to make them take UTF-8, see:

https://github.com/justinfrankel/WDL/blob/master/WDL/win32_u...

https://github.com/justinfrankel/WDL/blob/master/WDL/win32_u...

(This is used in REAPER so it's relatively well tested!)

Post reply on HN