Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UTF-8 Everywhere

utf8everywhere.org

Post reply on HN
Print view View original

201–210 of 289 posts

Re: UTF-8 Everywhere

#201

Post by gumby »

legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

> Unix paths don’t need to be valid UTF-8

And a lucky thing too; OSes that do have UTF-8 filesystems don’t always agree on how to apply canonicalization, much less how to deal with canonicalization differences between user entered data and normalized filesystem names.

Re: UTF-8 Everywhere

#202

Post by gpvos »

xeeeeeeeeeeenu wrote: post #10
gpvos wrote: post #5

Earlier quoted context omitted.

Have they fixed all the bugs with that pseudocodepage?

Bugs like WriteFile() reporting the wrong number of bytes written with 65001 codepage were fixed years ago.

Ah, that's surprising, Microsoft was very stubbornly not doing that for at least a decade and a half.

In fact, the FAQ in TFA (questions 9 and 20) mentions that there are still problems with CP_UTF8 (65001). Is the article out of date? Can someone respond to those statements?

Re: UTF-8 Everywhere

#203

Post by robocat »

mika9090 wrote: post #146
DannyB2 wrote: post #78

Earlier quoted context omitted.

> Rust is one of the few programming languages that correctly doesn’t treat file paths as strings. Imagine if languages allowed subtypes of strings which are not directly assignment compatible. HtmlString SqlString String A String could be converted to HtmlString not by assignment, but through a function call, which escapes characters that the browser would recognize as markup. Similarly a String would be converted t…

Try Pascal (free pascal or Delphi)

AFAIK they just provide type name aliases, which do not enforce or warn of you if you mix the “types”.

Re: UTF-8 Everywhere

#204

Post by macintux »

cryptonector wrote: post #122
klodolph wrote: post #111

Earlier quoted context omitted.

I’m not going to try and minimize the problem, here. Han unification was pushed through by western interests, by my understanding. However, most Unicode characters are identical or nearly identical in Chinese and Japanese. Characters with “significant” visual differences got encoded as different Unicode characters. The same thing applies to simplified and traditional Chinese characters. So for a given “Han character”…

As I understand it Han unification happened because at the time all there was was UCS-2 -no UTF-16, no UTF-8- so codespace was tight and precious, and that motivated codespace preserving optimizations, of which Han unification is the notable one. To avoid that they needed to have invented UTF-8 many years earlier. Perhaps if the people designing UTF-8 were more diverse they might have felt the necessity to invent UTF…

UTF-8 was sketched on a placemat as a response to a different idea. It seems likely that had it not arisen in a moment of inspiration by a genius, we would be stuck with another inferior design by committee.

https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt

Re: UTF-8 Everywhere

#205

Post by ignoramous »

TheCoelacanth wrote: post #196
ignoramous wrote: post #159

Earlier quoted context omitted.

> You only need one sentence to explain why ASCII isn't sufficient Nitpick: ASCII is sufficient when you consider that Base64, despite its 33% overhead from representing 6 bits with 8 bits, makes life easier for certain classes of software.

Base64 is an encoding for representing bytes[0] in ASCII. That doesn't help you represent text unless you already have an encoding for representing text in bytes (e.g. UTF8). [0] Octets if you want to be pedantic

What I was alluding to is, I often convert any binary data, including text, to Base64 to avoid dealing with cross platform, cross language, cross format, cross storage, cross network data-handling. Only the layer that needs to deal with the blob's actual string representation needs to worry about encoding schemes that are outside the purview of the humble ASCII table.

Re: UTF-8 Everywhere

#206

Post by benibela »

sitzkrieg wrote: post #48
legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

git will also do this, so on a fs that allowa arbitrarily byte named files, you end up with tree objects of same name which makes digging them out later "fun"

I have a repository full of such files: https://github.com/benibela/nasty-files

You can clone the repository, and then you cannot delete it with tools that expect utf-8 names (like KDE's Dolphin)

Re: UTF-8 Everywhere

#207

Post by roblabla »

acdha wrote: post #200
jcranmer wrote: post #89

Earlier quoted context omitted.

> It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) A decent fraction of software can impose rules on the portion of the filesystem within their control. A tool like mv or vim has to be prepared to handle any filepath encoding. But something like a VCS…

The history of Git and Subversion handling filenames makes me think that the opposite is true: A VCS which doesn't handle arbitrary byte-strings will have weird edge cases which prevent users from adding files or accessing them, possibly even “losing” data in a local checkout. This is especially tedious because it'll appear to work for a while until someone first tries to commit an unusual file or checks it out with…

My understanding is, you can't treat the filename as an arbitrary bytestring, since you have to transcode it across platforms, otherwise the filename won't show up properly everywhere. E.G. if I make a file named "test" on unix, it will be UTF-8 (assuming sane unix). If on windows I create a file with the filename "test", encoded as UTF-8, it will show up as worthless garbage in explorer.exe since it will decode it to UTF-16.

So VCS needs to know the filename encoding in order to work properly.

Re: UTF-8 Everywhere

#208

Post by toast0 »

xg15 wrote: post #162

> When writing a UTF-8 string to a file, it is the length in bytes which is important. Counting any other type of ‘characters’ is, on the other hand, not very helpful. So, suppose I have a UTF-8 string of n code units (bytes) length. Unfortunately my data structure only permits strings of length m How do I correctly truncate the string so it doesn't become invalid UTF-8 and won't show any unexpected gibberish when re…

> How do I correctly truncate the string?

Refuse to accept a string that is overlong, and require an interactive user (hopefully one literate in the language) to truncate it for you. In a non-interactive context, you can't.

Re: UTF-8 Everywhere

#209

Post by ori_b »

akiselev wrote: post #172
mhh__ wrote: post #115

Earlier quoted context omitted.

Allowed you to? You could do that in C++ quite happily, it's just not useful enough. To bother implementing, at least.

They're not worth the effort in C++ because it doesn't have strictly enforced affine/dependent types. The GP is invisioning a language that does.

Why do you need them to enforce that only escaped strings are passed to functions?

     html::append(html::string text);
with an constructor

     html::string(std::string)
that handled escaping seems like it'd work just fine.

Re: UTF-8 Everywhere

#210

Post by hutzlibu »

As someone who experienced serious pain with broken strings that I sometimes only discovered, after the original files were gone and new special characters were integrated, I directed quite some anger to the fact, that computer systems are internal mostly operated in english only, so usually nobody notices bugs with wrong character encoding. So I share the sentiment of the article ..

I do not want to think about UTF encoding, when I simply create a 7z or tar file, without even programming. But I learned the hard way, I had to. I never even found out for example, if it was/is a bug with 7z, tar, rsync, scite text editor/ notepad++ .. or just wrong usage/configuration. I just had(and still have even now my workflow is clean) a special first file/codeline with special characters, I checked to be correct, after compressing, rsyncing between different systems. Especially between windows and linux. But it probably helps, that I don't have to do that anymore.

Post reply on HN