Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UTF-8 Everywhere

utf8everywhere.org

Post reply on HN
Print view View original

181–190 of 289 posts

Re: UTF-8 Everywhere

#181

Post by eska »

mhh__ wrote: post #115
DannyB2 wrote: post #78

Earlier quoted context omitted.

> Rust is one of the few programming languages that correctly doesn’t treat file paths as strings. Imagine if languages allowed subtypes of strings which are not directly assignment compatible. HtmlString SqlString String A String could be converted to HtmlString not by assignment, but through a function call, which escapes characters that the browser would recognize as markup. Similarly a String would be converted t…

Allowed you to? You could do that in C++ quite happily, it's just not useful enough. To bother implementing, at least.

It's absolutely useful enough, it's just that it's awful in C++ due to language limitations as opposed to other languages such as Haskell, where it is standard.

Re: UTF-8 Everywhere

#182

Post by tachyonbeam »

crazygringo wrote: post #156
droopyEyelids wrote: post #141

Earlier quoted context omitted.

counterpoint: A complicated program is never an easy win, and English is already spoken in every country in the world.

So your argument is... it's easier to teach billions of people fluent English... than for software to support UTF-8? You are aware that a majority of the world's population speaks no English whatsoever?

Playing the devil's advocate here. I am not a native English speaker, I'm a French speaker, but I'm happy that English is kind of the default international language. It's a relatively simple language. I actually make less grammar mistakes in English than I do in my native language. I suppose it's probably not a politically correct thing to say, the English are the colonists, the invaders, the oppressors, but eh, maybe it's also kind of a nice thing for world peace, if there is one relatively simple language that's accessible to everyone?

Go ahead and make nice libraries that support Unicode effectively, but I think it's fair game, for a small software development shop (or a one-person programming project), to support ASCII only for some basic software projects. Things are of course different when you're talking about governments providing essential services, etc.

Re: UTF-8 Everywhere

#183

Post by eska »

oconnor663 wrote: post #114
legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

> It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 Yes but , most programs expect to be able to print filepaths at least under some circumstances, like printing error messages. Even if a program is fully correct and doesn't assume an encoding in normal operation, it still has to assume one for printing. Filepaths that aren't utf-8 lead to a bunch of ����� in your output (at best). So I t…

In the Rust std one can easily use the lossless presentation with file APIs, and print a lossy version in error messages. I find this to be good enough.

Re: UTF-8 Everywhere

#184

Post by samatman »

xg15 wrote: post #162

> When writing a UTF-8 string to a file, it is the length in bytes which is important. Counting any other type of ‘characters’ is, on the other hand, not very helpful. So, suppose I have a UTF-8 string of n code units (bytes) length. Unfortunately my data structure only permits strings of length m How do I correctly truncate the string so it doesn't become invalid UTF-8 and won't show any unexpected gibberish when re…

Avoiding invalid UTF-8 is easy, almost trivial: just make sure you don't truncate in the middle of a code point.

The latter is fiendishly difficult to get right in all cases, the ugliest case being emoji flags. Being all-or-nothing on both sides of a ZWJ will get you most of the way there, however.

Re: UTF-8 Everywhere

#185

Post by DannyB2 »

torstenvl wrote: post #107
DannyB2 wrote: post #78

Earlier quoted context omitted.

> Rust is one of the few programming languages that correctly doesn’t treat file paths as strings. Imagine if languages allowed subtypes of strings which are not directly assignment compatible. HtmlString SqlString String A String could be converted to HtmlString not by assignment, but through a function call, which escapes characters that the browser would recognize as markup. Similarly a String would be converted t…

Failing that, you could also adopt a naming convention with prefixes to indicate what sort of thing it is you're storing there: hsCode = hsFromUs(usInputBuffer); ssStoredCode = ssFromHs(hsCode); https://www.joelonsoftware.com/2005/05/11/making-wrong-code-...

Yes. But having the compiler enforce it is your first line of defense. If it doesn't compile, you know there is an actual problem. In modern IDEs, you see these compile errors as quickly as you type them.

Re: UTF-8 Everywhere

#186

Post by DannyB2 »

mika9090 wrote: post #146
DannyB2 wrote: post #78

Earlier quoted context omitted.

> Rust is one of the few programming languages that correctly doesn’t treat file paths as strings. Imagine if languages allowed subtypes of strings which are not directly assignment compatible. HtmlString SqlString String A String could be converted to HtmlString not by assignment, but through a function call, which escapes characters that the browser would recognize as markup. Similarly a String would be converted t…

Try Pascal (free pascal or Delphi)

I used Pascal for the 80's and part of the 90's. Currently use Java. I almost tried Delphi, but my shop moved on to something else between Pascal and Java.

Re: UTF-8 Everywhere

#187

Post by samatman »

unwind wrote: post #99
mercer wrote: post #35

Earlier quoted context omitted.

"Ij" is also one sounds represented bij two letters, and I think capitalizing just the 'I' is pretty standard. As a Dutch person myself, I didn't even know that there's a glyph for it! We also have "ei", which sounds the same and was invented to annoy people learning Dutch. Then there's "oe", "eu", "ui". And just to fuck even more with people learning the language, we have "au" and "ou" which also sound the same. Oh,…

Spelling it "dike" helps keep people's minds on the right thing. :)

If you spell it "dijk" it's even less racy, because it's no longer a four-letter word.

Re: UTF-8 Everywhere

#188

Post by ghettoimp »

qiqitori wrote: post #22
malkia wrote: post #8

Still doesn't solve the fact that filesystems across different OS's allow invalid UTF8 sequences in the filenames. Maybe 99% of apps do not care, but even a simple "cp" tool should care. Filenames (and maybe other named resoureces) should be treated completely differently, and not blindly assumed that they are utf8 compatible.

Are you saying that operating systems (i.e. the kernel) should check and enforce encodings in filenames? 1) Why? 2) Bye bye backward compatibility and interoperability

Backward compatibility is a laudable goal and is not to be broken lightly. But sometimes, things are so fundamentally broken that we would be far better off with a clean break.

Interoperability is quite possibly a good argument for coming up with some reasonable restrictions on filenames. Today you could easily (case sensitive names, special characters, etc.) create a ZIP file or similar that cannot be successfully extracted on this platform or that.

In an excellent article, David A. Wheeler [1] lays out a compelling case against the status quo. TL;DR: bad filenames are too hard to handle correctly. Programs, standards, and operating systems already assume there are no bad filenames. Your programs will fail in numerous ways when they encounter bad filenames. Some of these failures are security problems.

He concludes: "In sum: It’d be far better if filenames were more limited so that they would be safer and easier to use. This would eliminate a whole class of errors and vulnerabilities in programs that “look correct” but subtly fail when unusual filenames are created (possibly by attackers)." He goes on to consider many ideas towards getting to this goal.

[1] https://dwheeler.com/essays/fixing-unix-linux-filenames.html

Re: UTF-8 Everywhere

#189

Post by microtherion »

ksec wrote: post #171
cryptonector wrote: post #118

Earlier quoted context omitted.

First of all, there is no new unification work ongoing. The Unicode Consortium moved on from that by moving on from UCS-2. UCS-2 drove unification as a way to preserve precious codespace. There used to be language tag codepoints for this, but they've been deprecated. Han unification is an accident of history: a result of UTF-8 not having existed until it was too late! There's not going to be a different new Unicode f…

The same could be said whether è é should be the same as e with different fonts. People who cares about it would complain. To those who only uses English it is only the same e .

I don't think that's the same, because e.g. in French, e, é, è, and ê are all used, with different pronunciations.

Re: UTF-8 Everywhere

#190

Post by DannyB2 »

gnarbarian wrote: post #85
DannyB2 wrote: post #78

Earlier quoted context omitted.

> Rust is one of the few programming languages that correctly doesn’t treat file paths as strings. Imagine if languages allowed subtypes of strings which are not directly assignment compatible. HtmlString SqlString String A String could be converted to HtmlString not by assignment, but through a function call, which escapes characters that the browser would recognize as markup. Similarly a String would be converted t…

You would probably like Java 1.4

I use Java 14 now. Java 11 in production.
Post reply on HN