Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

UTF-8 Everywhere

utf8everywhere.org

Post reply on HN
Print view View original

241–250 of 289 posts

Re: UTF-8 Everywhere

#241

Post by jkoudys »

DannyB2 wrote: post #78
legulere wrote: post #12

> In the UNIX world, narrow strings are considered UTF-8 by default almost everywhere. Because of that, the author of the file copy utility would not need to care about Unicode It couldn’t be further from the truth. Unix paths don’t need to be valid UTF-8 and most programs happily pipe the mess through into text that should be valid. (Windows filenames don’t have to be proper UTF-16 either) Rust is one of the few pro…

> Rust is one of the few programming languages that correctly doesn’t treat file paths as strings. Imagine if languages allowed subtypes of strings which are not directly assignment compatible. HtmlString SqlString String A String could be converted to HtmlString not by assignment, but through a function call, which escapes characters that the browser would recognize as markup. Similarly a String would be converted t…

Not quite at that level, but rust does have OsStrings (managed the same way as the OS, often but not always utf8), and CStrings (basically just byte buffers - just like c likes). There are special rules around inclusion of nulls and null terminators. It'll give the benefits of the behaviour you mentioned - not allowing an invalid string type for a function call.

The sqlx crate for rust also has a macro called query!, which (at compile time) validates the SQL and created a value of type "record". Similar idea there, since you'll get early exceptions thrown by the compiler if you write sql with errors in it.

Re: UTF-8 Everywhere

#242

Post by lisper »

tester89 wrote: post #222
lisper wrote: post #192

Earlier quoted context omitted.

That makes two of us. But they aren't strings :-) (Seriously though, is it pathnames you don't understand or logical hosts? Because CL pathnames are actually pretty straightforward. Logical hosts, on the other hand, are a hot mess.)

I don’t really understand how the #P”” “strings” aren’t différent from strings.

They are different types.

  ? (type-of "this is a string")
  (SIMPLE-BASE-STRING 16)
  ? (type-of #P"/this/is/a/pathname")
  PATHNAME
You can't perform string operations on a pathname.

  ? (subseq "This is a string" 5 15)
  "is a strin"
  
  ? (subseq #P"/This/is/a/pathname" 5 15)
  > Error: The value #P"/This/is/a/pathname" is not of the expected type SEQUENCE.
You can perform pathname operations on a string, but only because the string is automatically converted into a pathname first.

Re: UTF-8 Everywhere

#243

Post by lisper »

tester89 wrote: post #223
gumby wrote: post #199

Earlier quoted context omitted.

They are pretty straightforward: they are just path structures rather than path names that may turn into single strings when supplied to your kernel. Or, depending on the OS maybe only part of the name is turned into a string and part determines which device or syntax applies. All of which is abstracted away by the path objects. Back in the 1970s when thins first appeared on lisp machines is was not uncommon to use r…

Ohh, so they’re kinda like date objects?

Exactly. A pathname in CL is a data structure. Some of its fields are strings, but others (like version) are not.

Re: UTF-8 Everywhere

#244

Post by naniwaduni »

TheCoelacanth wrote: post #196
ignoramous wrote: post #159

Earlier quoted context omitted.

> You only need one sentence to explain why ASCII isn't sufficient Nitpick: ASCII is sufficient when you consider that Base64, despite its 33% overhead from representing 6 bits with 8 bits, makes life easier for certain classes of software.

Base64 is an encoding for representing bytes[0] in ASCII. That doesn't help you represent text unless you already have an encoding for representing text in bytes (e.g. UTF8). [0] Octets if you want to be pedantic

Base64 encodes sextets. The mapping from octets to sextets is mostly settled for set of three octets at a time, but the situation for lengths not divisible by 6 is a mess.

Re: UTF-8 Everywhere

#245

Post by anoncake »

tialaramex wrote: post #211
anoncake wrote: post #161

Earlier quoted context omitted.

That's not the same thing. Fraktur is just a style of fonts, antiqua and fraktur letters are semantically the same.

It's actually exactly the same thing. The Han Unification didn't smash together unrelated squiggles that just happened to look similar, they were semantically the same - scholars of the Han writing system spent a bunch of time deciding what is or is not the same squiggle just drawn differently, like Fraktur, and today people are annoyed because, as you'd expect some of them believed that "style of fonts" was integral…

Chinese characters represent the Chinese words or parts thereof, Japanese ones represent Japanese words and parts thereof. That is a semantic difference.

Re: UTF-8 Everywhere

#246

Post by anoncake »

cygx wrote: post #165
anoncake wrote: post #161

Earlier quoted context omitted.

That's not the same thing. Fraktur is just a style of fonts, antiqua and fraktur letters are semantically the same.

There are differences as well as similarities. I'm no expert, but shouldn't, say, U+4ECA still translate to 'now' no matter if you draw a particular line horizontally or diagonally? There are also some mandatory[1] ligatures in Fraktur unavailable in Unicode. What if I wanted to preserve that distinction in historic writing? edit: [1] I think the mandatory ones are actually there (just not in Fraktur), it's some opti…

> There are differences as well as similarities. I'm no expert, but shouldn't, say, U+4ECA still translate to 'now' no matter if you draw a particular line horizontally or diagonally?

No, since "now" is an English word, not a Japanese or Chinese one.

> There are also some mandatory[1] ligatures in Fraktur unavailable in Unicode.

Unicode doesn't encode ligatures except for backwards compatibility.

Re: UTF-8 Everywhere

#247

Post by bumblebritches5 »

This article is actually really bad.

It confuses Code Units (bytes in UTF-8, "short" in UTF-16), and CodePoints aka Unicode Scalar Values aka the actual assigned value for each "character", and Graphemes aka "characters".

among other things.

Re: UTF-8 Everywhere

#248

Post by tialaramex »

anoncake wrote: post #245
tialaramex wrote: post #211

Earlier quoted context omitted.

It's actually exactly the same thing. The Han Unification didn't smash together unrelated squiggles that just happened to look similar, they were semantically the same - scholars of the Han writing system spent a bunch of time deciding what is or is not the same squiggle just drawn differently, like Fraktur, and today people are annoyed because, as you'd expect some of them believed that "style of fonts" was integral…

Chinese characters represent the Chinese words or parts thereof, Japanese ones represent Japanese words and parts thereof. That is a semantic difference.

So what you're saying is that because 'chat' in English and 'chat' in French are quite different words with very different meanings, you believe there should be a separate letter 'c' for English and French to enable us to tell those words apart?

Re: UTF-8 Everywhere

#249

Post by [deleted] »

ghettoimp wrote: post #188
qiqitori wrote: post #22

Earlier quoted context omitted.

Are you saying that operating systems (i.e. the kernel) should check and enforce encodings in filenames? 1) Why? 2) Bye bye backward compatibility and interoperability

Backward compatibility is a laudable goal and is not to be broken lightly. But sometimes, things are so fundamentally broken that we would be far better off with a clean break. Interoperability is quite possibly a good argument for coming up with some reasonable restrictions on filenames. Today you could easily (case sensitive names, special characters, etc.) create a ZIP file or similar that cannot be successfully e…

[deleted]

Re: UTF-8 Everywhere

#250

Post by glandium »

klodolph wrote: post #111
shpx wrote: post #93

What I never see mentioned about Unicode is Han Unification https://en.m.wikipedia.org/wiki/Han_unification As I understand it, it's impossible to have a txt file that uses Japanese and Chinese characters at the same time. The file will either use the Chinese or Japanese forms of the characters, depending on your font. I would think this is a big gotcha people must run into all the time, but I never hear anyone talk…

I’m not going to try and minimize the problem, here. Han unification was pushed through by western interests, by my understanding. However, most Unicode characters are identical or nearly identical in Chinese and Japanese. Characters with “significant” visual differences got encoded as different Unicode characters. The same thing applies to simplified and traditional Chinese characters. So for a given “Han character”…

Here's some text I could write about some Japanese characters, that, thanks to Han Unification, may be confusing:

In 1946, the Japanese government created a (non-exhaustive) list of common characters, some of which were simplified from their more traditional form. One of them is 臭. Its older form was 臭. Another character that shares the same root, 嗅, was not part of that list of common characters. It was added later, in 2010, and was never simplified, such that the stroke that was removed in 臭 is still there, making it just slightly different.

If your fonts are biased towards Chinese, 臭 and 臭 will be identical, and you won't know what I'm talking about. The former is 自 above 大, the latter is 自 above 犬.

You could think the difference is trivial, but 大 is big and 犬 is dog. Not that it alters the meaning of 臭, 臭, or 嗅, but when talking about how 嗅 is not 口 alongside 臭 anymore, it does make a difference.

Post reply on HN