Live data from Hacker News

Swift UTF-8 String

swift.org

71–80 of 97 posts

Re: Swift UTF-8 String

#71
post #53

Earlier quoted context omitted.

So in one and half months they haven't found time nor a volunteer willing to do another podcast? My favourite podcasts have weekly and bi-weekly updates.

Yeah, he can't even release a podcast episode whenever you want him to. Does he not know he won't make it to your list of favorite podcasts? Why does he even wake up? Lattner is such a loser.

If he cares about actually grooming a community outside Apple platforms, maybe.

Re: Swift UTF-8 String

#72

If I ever wrote a language, handling strings wrt unicode would be the first thing I would fix so so that it's clear, concise and especially consistent. It bothers me that in 2019 it can still be difficult to manage even basic things with strings, and yet, whenever we talk/argue/bike-shed about 'languages' we get caught up in intellectual meandering. Please, give us strings that make sense. Then argue about monads.

In what language do you want them to make sense? A Hindi string would be very different to an Arabic strig, or a japanese string, or an emoji string.

Re: Swift UTF-8 String

#73
UTF-16 allows to quickly index a char in a string covering the most languages that way.

UTF-8 is a different story, as nearly every byte of UTF-8 string is a surrogate for most languages except English/ASCII.

Now, how can a char be indexed then? Only by slow iteration from the beginning of a string?

UTF-8 is a good choice for storage, true. However, it seems to be a not so good choice for string processing from the algorithmic point of view.

Or maybe there is a catch that makes UTF-8 strings well-suited for string processing as well?

Re: Swift UTF-8 String

#74

UTF-16 allows to quickly index a char in a string covering the most languages that way. UTF-8 is a different story, as nearly every byte of UTF-8 string is a surrogate for most languages except English/ASCII. Now, how can a char be indexed then? Only by slow iteration from the beginning of a string? UTF-8 is a good choice for storage, true. However, it seems to be a not so good choice for string processing from the a…

> UTF-16 allows to quickly index a char in a string covering the most languages that way.

Nope. UTF-16 is not UCS-2 anymore for multiple reasons. You cannot assume you can index a UTF-16 string as if it was UCS-2 anymore. UCS-4 (32-bit) is a possible option, though not generally recommended either.

There are multiple tricks for getting great performance in UTF-8 string processing such as codepoint maps of various sorts. It's quite well suited for string processing, and it's better to use UTF-8 at this point simply because it also helps from falling into the UTF-16 is not (and has not been for some time) UCS-2 trap, and UCS-2 cannot represent a lot of Unicode today (including many emoji).

Re: Swift UTF-8 String

#75

I have never written a single line of iOS/macos code, but I'm interested in Swift because it's a cool language. Can anyone explain why this is done now and not when Swift was first released? I mean, UTF-8 was already the clear winner when Swift started. Is it some obj-c compat story?

The commonly-accepted wisdom among Unicode people was that UTF-8 is good for transmission and storage, but bad for in memory representation. It now appears pretty conclusive that this was wrong.

Re: Swift UTF-8 String

#76

UTF-16 allows to quickly index a char in a string covering the most languages that way. UTF-8 is a different story, as nearly every byte of UTF-8 string is a surrogate for most languages except English/ASCII. Now, how can a char be indexed then? Only by slow iteration from the beginning of a string? UTF-8 is a good choice for storage, true. However, it seems to be a not so good choice for string processing from the a…

> UTF-16 allows to quickly index a char in a string covering the most languages that way. Nope. UTF-16 is not UCS-2 anymore for multiple reasons. You cannot assume you can index a UTF-16 string as if it was UCS-2 anymore. UCS-4 (32-bit) is a possible option, though not generally recommended either. There are multiple tricks for getting great performance in UTF-8 string processing such as codepoint maps of various sor…

Never made a claim that UTF-16 is UCS-2. "Most languages" does not mean "all languages".

Still, many popular platforms treat it like that in terms of char indexing. They are able to get away with it since a surrogate char is a rare guest in UTF-16.

P.S. It's surprising how easily HN crowd downvotes something when it falls out of a whimsical "popular contemporary view on things". Having a lot of years of text processing behind my shoulders, I raised an important question and got an immediate down-vote attack. But never mind, I'll survive.

Re: Swift UTF-8 String

#77
post #75

I have never written a single line of iOS/macos code, but I'm interested in Swift because it's a cool language. Can anyone explain why this is done now and not when Swift was first released? I mean, UTF-8 was already the clear winner when Swift started. Is it some obj-c compat story?

The commonly-accepted wisdom among Unicode people was that UTF-8 is good for transmission and storage, but bad for in memory representation. It now appears pretty conclusive that this was wrong.

It's not just now — the memory-access argument has been known to be invalid since 1996 when Unicode 2.0 added surrogate pairs and broke the ability to use C-style indexing without having to decode the entire sequence. UTF-16 added overhead versus UTF-8 but it doesn't make the implementation any more efficient unless you don't care about correctness.

Swift was created two decades later so the only justification which seems to make sense would be compatibility with Objective C and UTF-16 APIs.

Re: Swift UTF-8 String

#78
post #29

Earlier quoted context omitted.

It looks like Swift has automatic currying though, so that's pretty close.

It doesn't have automatic currying. What code/docs are you getting that impression from?

Swift used to have a special syntax for defining automatically curried functions, but it was removed in Swift 3: https://github.com/apple/swift-evolution/blob/master/proposa...

Re: Swift UTF-8 String

#79

Earlier quoted context omitted.

> UTF-16 allows to quickly index a char in a string covering the most languages that way. Nope. UTF-16 is not UCS-2 anymore for multiple reasons. You cannot assume you can index a UTF-16 string as if it was UCS-2 anymore. UCS-4 (32-bit) is a possible option, though not generally recommended either. There are multiple tricks for getting great performance in UTF-8 string processing such as codepoint maps of various sor…

Never made a claim that UTF-16 is UCS-2. "Most languages" does not mean "all languages". Still, many popular platforms treat it like that in terms of char indexing. They are able to get away with it since a surrogate char is a rare guest in UTF-16. P.S. It's surprising how easily HN crowd downvotes something when it falls out of a whimsical "popular contemporary view on things". Having a lot of years of text processi…

All those languages make it quite clear that you only get access to a code unit that way and other APIs usually exist to actually get access to code points, which then usually take a string and an index.

That the most visible mechanism of accessing parts of a string is based on code units along with UTF-16 masking bad code unless Emoji are involved (for most developers at least), is a problem, though. UTF-8 makes the failure case pop up much faster as pretty much every language doesn't only use ASCII.

Re: Swift UTF-8 String

#80

Earlier quoted context omitted.

We've been eye-ing it for a while at remind.com. Our hangup has been that we're heavily aws focused, but https://github.com/amzn/smoke-aws is looking promising. We're interested because it's one of a handful of statically typed functional programming languages. Haskell/Ocaml are great as ML type languages, but it's a big paradigm shift for a team to jump into. Rust still has a pretty decent learning curve. Swift seem…

I don't really consider a language to be functional without a concise application and composition operator (also automatic currying and partial application). How can you be functional if you don't have the basic operators for combining and applying functions?

Automatic currying and partial application are really just nice to haves though. You can manually do both as long as the language supports higher order functions.
Post reply on HN