The string type is broken (2013) • Hacker News Board

Re: The string type is broken (2013)

#31

Post by mortoray » Wed, Jun 23, 2021, 5:46 AM UTC

ridiculous_fish wrote: post #22

jph wrote: post #19

Rust 1.53 seems to handle these correctly - feedback welcome. let s = String::from("noël"); println!("Printable? {}", s); println!("Countable? {}", s.chars().count()); println!("Reversable? {}", s.chars().rev().collect:: ()); println!("First three characters? {}", s.chars().take(3).collect:: ()); Output: Printable? noël Countable? 4 Reversable? lëon First three characters? noë Repo for the "noël" example and "cats" e…

It does not. Your code is using precomposed instead of decomposed chars. Try: let s = String::from("noe\u{0308}l"); println!("Reversable? {}", s.chars().rev().collect:: ()); println!( "First three characters? {}", s.chars().take(3).collect:: () ); you get l̈eon which (according to the article) is wrong; likewise "noe" as the first three chars, dropping the diacritic.

It's a bit disappointing that a relatively new language like Rust doesn't handle these correctly.

There's no value in a String type if it doesn't behave for text. One can simply use a "Array" to convey proper semantic meaning.

Re: The string type is broken (2013)

#32

Post by astrange » Wed, Jun 23, 2021, 6:02 AM UTC

adonovan wrote: post #9

remexre wrote: post #3

Any Swift developers who know how it fares on these things? I've heard it does much better, largely by virtue of the Character type being an extended grapheme cluster, but does it handle all these cases right?

I recently did a little survey of string types across programming languages, and found that they display an immense variety of designs for a datatype so fundamental. Swift stakes out one of the most interesting points in the design space by completely encapsulating the internal representation and requiring all accessors to be explicit in whether they operate on the string's encoding (e.g. as UTF-8), its Unicode code…

> Swift stakes out one of the most interesting points in the design space by completely encapsulating the internal representation and requiring all accessors to be explicit in whether they operate on the string's encoding (e.g. as UTF-8), its Unicode code points, or its grapheme clusters. All positions within the string are represented using abstract iterators that don't reveal their numeric quantities.

This is probably the only right way because 1. it admits multiple implementations (like packing into tagged pointers) and 2. a string isn't simply an array of anything. Strings are made of all of bytes/codepoints/grapheme clusters/words/lines/etc at different levels.

Re: The string type is broken (2013)

#33

Post by drpixie » Wed, Jun 23, 2021, 6:09 AM UTC

Mortoray, I'm inclined to agree with you. Modern strings/unicode has completely confused quite different things - storage (bytes), glyphs, typesetting, and language. So we're mixing single characters, with multiple characters, with character construction, with non-character emojis.

We are almost getting away with this confusion by using brute force - "check out the CPU power of my phone". We're probably struck with it for interweb purposes, but I'd like to see a nice clean alternative, or a cleaned-up version of unicode ... something we could use in small embedded/IoT scale systems.

Re: The string type is broken (2013)

#34

Post by leephillips » Wed, Jun 23, 2021, 7:16 AM UTC

By the way, Elixir passes all the tests.

Re: The string type is broken (2013)

#35

Post by [deleted] » Wed, Jun 23, 2021, 8:31 AM UTC

[deleted]

Re: The string type is broken (2013)

#36

Post by LinAGKar » Wed, Jun 23, 2021, 9:14 AM UTC

ridiculous_fish wrote: post #22

jph wrote: post #19

Rust 1.53 seems to handle these correctly - feedback welcome. let s = String::from("noël"); println!("Printable? {}", s); println!("Countable? {}", s.chars().count()); println!("Reversable? {}", s.chars().rev().collect:: ()); println!("First three characters? {}", s.chars().take(3).collect:: ()); Output: Printable? noël Countable? 4 Reversable? lëon First three characters? noë Repo for the "noël" example and "cats" e…

It does not. Your code is using precomposed instead of decomposed chars. Try: let s = String::from("noe\u{0308}l"); println!("Reversable? {}", s.chars().rev().collect:: ()); println!( "First three characters? {}", s.chars().take(3).collect:: () ); you get l̈eon which (according to the article) is wrong; likewise "noe" as the first three chars, dropping the diacritic.

Although it's not in the standard library, this can be handled properly using the unicode-segmentation crate: https://crates.io/crates/unicode-segmentation

Re: The string type is broken (2013)

#37

Post by moring » Wed, Jun 23, 2021, 11:36 AM UTC

The whole problem seems to start with a clear definition of what a string is. Java (IIRC) defines it as a list of code points, and its behavior follows from that. It's even "correct" in the sense of correct according to that definition.

So what would the right definition of a string be that implies everything the author would consider "correct" behaviour?

I've actually run into the same question before and so far my conclusion wasn't even that the implementation of strings is wrong on most platforms, but rather that "string" as a concept just doesn't make sense. Including simple things like the length of a string or concatenation.

If somebody knows a useful definition, I'd be interested ;)

Re: The string type is broken (2013)

#38

Post by elcaro » Wed, Jun 23, 2021, 12:46 PM UTC

adonovan wrote: post #9

remexre wrote: post #3

Any Swift developers who know how it fares on these things? I've heard it does much better, largely by virtue of the Character type being an extended grapheme cluster, but does it handle all these cases right?

I recently did a little survey of string types across programming languages, and found that they display an immense variety of designs for a datatype so fundamental. Swift stakes out one of the most interesting points in the design space by completely encapsulating the internal representation and requiring all accessors to be explicit in whether they operate on the string's encoding (e.g. as UTF-8), its Unicode code…

Raku also has a specific string type (Str) that is not just a bag of bytes. Strings are normalized (NFC) when they are created, and multi-codepoint graphemes are represented internally using synthetic codepoints (negative integers), but this is all opaque to the user. This also means that string indexing in Raku is O(1).

Raku passes all the tests. He mentions he didn't find any language that upper-cases "baﬄe" correctly but Raku does.

Here's my REPL test

    > my $n = "noe\x[0308]l"
    noël
    > $n.chars
    4
    > $n.flip
    lëon
    > $n.substr(0, 3)
    noë
    > my $b = "baﬄe"
    baﬄe
    > $b.substr(2, 1).uniname
    LATIN SMALL LIGATURE FFL
    > $b.uc
    BAFFLE
    > "noe\x[0308]l" eq "no\x[00EB]l"
    True

HN doesn't handle the cat emojis, so they are not included, but they worked fine.

Re: The string type is broken (2013)

#39

Post by qw3rty01 » Wed, Jun 23, 2021, 1:28 PM UTC

brundolf wrote: post #18

qw3rty01 wrote: post #16

Earlier quoted context omitted.

this isn't quite set up the way the article was mentioning. Here's with it set up correctly: https://play.rust-lang.org/?version=stable&mode=debug&editio... It looks like it still misses some of the cases (specifically with decomposed vs precomposed). As a side note, it looks like the playground editor also doesn't correctly render some of the lengths (the cursor is 1 character off on the decomposed case)

Huh, I'd never heard of decomposed/precomposed before and I kind of skimmed over them in the original article. Looking at your example, I still can't figure out what the difference is between the two (I guess the Playground editor itself just can't represent the difference?). I'll have to go read up on that concept. Edit: Having read about them, I guess I assumed that if my keyboard inputs an ë it's actually inputtin…

After a little bit of research, it looks like this is how you would get the expected outputs: https://play.rust-lang.org/?version=stable&mode=debug&editio...

Re: The string type is broken (2013)

#40

Post by kibibu » Wed, Jun 23, 2021, 1:46 PM UTC

moring wrote: post #37

The whole problem seems to start with a clear definition of what a string is . Java (IIRC) defines it as a list of code points, and its behavior follows from that. It's even "correct" in the sense of correct according to that definition. So what would the right definition of a string be that implies everything the author would consider "correct" behaviour? I've actually run into the same question before and so far my…

The author of the article appears to think "a sequence of graphemes" is the right abstraction.