Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The string type is broken (2013)

mortoray.com

Post reply on HN
Print view View original

21–30 of 45 posts

Re: The string type is broken (2013)

#21

Post by paulmd »

thrower123 wrote: post #20

Reversing strings is such an odd thing to focus on. I have written dozens of implementations of string reversers for school and interviews and following books, but I would be hard pressed to come up with any real case where I needed to reverse a string. And then, it was because I was abusing a string to store some data that was not really text, at which point I really didn't care that it textually supported reversing…

rev | cut | rev

is a pretty common idiom in the shell when you want to keep "all but the last N chunks" from some lines of input.

At least for me. I'm sure there's probably a better way, and performance of this is definitely notably poor.

And of course it's not like I'm actually writing the utility. ;)

Re: The string type is broken (2013)

#22

Post by ridiculous_fish »

jph wrote: post #19

Rust 1.53 seems to handle these correctly - feedback welcome. let s = String::from("noël"); println!("Printable? {}", s); println!("Countable? {}", s.chars().count()); println!("Reversable? {}", s.chars().rev().collect:: ()); println!("First three characters? {}", s.chars().take(3).collect:: ()); Output: Printable? noël Countable? 4 Reversable? lëon First three characters? noë Repo for the "noël" example and "cats" e…

It does not. Your code is using precomposed instead of decomposed chars.

Try:

    let s = String::from("noe\u{0308}l");
    println!("Reversable? {}", s.chars().rev().collect::());
    println!(
        "First three characters? {}",
        s.chars().take(3).collect::()
    );
you get l̈eon which (according to the article) is wrong; likewise "noe" as the first three chars, dropping the diacritic.

Re: The string type is broken (2013)

#23

Post by jph »

ridiculous_fish wrote: post #22
jph wrote: post #19

Rust 1.53 seems to handle these correctly - feedback welcome. let s = String::from("noël"); println!("Printable? {}", s); println!("Countable? {}", s.chars().count()); println!("Reversable? {}", s.chars().rev().collect:: ()); println!("First three characters? {}", s.chars().take(3).collect:: ()); Output: Printable? noël Countable? 4 Reversable? lëon First three characters? noë Repo for the "noël" example and "cats" e…

It does not. Your code is using precomposed instead of decomposed chars. Try: let s = String::from("noe\u{0308}l"); println!("Reversable? {}", s.chars().rev().collect:: ()); println!( "First three characters? {}", s.chars().take(3).collect:: () ); you get l̈eon which (according to the article) is wrong; likewise "noe" as the first three chars, dropping the diacritic.

Can you explain more?

I updated my post with more info and a link to a code repo.

The code correctly shows me the reverse "lëon" with the umlaut, and the first three characters "noë" with the umlaut.

Re: The string type is broken (2013)

#24

Post by pjgalbraith »

natanbc wrote: post #13
brundolf wrote: post #4

Rust's string type doesn't have the mentioned problems: https://play.rust-lang.org/?version=stable&mode=debug&editio... Though I would bet this mainly has to do with being a language born in the age of the web. It seems like something that would be very hard to back-port (as opposed to designing for it from the beginning), but proper handling of Unicode is practically table-stakes for a new language in today's world.…

It does for multi-codepoint characters (such as the rainbow flag): https://play.rust-lang.org/?version=stable&mode=debug&editio... Elixir gets it right: https://replit.com/@natanbc/reverse

Here is a working rust version https://play.rust-lang.org/?version=stable&mode=debug&editio...

Re: The string type is broken (2013)

#25

Post by marcosdumay »

thrower123 wrote: post #20

Reversing strings is such an odd thing to focus on. I have written dozens of implementations of string reversers for school and interviews and following books, but I would be hard pressed to come up with any real case where I needed to reverse a string. And then, it was because I was abusing a string to store some data that was not really text, at which point I really didn't care that it textually supported reversing…

What is even the meaning of reversing an string?

It's more meaningless the more a naive algorithm is broken... But shouldn't it at least use those reverse letters Unicode has somewhere? Or is it just a matter of putting the right to left symbol at its start?

Re: The string type is broken (2013)

#26

Post by marcosdumay »

paulmd wrote: post #21
thrower123 wrote: post #20

Reversing strings is such an odd thing to focus on. I have written dozens of implementations of string reversers for school and interviews and following books, but I would be hard pressed to come up with any real case where I needed to reverse a string. And then, it was because I was abusing a string to store some data that was not really text, at which point I really didn't care that it textually supported reversing…

rev | cut | rev is a pretty common idiom in the shell when you want to keep "all but the last N chunks" from some lines of input. At least for me. I'm sure there's probably a better way, and performance of this is definitely notably poor. And of course it's not like I'm actually writing the utility. ;)

I would be very surprised if rev | cut | rev gave you the correct answer to anything that isn't "drop the last X lines or words" on a non western alphabet.

And for "drop the last X lines or words", you would do better with some specialized command.

Re: The string type is broken (2013)

#27

Post by mannerheim »

adonovan wrote: post #9
remexre wrote: post #3

Any Swift developers who know how it fares on these things? I've heard it does much better, largely by virtue of the Character type being an extended grapheme cluster, but does it handle all these cases right?

I recently did a little survey of string types across programming languages, and found that they display an immense variety of designs for a datatype so fundamental. Swift stakes out one of the most interesting points in the design space by completely encapsulating the internal representation and requiring all accessors to be explicit in whether they operate on the string's encoding (e.g. as UTF-8), its Unicode code…

Haskell just uses a list of Unicode code points, i.e., type String = [Char]. Not a good choice, but not a particularly complicated one either.

What does make it a bit complicated is that there are about four other different string types to choose between, besides the default String type, which should never be used for anything performance-sensitive.

Re: The string type is broken (2013)

#28

Post by ridiculous_fish »

jph wrote: post #23
ridiculous_fish wrote: post #22

Earlier quoted context omitted.

It does not. Your code is using precomposed instead of decomposed chars. Try: let s = String::from("noe\u{0308}l"); println!("Reversable? {}", s.chars().rev().collect:: ()); println!( "First three characters? {}", s.chars().take(3).collect:: () ); you get l̈eon which (according to the article) is wrong; likewise "noe" as the first three chars, dropping the diacritic.

Can you explain more? I updated my post with more info and a link to a code repo. The code correctly shows me the reverse "lëon" with the umlaut, and the first three characters "noë" with the umlaut.

ë may be represented in two ways:

1. One code point: U+00EB. This is the "precomposed" form.

2. Two code points: U+0065 U+0308, aka e followed by ¨. This is the "decomposed" form, also known as a "combining sequence" since the diaeresis combines with the base character.

If your string type is a sequence of code points, then reversing a decomposed string will tear the combining sequence, and apply the diaeresis to the wrong character. Most string types are affected by this (or worse), Rust included.

The two forms get rendered identically, so you more or less need a hex editor to figure out which form you've got. I forked your repo and switched it to decomposed (the diff looks like a noop), and now it produces the wrong output:

https://github.com/ridiculousfish/demo-rust-string-issues

Another Rust example using explicit Unicode literals: https://play.rust-lang.org/?version=stable&mode=debug&editio...

One could reasonably conclude that precomposed forms are just better and easier. But they're considered legacy: we can't encode every possible combining sequence into a code point, so we might as well go the other way and decompose whenever possible. That's what Normalization Form D is about.

Re: The string type is broken (2013)

#29

Post by ademarre »

adonovan wrote: post #9
remexre wrote: post #3

Any Swift developers who know how it fares on these things? I've heard it does much better, largely by virtue of the Character type being an extended grapheme cluster, but does it handle all these cases right?

I recently did a little survey of string types across programming languages, and found that they display an immense variety of designs for a datatype so fundamental. Swift stakes out one of the most interesting points in the design space by completely encapsulating the internal representation and requiring all accessors to be explicit in whether they operate on the string's encoding (e.g. as UTF-8), its Unicode code…

Now I am curious to look at Swift's strings. I am a big fan of explicit encodings. I have long believed that an ideal text API would not use string as a concrete data type. Instead, encodings themselves (UTF-8 et al.) should be concrete types and string should be nothing more than an abstraction over them for polymorphism.

Re: The string type is broken (2013)

#30

Post by mortoray »

thrower123 wrote: post #20

Reversing strings is such an odd thing to focus on. I have written dozens of implementations of string reversers for school and interviews and following books, but I would be hard pressed to come up with any real case where I needed to reverse a string. And then, it was because I was abusing a string to store some data that was not really text, at which point I really didn't care that it textually supported reversing…

It's not about reversing, but about checking whether reverse iteration works. This is required to perform proper truncation (for limited length fields) as well as visual editing for a cursor to go through the text.

It can additionally be an indication as to whether regular expression matching works, though that is usually handled by a library, and not the core string types.

My contention was always that if a "string" does not reverse text, then it's no better than an "array". The existence of a "string" type implies it should do more.

Post reply on HN