Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The string type is broken (2013)

mortoray.com

Post reply on HN
Print view View original

41–45 of 45 posts

Re: The string type is broken (2013)

#41

Post by paulmd »

marcosdumay wrote: post #26
paulmd wrote: post #21

Earlier quoted context omitted.

rev | cut | rev is a pretty common idiom in the shell when you want to keep "all but the last N chunks" from some lines of input. At least for me. I'm sure there's probably a better way, and performance of this is definitely notably poor. And of course it's not like I'm actually writing the utility. ;)

I would be very surprised if rev | cut | rev gave you the correct answer to anything that isn't "drop the last X lines or words" on a non western alphabet. And for "drop the last X lines or words", you would do better with some specialized command.

you don't ever work with directories? fortunately there is a pretty defined separator in my directory hierarchies, so that is pretty much always correct ;)

(although if you are taking arbitrary user data I suppose there is nothing stopping you doing something stupid like putting an escaped '/' in a filename, which would break this.)

Re: The string type is broken (2013)

#42

Post by Wevah »

Wevah wrote: post #11
remexre wrote: post #3

Any Swift developers who know how it fares on these things? I've heard it does much better, largely by virtue of the Character type being an extended grapheme cluster, but does it handle all these cases right?

All of the things I know the answers to off the top of my head it passes, but I’d have to check a few of the others (e.g. the “ffl” ligature) I’d have to check.

(Addendum: "baffle".uppercased() does correctly return "BAFFLE".)

Re: The string type is broken (2013)

#43

Post by jph »

ridiculous_fish wrote: post #28
jph wrote: post #23

Earlier quoted context omitted.

Can you explain more? I updated my post with more info and a link to a code repo. The code correctly shows me the reverse "lëon" with the umlaut, and the first three characters "noë" with the umlaut.

ë may be represented in two ways: 1. One code point: U+00EB. This is the "precomposed" form. 2. Two code points: U+0065 U+0308, aka e followed by ¨. This is the "decomposed" form, also known as a "combining sequence" since the diaeresis combines with the base character. If your string type is a sequence of code points, then reversing a decomposed string will tear the combining sequence, and apply the diaeresis to the…

Much obliged, thank you. I'm adding info about this to the repo, and also adding precomposed characters versus decomposed characters. Now I do see the problem you're describing and the article author is describing.

Re: The string type is broken (2013)

#44

Post by adonovan »

astrange wrote: post #32
adonovan wrote: post #9

Earlier quoted context omitted.

I recently did a little survey of string types across programming languages, and found that they display an immense variety of designs for a datatype so fundamental. Swift stakes out one of the most interesting points in the design space by completely encapsulating the internal representation and requiring all accessors to be explicit in whether they operate on the string's encoding (e.g. as UTF-8), its Unicode code…

> Swift stakes out one of the most interesting points in the design space by completely encapsulating the internal representation and requiring all accessors to be explicit in whether they operate on the string's encoding (e.g. as UTF-8), its Unicode code points, or its grapheme clusters. All positions within the string are represented using abstract iterators that don't reveal their numeric quantities. This is proba…

If my survey revealed anything, it's that there is no one right way.

Re: The string type is broken (2013)

#45

Post by b2gills »

elcaro wrote: post #38
adonovan wrote: post #9

Earlier quoted context omitted.

I recently did a little survey of string types across programming languages, and found that they display an immense variety of designs for a datatype so fundamental. Swift stakes out one of the most interesting points in the design space by completely encapsulating the internal representation and requiring all accessors to be explicit in whether they operate on the string's encoding (e.g. as UTF-8), its Unicode code…

Raku also has a specific string type (Str) that is not just a bag of bytes. Strings are normalized (NFC) when they are created, and multi-codepoint graphemes are represented internally using synthetic codepoints (negative integers), but this is all opaque to the user. This also means that string indexing in Raku is O(1). Raku passes all the tests. He mentions he didn't find any language that upper-cases "baffle" correc…

Technically it is MoarVM that handles graphemes as synthetic codepoints.

We call it NFG for Normalized Form Grapheme.

The JVM and JavaScript backends currently use the built-in string handling features. Which means that they are just as broken as the rest of the code running on them.

---

Of further note, you can use ignorecase in regexes to match "baffle".

    "baffle" ~~ /:ignorecase  BAFfLE  /;
Post reply on HN