Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don't need a string type (2013)

mortoray.com

Post reply on HN
Print view View original

51–60 of 70 posts

Re: We don't need a string type (2013)

#51

Post by BlueTemplar »

TL;DR : Characters and Strings considered harmful.

And he's right, they totally are ! (Also, 'string' can mean an ordered sequence of similar objects of any kind, not just characters.)

But (as these discussions also mention) replacing them by much more clearly defined concepts like byte arrays, codepoints, glyphs, grapheme clusters and text fields is only the first step...

The big question (these days) is what to do with text, specifically the 'code' kind of text (either programming or markup, and poor separation between 'plain' text and code keeps causing security issues).

To start with, even code needs formatting, specifically some way to signal a new line, or it will end up unreadable.

Then, code can't be just arbitrary Unicode text, some limits have to apply, because Unicode can get verrrry 'fancy' ! (Arbitrary Unicode is fine in text fields and comments embedded in code.)

So, I'm curious, is there any Unicode normalization specifically designed for code ? (If not, why, and which is the closest one ?)

I'm thinking of Python (3), which has what seems to be a somewhat arbitrary list of what can and what can't be used as a variable name ? (And the language itself seemingly only uses ASCII, though this shouldn't be a restriction for programming/markup languages !)

Also I hear that Julia goes much further than that (with even (La)TeX-like shortcuts for characters that might not be available on some keyboards), what kind of 'normalization' have they adopted ?

Re: We don't need a string type (2013)

#52

Post by Koshkin »

throwaway894345 wrote: post #35
KMag wrote: post #15

Earlier quoted context omitted.

Filesystem paths are not strings. Linux doesn't enforce an encoding. Windows at least didn't used to enforce proper use of conjugate UTF-16 pairs (see WTF-8 encoding). I think OS X does perform UTF-8 normalization, which might include sanity checking and rejecting malformed UTF-8, but I'm not sure. A byte array (or a ref-counted singly-linked list of immutable byte arrays to save space/copying) is a much better repre…

A string is a byte array for all intents and purposes. In Go specifically, it’s an immutable byte slice with some built-in operator overloading, some of which is sugar for dealing with utf-8, but there’s nothing that suggests a string must be encoded any particular way.

> A string is a byte array for all intents and purposes.

This smacks of reductionism. String as an abstract type only needs to conform to a number of certain axioms and support certain operations. (Thus, for example, a text editor, where a string can be mutable, could choose a representation of this type that is different from a simple byte array.)

Re: We don't need a string type (2013)

#53

Post by BlueTemplar »

Anyone else thinks that we missed an opportunity to make text much simpler to deal with by not increasing the size of a byte from 8 to 32 bits when we moved from 32-bit to 64-bit word length CPUs ?

I mean, isn't the 7-bit ASCII text the reason why the byte length was standardized to the next power of two bits ?

(With e-mail still supporting non-padded 7-bit ASCII until recently for performance reasons.)

Re: We don't need a string type (2013)

#54

Post by lisper »

Koshkin wrote: post #49
lisper wrote: post #34

Earlier quoted context omitted.

I fully endorse the general idea here, but this: > `someText.firstCharacter()` would have a return type of `text`, with logical length 1 is a huge mistake. There are operations that make sense on characters that do not make sense on texts whose length happens to be 1. The most obvious of these is inquiring about the numerical value of the unicode code point of a character. Conflating characters and texts-of-length-1…

Functors are everywhere. That's why we need monads!

Gnats and sledgehammers something something...

Re: We don't need a string type (2013)

#55

Post by bobthepanda »

BlueTemplar wrote: post #46
bobthepanda wrote: post #13

Earlier quoted context omitted.

I would hazard that very few people think about what an underlying String is at all. String encoding is something I encountered as a problem in college, but is up there with implementing a homemade red-black tree in terms of “things that are asked in interviews but have little to no bearing on my day-to-day.”

Really, they don't run into string/character issues regularly ? Because I do...

I certainly run into them rarely, and if I do have an issue it is usually solved by bunging it into some purpose built standard or third party library and calling it a day.

I’m sure people have jobs that deal with this, but the low-level form of the problem is not something that I could see one encountering in a meaningful way for building a standard CRUD app or service.

Re: We don't need a string type (2013)

#57

Post by eigenspace »

BlueTemplar wrote: post #51

TL;DR : Characters and Strings considered harmful. And he's right, they totally are ! (Also, 'string' can mean an ordered sequence of similar objects of any kind, not just characters.) But (as these discussions also mention) replacing them by much more clearly defined concepts like byte arrays, codepoints, glyphs, grapheme clusters and text fields is only the first step... The big question (these days) is what to do…

Yes, Julia really lets one get wild with Unicode. There are certain classes of unicode characters that we have marked as invalid for identifiers, some which are used for infix operators, and some which count as modifiers on previously typed characters which is useful for creating new infix operators, e.g. one might define

    julia> +²(x, y) = x^2 + y^2
    +² (generic function with 1 method)
such that

    julia> -2 +² 3
    13
If someone doesn't know how to type this, they can just hit the `?` button to open help mode in the repl and then paste it:

    help?> +²
    "+²" can be typed by +\^2

    search: +²

      No documentation found.

      +² is a Function.

      # 1 method for generic function "+²":
      [1] +²(x, y) in Main at REPL[65]:1
Note how it says

    "+²" can be typed by +\^2
Generally speaking we don't have a ton of strict rules on unicode, but it's a community convention that if you have a public facing API that uses unicode, you should provide an alternative unicode-free API. This works pretty well for us, and I think can be quite useful for some mathematical code if you don't overdo it (the above example was not an example of 'responsible' use).

I know we have a code formatter, but it doesn't do any unicode normalization. We generally just accept unicode as a first class citizen in code. This tends to cause some programmers to 'clutch their pearls' and act horrified, but in practice it works well. Maybe just because we have a cohesive community though

Re: We don't need a string type (2013)

#58

Post by BlueTemplar »

giardini wrote: post #56

Surprising to a Tcl programmer!8-)) b/c "Everything is a String": https://wiki.tcl-lang.org/page/everything+is+a+string and "Everything is a Symbol": https://wiki.tcl-lang.org/page/Everything+is+a+Symbol

Looks like that what Tcl means by 'string', the author names 'text' ?

What does Tcl mean by 'character' ?

See for instance, the author's HTML example :

> Combining characters can create an accented version of that symbol, https://mortoray.com/2014/03/17/strings-and-text-are-not-the...

EDIT: Ok, it looks like by 'character', Tcl means what the author (and Unicode ?) calls a 'grapheme cluster' ?

https://wiki.tcl-lang.org/page/Characters%2C+glyphs%2C+code%...

https://mortoray.com/2016/04/28/what-is-the-length-of-a-stri...

Re: We don't need a string type (2013)

#59

Post by throwaway894345 »

Koshkin wrote: post #52
throwaway894345 wrote: post #35

Earlier quoted context omitted.

A string is a byte array for all intents and purposes. In Go specifically, it’s an immutable byte slice with some built-in operator overloading, some of which is sugar for dealing with utf-8, but there’s nothing that suggests a string must be encoded any particular way.

> A string is a byte array for all intents and purposes. This smacks of reductionism. String as an abstract type only needs to conform to a number of certain axioms and support certain operations. (Thus, for example, a text editor, where a string can be mutable, could choose a representation of this type that is different from a simple byte array.)

Based on the context of the thread, the definition of "string" used in this thread must also include the properties possessed by Go strings in order for the original criticism to be coherent. It seems more likely (and charitable) that the criticism is incorrect rather than incoherent.

In whatever case, Go strings have all of the relevant properties for modeling file paths.

Re: We don't need a string type (2013)

#60

Post by BlueTemplar »

eigenspace wrote: post #57
BlueTemplar wrote: post #51

TL;DR : Characters and Strings considered harmful. And he's right, they totally are ! (Also, 'string' can mean an ordered sequence of similar objects of any kind, not just characters.) But (as these discussions also mention) replacing them by much more clearly defined concepts like byte arrays, codepoints, glyphs, grapheme clusters and text fields is only the first step... The big question (these days) is what to do…

Yes, Julia really lets one get wild with Unicode. There are certain classes of unicode characters that we have marked as invalid for identifiers, some which are used for infix operators, and some which count as modifiers on previously typed characters which is useful for creating new infix operators, e.g. one might define julia> +²(x, y) = x^2 + y^2 +² (generic function with 1 method) such that julia> -2 +² 3 13 If s…

Nice ! Python allows to define operators too, but AFAIK you can't use Unicode in those ? And ² (or any other sub/superscript number - at least some letters are fine) is not allowed in identifiers either.

The point is to get closer to math notation though, if anything x +² y is IMHO even farther away than (x + y)*2 !

Any way to have (x + y)² or √(x + y) to work ?

––––

The new AZERTY has a lot of improvements : ∞, ±, ≠, √, the whole Greek alphabet, () and [] and {} next to each other... but for some reason they've removed the ² that the old AZERTY had ?

http://norme-azerty.fr/

Post reply on HN