Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don't need a string type (2013)

mortoray.com

Post reply on HN
Print view View original

31–40 of 70 posts

Re: We don't need a string type (2013)

#31

Post by hollasch »

Curious. I have to come to exactly the opposite conclusion — that we should drop the idea of a fixed-length character type, and instead _only_ have (Unicode) string types. Actually, I'd prefer something like `std::text` to finally be free of the baggage of "string". Operations on text should work on logical text concepts. For example, something like `someText.firstCharacter()` would have a return type of `text`, with logical length 1. It's _data_ length is variable, since a Unicode character is variable length. So many Unicode-containing string design problems arise because of the stubborn insistence of having an integral character type.

I should be able to extract UTF-8, UTF-16 or whatever encoding I want from a `text` value. Something like `c_str()` would be pretty important, but the semantics would be a design problem, not an encoding problem. Any Unicode-encoding string should be able to encode U+0000, so you'd need to figure out how to handle that from `c_str()` (perhaps a substitution ASCII character could be specified to encode embedded nulls).

Basically, users should definitely _not_ need to understand the deeper details of Unicode. They shouldn't need to understand and worry about different entities such as code units, code points, graphemes, and the like, though they should be able to extract such encodings on demand.

Re: We don't need a string type (2013)

#32

Post by shadowgovt »

hollasch wrote: post #31

Curious. I have to come to exactly the opposite conclusion — that we should drop the idea of a fixed-length character type, and instead _only_ have (Unicode) string types. Actually, I'd prefer something like `std::text` to finally be free of the baggage of "string". Operations on text should work on logical text concepts. For example, something like `someText.firstCharacter()` would have a return type of `text`, with…

Essentially, different tools for different applications.

"A string is a vector of characters, which happen to each be one byte in length" was more of an artifact of a time where there happened to be representational overlap than some deep truism about proper data structure. Strings intended to be displayed to humans are specialized constructs, much as a "button" or a "file handle" are. A buffer of unstructured bytes is a separate specialized construct, suitable for tasks unrelated to "displaying text to a human."

Re: We don't need a string type (2013)

#33

Post by throwaway894345 »

jrimbault wrote: post #6
37ef_ced3 wrote: post #5

Go's immutable UTF-8 string type is one of the nice things about the language A Go string is almost exactly like this C struct: struct String { uint8_t* addr; ptrdiff_t len; }; The language guarantees you can't modify the bytes in memory range [addr, addr+len) Go's garbage collection makes it simple and natural to have one string alias ("point into", "overlap") part of another string. This works because strings are i…

I'd arguee Go's string type is "somewhat unusable"* since it doesn't enforce the guarantees it says/implies it does. The byte slice it points to is not guaranteed to be valid utf8. * of course to a degree, let's be reasonable, it's usable in a _lot_ of contexts, but I like my types to actually mean something.

I think of a string as an immutable byte slice. This is a little confusing since the language supports utf-8 literals only and it also lets you iterate over individual runes with for loops, but those are just conveniences over the fact that these are really just immutable byte slices. You could probably make your own “UTF8” type with the invariants you want (or at least someone would have to drop down into unsafe to violate the invariants) but in general Go programs don’t typically go that far, presumably because it doesn’t add much value in practice, which would suggest that your “somewhat unusable” claim (even with its caveat) is too strong. That said, I think it would be nice if Go made it a little easier/clearer to model a type that can only be created by a particular constructor or some such.

Re: We don't need a string type (2013)

#34

Post by lisper »

hollasch wrote: post #31

Curious. I have to come to exactly the opposite conclusion — that we should drop the idea of a fixed-length character type, and instead _only_ have (Unicode) string types. Actually, I'd prefer something like `std::text` to finally be free of the baggage of "string". Operations on text should work on logical text concepts. For example, something like `someText.firstCharacter()` would have a return type of `text`, with…

I fully endorse the general idea here, but this:

> `someText.firstCharacter()` would have a return type of `text`, with logical length 1

is a huge mistake. There are operations that make sense on characters that do not make sense on texts whose length happens to be 1. The most obvious of these is inquiring about the numerical value of the unicode code point of a character. Conflating characters and texts-of-length-1 is a mistake of the same order as conflating strings and byte vectors. Python makes this mistake even in version 3. As a result, a function like this:

def f(s, n, m): return ord(s[n:m])

will return a value iff m is one more than n. Not good.

Re: We don't need a string type (2013)

#35

Post by throwaway894345 »

KMag wrote: post #15
DougBTX wrote: post #8

Earlier quoted context omitted.

Go doesn't guarantee any encoding for strings, very deliberately (so that, eg, they can be used to represent file names).

Filesystem paths are not strings. Linux doesn't enforce an encoding. Windows at least didn't used to enforce proper use of conjugate UTF-16 pairs (see WTF-8 encoding). I think OS X does perform UTF-8 normalization, which might include sanity checking and rejecting malformed UTF-8, but I'm not sure. A byte array (or a ref-counted singly-linked list of immutable byte arrays to save space/copying) is a much better repre…

A string is a byte array for all intents and purposes. In Go specifically, it’s an immutable byte slice with some built-in operator overloading, some of which is sugar for dealing with utf-8, but there’s nothing that suggests a string must be encoded any particular way.

Re: We don't need a string type (2013)

#36

Post by KMag »

jerf wrote: post #20
KMag wrote: post #15

Earlier quoted context omitted.

Filesystem paths are not strings. Linux doesn't enforce an encoding. Windows at least didn't used to enforce proper use of conjugate UTF-16 pairs (see WTF-8 encoding). I think OS X does perform UTF-8 normalization, which might include sanity checking and rejecting malformed UTF-8, but I'm not sure. A byte array (or a ref-counted singly-linked list of immutable byte arrays to save space/copying) is a much better repre…

It is not clear to me if you're elaborating or think you're disagreeing, but that is what Go does. It is generally assumed in Go that strings are UTF-8, but in practice what they actually are are just bags of bytes. Nothing really "UTF-y" will happen to them until you directly call UTF functions on them, which may produce new strings. It's something that I don't think could work unless your language is as recent as G…

I'm saying it's useful to have valid strings and paths as separate types, but Go conflates the two types. Conflating the two is likely to lead to confused usage (such as programmers assuming there's a bijective mapping between valid paths and valid sequences of Unicode codepoints.)

Pervasive confused usage of this sort in the wild in Python 2 was the motivation behind splitting bytes and strings in Python 3.

Re: We don't need a string type (2013)

#37

Post by KMag »

throwaway894345 wrote: post #35
KMag wrote: post #15

Earlier quoted context omitted.

Filesystem paths are not strings. Linux doesn't enforce an encoding. Windows at least didn't used to enforce proper use of conjugate UTF-16 pairs (see WTF-8 encoding). I think OS X does perform UTF-8 normalization, which might include sanity checking and rejecting malformed UTF-8, but I'm not sure. A byte array (or a ref-counted singly-linked list of immutable byte arrays to save space/copying) is a much better repre…

A string is a byte array for all intents and purposes. In Go specifically, it’s an immutable byte slice with some built-in operator overloading, some of which is sugar for dealing with utf-8, but there’s nothing that suggests a string must be encoded any particular way.

I'm saying that it's useful to not conflate the types for sequences of Unicode codepoints and and filesystem paths. Using the same type for both is likely to result in code with baked-in assumptions that for any path, there is a standard encoding that will yield a sequence of Unicode codepoints.

Pervasive code with this sort of type confusion in the wild in Python2 is why Python3 separated bytes and strings.

Re: We don't need a string type (2013)

#38

Post by skybrian »

jrimbault wrote: post #6
37ef_ced3 wrote: post #5

Go's immutable UTF-8 string type is one of the nice things about the language A Go string is almost exactly like this C struct: struct String { uint8_t* addr; ptrdiff_t len; }; The language guarantees you can't modify the bytes in memory range [addr, addr+len) Go's garbage collection makes it simple and natural to have one string alias ("point into", "overlap") part of another string. This works because strings are i…

I'd arguee Go's string type is "somewhat unusable"* since it doesn't enforce the guarantees it says/implies it does. The byte slice it points to is not guaranteed to be valid utf8. * of course to a degree, let's be reasonable, it's usable in a _lot_ of contexts, but I like my types to actually mean something.

Go's standard library works with both possibly-malformed and verified UTF-8 strings, which is a nice property.

The type system needed to explain what they actually do (take one of two possible input types and return the corresponding output type) would require generics, which we don't have yet.

An alternative would be to duplicate the code to account for the different types, but we already have that for []byte versus string and that's bad enough already.

Re: We don't need a string type (2013)

#39

Post by donaldihunter »

lisper wrote: post #34
hollasch wrote: post #31

Curious. I have to come to exactly the opposite conclusion — that we should drop the idea of a fixed-length character type, and instead _only_ have (Unicode) string types. Actually, I'd prefer something like `std::text` to finally be free of the baggage of "string". Operations on text should work on logical text concepts. For example, something like `someText.firstCharacter()` would have a return type of `text`, with…

I fully endorse the general idea here, but this: > `someText.firstCharacter()` would have a return type of `text`, with logical length 1 is a huge mistake. There are operations that make sense on characters that do not make sense on texts whose length happens to be 1. The most obvious of these is inquiring about the numerical value of the unicode code point of a character. Conflating characters and texts-of-length-1…

Only if you ignore the rest of what the post said. First it should make things easy for ‘normal’ tasks, then it should make everything else possible.

> Basically, users should definitely _not_ need to understand the deeper details of Unicode. They shouldn't need to understand and worry about different entities such as code units, code points, graphemes, and the like, though they should be able to extract such encodings on demand.

Re: We don't need a string type (2013)

#40

Post by throwaway894345 »

KMag wrote: post #37
throwaway894345 wrote: post #35

Earlier quoted context omitted.

A string is a byte array for all intents and purposes. In Go specifically, it’s an immutable byte slice with some built-in operator overloading, some of which is sugar for dealing with utf-8, but there’s nothing that suggests a string must be encoded any particular way.

I'm saying that it's useful to not conflate the types for sequences of Unicode codepoints and and filesystem paths. Using the same type for both is likely to result in code with baked-in assumptions that for any path, there is a standard encoding that will yield a sequence of Unicode codepoints. Pervasive code with this sort of type confusion in the wild in Python2 is why Python3 separated bytes and strings.

Maybe, but a decade of experience with Go suggests that this isn’t a significant problem (i.e., more than a handful of instances).
Post reply on HN