Earlier quoted context omitted.
I believe strings are mutable in Ruby.
They are... s = "hello" s
Is that allocating a new buffer, leaving the "hello" string to be collected by the GC?
The string type is broken
191–200 of 230 posts
Re: The string type is broken
#192Earlier quoted context omitted.
So in your one, specific, performance-limited situation, Python 3's implementation of unicode doesn't work for you. Mostly because you are trying to optimize based on implementation details. I don't see how this equates to a general purpose language failing at strings, especially when the language isn't particularly focused on performance and optimization. And if memory usage is of concern, I would certainly think an…
>I don't see how this equates to a general purpose language failing at strings And I don't see where I said it did. I used to favor a dual Python/C++ strategy, but Python's multithreading limitations and the decisions around unicode have convinced me to move on. It's not like anything has gotten worse in Python 3, it's just that there has been a major change and the opportunity to do the right thing was missed. I hap…
[deleted]
Re: The string type is broken
#193Earlier quoted context omitted.
Many languages fit into 8 bits, but English is particularly simple in its alphabet. Even many of the European languages that can fit in 8 bits have things like accented characters that complicates things somewhat. Of course this isn't to say English is simple overall. Just that it's complexities lie elsewhere, and it's simplicities lie in an area that made it particularly simple for early computer systems to process.
> Even many of the European languages that can fit in 8 bits have things like accented characters that complicates things somewhat. I don't see your point here, with respect to English orthography making computer implementation easier. How exactly does not needing representations for accented characters make anything easier?
If it was just some additional characters like ñ (which is considered a letter of its own, not an accented n) then it wouldn't be a big deal – but e and é are the same letter with different accents, which adds some subtlety that English simply doesn't have. Given a small enough number of accented characters you can punt on that, call them each a character, but English is objectively simpler since the only real distinction it has between letters is caps or not-caps. (I was just watching the Mother Of All Demos, though, and everything was in caps but they put an overline over capital letters. So even normal English lettering was too complicated for a while.)
Re: The string type is broken
#194This article is mostly written from a European language perspective. For Indian scripts, storing combining characters as a separate code points is the right thing to do. For example, कि (ki) is composed of क and ि When I'm writing this in an editor, say, I typed ku (कु) instead of ki (कि) and I press backspace, I indeed want to see क rather than deleting the whole "कि".
Only some times I figure, because if you want to make the first letter green, you'd want that to apply to the whole कि.
Re: The string type is broken
#195Earlier quoted context omitted.
> Most languages that claim Unicode support still only have UCS-2 libraries (Python 3 is a notable exception) Most non-JVM languages[1] actually use UTF-8 as the internal encoding so they should not suffer from this. Python 3 does not use UTF-16 either, it selects an encoding based on the contents of the string. http://www.python.org/dev/peps/pep-0393/ 1. I think .NET too uses UCS-2 or UTF-16, but I am not a Windows…
Python Ruby >1.8 lets you choose the encoding .NET UCS2/UTF-16 (I know the difference, imho if the stdlib has a .size, .length or .count that works on code units instead of code points it's broken... thus I'll mention only UCS2 from now on) Java UCS2 Clojure UCS2 Scala UCS2 QT UCS2 Haskell String UCS4 Haskell Data.Text UTF-16 (yes, not a naive UCS-2) Rust UCS4 (last time I checked) Javascript UCS2 Dart UCS2 PHP Unico…
Common Lisp comes with two character types, base-char and character, the former being allowed to be a subset of the latter. Clozure Common Lisp uses UTF-32 for all characters and strings internally. SBCL uses base-char and simple-base-string types for ASCII and character and (simple-array character) types for UTF-32 internally. IMO having this option for two types of characters that are compatible but may have different internal representations is a really good part of the Common Lisp standard.
Re: The string type is broken
#196Earlier quoted context omitted.
They are... s = "hello" s
Is that allocating a new buffer, leaving the "hello" string to be collected by the GC?
No, it expands the existing buffer. (leaving " world" to be collected). Note that the following is different and more like what you're thinking.
a = a + " world"Re: The string type is broken
#197Earlier quoted context omitted.
>I agree that only using UTF-8 would be the right thing, but only if you don't want to have "array of codepoints" Then we agree entirely. I want all strings to be UTF-8. Period. What I said about an array of codepoints was that I would create one seperately from the string if I ever had a requirement to access individual code point positions repeatedly in a tight loop. >the problem is: every language, and every devel…
> If by random access you mean constant time access then those developers would be very disappointed to learn that they cannot do that in Java, C#, C++, JavaScript or Python, unless they happen to know that their string cannot possibly contain any characters outside the ASCII or BMP range. Actually, you can in Python... and obviously most developers ignore such issues [citation needed] My point is that most developer…
>Actually, you can in Python
You are right (apart from combining characters as masklinn explained), but as I said, that's only possible if an array of 32 bit ints is used to hold string data or if it can be guaranteed that there are no characters from outside ASCII or BMP. If I understand PEP 393 correctly, what Python 3.3 does is to use 32 bit ints to hold the entire string if even one such code point occurs. So if you load a (possibly large) text file into a string and one such code point exists then the file's size is going to quadruple in memory. All of that is done just to implement one very rare operation efficiently. http://www.python.org/dev/peps/pep-0393/#new-api
Re: The string type is broken
#198Earlier quoted context omitted.
Agreed with most of it except: "because their string API is correct" Apparently they have a bug in their UTF-7 parser that can lead to invalid unicode strings. Don't know if it's already fixed.
That would be an implementation flaw, not an API issue.
Indeed.
Re: The string type is broken
#199˙ƃuᴉuɐǝɯ ⅋ 'spɹoʍ 'sɥdʎlƃ 'sɹǝʇɔɐɹɐɥɔ uǝǝʍʇǝq 'ɹǝʌǝʍoɥ 'ǝɔuǝɹǝɟɟᴉp ɐ sᴉ ǝɹǝɥ┴ ˙ʇxǝʇ ɥʇᴉʍ punoɹɐ ƃuᴉsɹɐ oʇ sǝɯoɔ ʇᴉ uǝɥʍ sǝᴉʇᴉlᴉqᴉssod ƃuᴉʇsǝɹǝʇuᴉ ǝɯos sɹǝɟɟo ǝpoɔᴉu∩
Which looks cleaner, "丄" or "┴" ? I.e. "ǝɹǝɥ┴" or ǝɹǝɥ丄" ?
Re: The string type is broken
#200Earlier quoted context omitted.
Unicode is a standard. It says how to act in these circumstances. Calling out incorrect unicode implementations is useful. You shouldn't have to worry about inconsistent behavior between different languages that purport to support unicode strings. That's the point of a standard.
So who is the authority about correct unicode implementations, exactly? And how to different languages with different use cases and power conform to such a standard? Why doesn't this authority extend over language implementations? Because they know what they are doing, and understand the domain, unlike the author of this article. Look, I'm all for open standards, but saying that standards are required to be adhered t…
So who is the authority about correct unicode implementations, exactly?
The Unicode Consortium[1] publishes standards. If a language advertises unicode support, I expect it to follow that standard.
Look, I'm all for open standards, but saying that standards are required to be adhered to at the programming language level is just ignorance of the real world
I'm not saying a language has to do anything, but if it's advertising support for a well defined feature, and does not deliver correctly on that, I will call them out on it, and support anyone else who does as well. Should we all just throw our hands up and say "Well, it's done now, no point in making a big deal of it?" I would rather apply pressure to get things fixed, or at least make it well known enough that future language designers give it the care and attention it's due.
There is no idea here other than the writer's unjustified expectation that he should just know how every language handles Unicode because??? Because Unicode is a standard? No..
Are you under the impression that what the author is attempting is not well defined? The unicode standard has conformance clauses about how to interpret unicode strings[2]. That means that if a language advertises it has/supports unicode strings, and fails the tests we've just seen, it's not conformant with the unicode standard. That would make this useful because it's pointing out bugs. If a language does not advertise unicode support, but supports some unicode features, then this is useful because it's making sure people are aware of the limits of their language. All too often people refer to the native string implementation in their language as supporting unicode, when clearly there are problems.
2: http://www.unicode.org/versions/Unicode6.2.0/ch03.pdf (see section 3.2)