[deleted]
Try the correct test input: noe\u0308l
The string type is broken
161–170 of 230 posts
Re: The string type is broken
#162Earlier quoted context omitted.
Uppercasing and lowercasing is inherently lossy. E.g. the German ß becomes SS when uppercased, yet there is no way to know whether SS should be lowercased to ss or ß again. That's a reason why those things should be used, if at all, only as display transformations. Same goes for ligatures, but even those actually shouldn't be applied automatically, depending on the language. E.g. in German ligatures cannot span sylla…
I feel like I should learn German only so that I would be able to comment on the ß issue every time a Unicode thread pops up. From my uninformed point of view it is not really clear if ß should really be handled as a separate character/grapheme, or just as a ligature in rendering phase and stored as 'ss'. Or even if current-day orthography should be held at such a sacrosanct position that it shouldn't be changed to s…
> or just as a ligature in rendering phase and stored as 'ss'.
Probably.
> to save significant amount of collective effort
I've seen this kind of suggestion a number of times on HN, and I find it highly amusing. When confronted with a difficult challenge in representing the world on a computer, apparently the answer is to instead change the world.
OK, but then how are you going to handle hundreds of years of legacy texts?
Re: The string type is broken
#163Earlier quoted context omitted.
> Python 3 gets so much of this right. It's one of the things I really loved about python 3 as it allows for correct string handling in most cases (see below). One of the biggest things that I feel Python gets right with the string type is that strings are immutable . It makes a lot of things easier. It really makes sense to have a good string type for small strings, stored in unicode. Immutability makes everything s…
Isn't the string type immutable in many (most?) other languages as well? In Objective-C the default is an immutable string (though optionally one can create mutable strings as well). Lua also uses immutable strings. In Java and C# I think the situation is the same, since if you want to use high performance string manipulation, you'll generally resort some form of StringBuilder helper class.
I believe strings are mutable in Ruby.
Re: The string type is broken
#164A nitpick from the article >This spells trouble for languages using UTF-16 encodings (Java, C#, JavaScript). if they were using UTF-16, this wouldn't be a problem as UTF-16 can be used to perfectly well encode code points outside of the BMP (at the cost of losing ability for O(1) access to specific code points of course. If you need to know what the n-th code point is, you have to scan the string until the n-th posit…
> Most languages that claim Unicode support still only have UCS-2 libraries (Python 3 is a notable exception) Most non-JVM languages[1] actually use UTF-8 as the internal encoding so they should not suffer from this. Python 3 does not use UTF-16 either, it selects an encoding based on the contents of the string. http://www.python.org/dev/peps/pep-0393/ 1. I think .NET too uses UCS-2 or UTF-16, but I am not a Windows…
Python Ruby >1.8 lets you choose the encoding
.NET UCS2/UTF-16 (I know the difference, imho if the stdlib has a .size, .length or .count that works on code units instead of code points it's broken... thus I'll mention only UCS2 from now on)
Java UCS2
Clojure UCS2
Scala UCS2
QT UCS2
Haskell String UCS4
Haskell Data.Text UTF-16 (yes, not a naive UCS-2)
Rust UCS4 (last time I checked)
Javascript UCS2
Dart UCS2
PHP Unicode-oblivious
Vala UCS4
Go UTF-8 (but it lets you call len() on strings, and it doesn't return the length of the string, but its size in bytes)
I can't really think of another language that uses UTF-8 internally, are you sure?
Re: The string type is broken
#165Re: The string type is broken
#166Hat tip to Guido van Rossum for passing (nearly) all the tests in Python 3. Is the "ffl-ligature to uppercase" test really relevant? Isn't that fixed by appropriate use of string normalisation?
I also doubt the validity of the upper-casing, it feels like in an internationalization/localization context, converting a string to all upper case is not a valid thing to be doing. Not all languages (or even characters) have a well-defined upper-case versions of their glyphs. Even if they all did, I would expect the interpretation by a (human) reader to vary culturally.
Many designers use
abbr{
text-transform:lowercase;
font-variant:small-caps;
letter-spacing:.1em;
}
Re: The string type is broken
#167Re: The string type is broken
#168Earlier quoted context omitted.
ASCII doesn't make the U.S. special. ASCII is special because it's from the U.S. Lots of people speak languages that trivially fit in 8 bits with no real "figuring out" to do. Before Unicode, we all had our different codepages or encodings. Including the U.S. The U.S. is pretty central to computing. Because of that, and because ASCII only uses 7 bits, some other 8-bit cultures use it as a subset for their native 8-bi…
Many languages fit into 8 bits, but English is particularly simple in its alphabet. Even many of the European languages that can fit in 8 bits have things like accented characters that complicates things somewhat. Of course this isn't to say English is simple overall. Just that it's complexities lie elsewhere, and it's simplicities lie in an area that made it particularly simple for early computer systems to process.
> Even many of the European languages that can fit in 8 bits have things like accented characters that complicates things somewhat.
I don't see your point here, with respect to English orthography making computer implementation easier. How exactly does not needing representations for accented characters make anything easier?
Re: The string type is broken
#169Earlier quoted context omitted.
I agree, it seems like a much saner thing to do. Now that you make me think of that, I do not know many instances of this. I just could think of https://github.com/clojure-numerics/core.matrix upon which I stumbled recently. Do you have other example of efforts to separate a type from its implementations?
doesn't every statically typed imperative language do this, and recommend it?
You are right. It's interesting that I didn't think of it, probably because switching implementations in compiled languages is often less trivial, and I don't remember doing it. Actually, are there many alternative implementations of, say, the C++ STL?
Re: The string type is broken
#170Now, look over here! When I substitute this context with that, ka-pow! now it's an array of characters! Big deal. I don't understand what the point of this article is when it shows the shortcomings of half a dozen different string implementations in random languages. Yes, if you don't understand the language, then your assumptions about how it works may be wrong. Big surprise, that doesn't mean every string implement…
Unicode is a standard. It says how to act in these circumstances. Calling out incorrect unicode implementations is useful. You shouldn't have to worry about inconsistent behavior between different languages that purport to support unicode strings. That's the point of a standard.
So who is the authority about correct unicode implementations, exactly? And how to different languages with different use cases and power conform to such a standard? Why doesn't this authority extend over language implementations? Because they know what they are doing, and understand the domain, unlike the author of this article.
Look, I'm all for open standards, but saying that standards are required to be adhered to at the programming language level is just ignorance of the real world. The point of a standard isn't to dictate how data is architectured internally, it's to facilitate interoperability of systems at their endpoints. If you want interoperability of programmers, than make your own conforming language and get programmers to adopt it the right way, by competing in the market of ideas.
There is no idea here other than the writer's unjustified expectation that he should just know how every language handles Unicode because??? Because Unicode is a standard? No.. that doesn't make sense at all. Mixing contexts to make the point here means there is no ground for his argument to stand on.