1–10 of 55 posts
JEP254: proposal to represent Java Strings as ISO-8859-1
openjdk.java.net
I have always wondered why they didn't do that, it seems to be such a simple optimization. With UTF-8 you would give up some constant time operations like character lookups, while ISO-8859-1 shouldn't have any performance regressions I can think of. Because once you add a non-latin character to a String and the encoding has to change, the data has to be copied anyway due to String immutability.
I have always wondered why they didn't do that, it seems to be such a simple optimization. With UTF-8 you would give up some constant time operations like character lookups, while ISO-8859-1 shouldn't have any performance regressions I can think of. Because once you add a non-latin character to a String and the encoding has to change, the data has to be copied anyway due to String immutability.
I guess at the time the JVM was less aggressively optimised, and UCS-2 was a single simple approach that represented all required code points. Anything else would probably have looked like a micro-optimisation.
There are numerous alternatives for encoding sequences of Unicode code points, each with its own pros and cons. UTF-8, UTF-16, UTF-32, and various compression schemes emphasize backward compatibility, or space savings, or rapid, random access of characters, or whatever. Different designs optimized for different benefits, with different associated costs. The UCS character sequence is the universal thing, but how it is represented should be optimized for the application.
It sounds as though this proposal is only about minimizing the number of bytes needed to represent a sequence of UCS characters, but ISO 8859-1 was never designed as a minimal-space representation of UCS characters. Its design reflects entirely different goals. If all you care about is an internal (not externally visible) representation that minimizes string size, you should do a proper statistical analysis first. You will find, for example, that the curly quotes and dashes that are nearly ubiquitous in serious English text and the Euro character so fundamental to international economics are far more commonly included in Java strings than are most control characters. Yet control characters are a part of ISO 8859-1, and those vastly more useful characters are not. Why, if your goal is to optimize for space, would you privilege so many obscure control characters in your default, internal representation, yet force a switch to a different encoding for any string on a blog containing an m-dash or on an e-commerce site containing a Euro sign?
Instead, if it's all about space, and it is an internal, hidden representation, you do a proper statistical analysis of exactly what it is you most need to represent, and either assign bytes in inverse proportion to commonness, or you use a proper information-theoretic compression "encoding" scheme based on the results of your analysis and your weighted goals. Or you stick with the existing, standard, run-anywhere representation Java has used since the beginning.
Either way, switching to ISO 8859-1 for this makes no sense.
This sounds like a ridiculous idea. Java treats text as all modern systems should: as a sequence of the open-ended Universal Character Set (UCS). This proposal doesn't challenge that claim, nor could it. Java strings are sequences of Unicode code points, logically at least. This proposal is just about how to encode such a sequence internally. There are numerous alternatives for encoding sequences of Unicode code poin…
This sounds like a ridiculous idea. Java treats text as all modern systems should: as a sequence of the open-ended Universal Character Set (UCS). This proposal doesn't challenge that claim, nor could it. Java strings are sequences of Unicode code points, logically at least. This proposal is just about how to encode such a sequence internally. There are numerous alternatives for encoding sequences of Unicode code poin…
Firefox did this recently, with great success. It's a simple solution that will reduce memory usage (and this, garbage collection). It makes perfect sense.
The same is not true of Java.
This sounds like a ridiculous idea. Java treats text as all modern systems should: as a sequence of the open-ended Universal Character Set (UCS). This proposal doesn't challenge that claim, nor could it. Java strings are sequences of Unicode code points, logically at least. This proposal is just about how to encode such a sequence internally. There are numerous alternatives for encoding sequences of Unicode code poin…
unfortunately, Java treats text as data encoded in UCS-2 which uses a maximum of two bytes to represent a character and all the Java string APIs assume the UCS-2 encoding.
The problem with UCS-2 is that it doesn't allow to encode characters outside of the Unicode BMP.
In order to solve this, Java actually uses UTF-16 encoding to encode characters outside of the BMP, but none of the APIs actually know about this and still assume UCS-2.
This leads to the API being wrong about character lengths and to regular expressions wrongly matching certain strings and sometimes destroying data when you apply them to replace substrings.
Case in point is this little test class here: https://gist.github.com/5745601abbfbf7068fcd
which prints 2 on the console. I've also given a talk about this at a swiss JS conference in 2012: http://pilif.me/unicode.pdf - JS has/had the same issues as Java in that regard.
The only well-known contemporary languages that get this (somewhat) right are Perl (since forever), Python 3, Ruby 1.9 and Swift, though ES6 is in process of getting up to speed too.
Python 3 and Swift have the issue of being totally tied to unicode, so if the politics and issues around Han Unification matter to you then those two are actually also not usable for you.
I have always wondered why they didn't do that, it seems to be such a simple optimization. With UTF-8 you would give up some constant time operations like character lookups, while ISO-8859-1 shouldn't have any performance regressions I can think of. Because once you add a non-latin character to a String and the encoding has to change, the data has to be copied anyway due to String immutability.
I guess at the time the JVM was less aggressively optimised, and UCS-2 was a single simple approach that represented all required code points. Anything else would probably have looked like a micro-optimisation.
One language whose name escapes me took an approach where the internal representation is always unicode codepoints, and Strings have methods for projecting those .toUTF8() or whatever to interact with the outside world - I liked that a lot due to its simplicity, anyone knows which language that was?