Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

JEP254: proposal to represent Java Strings as ISO-8859-1

openjdk.java.net

Post reply on HN
Print view View original

41–50 of 55 posts

Re: JEP254: proposal to represent Java Strings as ISO-8859-1

#41

Post by TheLoneWolfling »

ygra wrote: post #34
MrBuddyCasino wrote: post #27

Earlier quoted context omitted.

That is sadly true, but at least for the Basic Multilingual Plane, charAt() and codePointAt() return correct results in constant time. The only non-broken methods are afaik codePointCount() and offsetByCodePoint(), which, as you pointed out, are not that interesting anyway. Do you know of an example of a well-designed String API? I'd be interested in what trade-offs other language designers made (newer ones, like Rus…

IMHO Python gets strings very right. The internal representation is variable, depending on the string's content (to save memory), and the external API only gives you access to code points.

The problem with Python strings is that innocuous operations can occasionally cause drastic memory usage ballooning.

A simple string append can, for instance, require up to 5x the memory consumed by the string.

Worse, this problem will only show up if someone ever enters in a high enough code point. Better hope any user-defined data never gets turned into part of a large string at any point, or you have a potential memory problem down the line.

Re: JEP254: proposal to represent Java Strings as ISO-8859-1

#42

Post by Someone »

pilif wrote: post #9
SiVal wrote: post #6

This sounds like a ridiculous idea. Java treats text as all modern systems should: as a sequence of the open-ended Universal Character Set (UCS). This proposal doesn't challenge that claim, nor could it. Java strings are sequences of Unicode code points, logically at least. This proposal is just about how to encode such a sequence internally. There are numerous alternatives for encoding sequences of Unicode code poin…

> Java treats text as all modern systems should: as a sequence of the open-ended Universal Character Set (UCS) unfortunately, Java treats text as data encoded in UCS-2 which uses a maximum of two bytes to represent a character and all the Java string APIs assume the UCS-2 encoding. The problem with UCS-2 is that it doesn't allow to encode characters outside of the Unicode BMP. In order to solve this, Java actually us…

"that get this (somewhat) right [...] and Swift"

I don't know how Perl, Ruby and Python handle this, but with Unicode, I don't think we can get better than 'somewhat right'.

For example, Swift removes some invariants on string operations that many people take for granted, such as:

  - appending to a string of length n doesn't change its first n characters.

  - appending a string of length m to one of length n gives you a string of length (m+n)

  - strings that are equal to each other have the same byte representation (=> whenever you use a set of strings, you have to choose whether you want the ability to get the same bytes out or not)
I also am not sure whether s==t and u==v implies s+u==t+v.

Re: JEP254: proposal to represent Java Strings as ISO-8859-1

#43

Post by blibble »

Skinney wrote: post #12
guard-of-terra wrote: post #11

Most of software developers don't care about encodings all day. This will give them enough rope to hang half of the world. Memory is dirt cheap these days anyway, why now? And if you have mounds of text, compress it.

Reducing memory usage also reduces time spent in garbage collection.

I would imagine most of the time would likely be spent traversing the object graph (i.e. chasing references), making it O(number of references) rather than O(allocated size)

Re: JEP254: proposal to represent Java Strings as ISO-8859-1

#44

Post by SiVal »

pilif wrote: post #9
SiVal wrote: post #6

This sounds like a ridiculous idea. Java treats text as all modern systems should: as a sequence of the open-ended Universal Character Set (UCS). This proposal doesn't challenge that claim, nor could it. Java strings are sequences of Unicode code points, logically at least. This proposal is just about how to encode such a sequence internally. There are numerous alternatives for encoding sequences of Unicode code poin…

> Java treats text as all modern systems should: as a sequence of the open-ended Universal Character Set (UCS) unfortunately, Java treats text as data encoded in UCS-2 which uses a maximum of two bytes to represent a character and all the Java string APIs assume the UCS-2 encoding. The problem with UCS-2 is that it doesn't allow to encode characters outside of the Unicode BMP. In order to solve this, Java actually us…

Python 3 and Swift have the issue of being totally tied to unicode, so if the politics and issues around Han Unification matter to you then those two are actually also not usable for you.

Those issues do matter to me, yet Unicode-based languages like Python 3 and Swift are not only usable, they are best general approach (among actually existing technologies, not among all theoretically possible approaches). I've been building Asian (CJK) infosystems since the 1980s, and there are far fewer problems with Unicode-based approaches than with the encoding soup we had before.

Re: JEP254: proposal to represent Java Strings as ISO-8859-1

#45

Post by guard-of-terra »

cesarb wrote: post #40
guard-of-terra wrote: post #33

Earlier quoted context omitted.

This might actually be slower (you have a lot of strings which have UCS2 characters down the line, have to reallocate array and copy everything over)

Aren't Java Strings immutable? Once built, they won't be modified.

You still need t construct them from e.g. array of bytes.

Re: JEP254: proposal to represent Java Strings as ISO-8859-1

#46

Post by TheLoneWolfling »

needusername wrote: post #35
moru0011 wrote: post #28

pointless. Its the number of objects not the size of primitive fields (e.g. the char array) which hurts GC and consumes memory. This proposal will save <10% on an average short string instance but probably waste performance.

I assume this is motivated by heap analysis done by Oracle on their could/SAAS/... applications. They may have quite a few large strings in old gen. To give you an example, for every application deployed Tomcat builds an retains a 200kb String. Other candidates are SQL queries, manifests or in-heap caches. But I agree with you on the performance side. My impression is that a lot of Java applications are simple data p…

Can this particular case not be solved by adding a constructor that doesn't compress the string?

Edit: "There are no plans to add any new public APIs or other interfaces.". :(

Re: JEP254: proposal to represent Java Strings as ISO-8859-1

#47

Post by Skinney »

blibble wrote: post #43
Skinney wrote: post #12

Earlier quoted context omitted.

Reducing memory usage also reduces time spent in garbage collection.

I would imagine most of the time would likely be spent traversing the object graph (i.e. chasing references), making it O(number of references) rather than O(allocated size)

Larger allocations causes the heap to fill up faster, causing garbage collection to happen more often (more traversing the graph). Also, a generational garbage collector like you have in Java will copy survivors to a new heap. The larger the object, the more you will have to copy.

Re: JEP254: proposal to represent Java Strings as ISO-8859-1

#48

Post by raiph »

Someone wrote: post #42
pilif wrote: post #9

Earlier quoted context omitted.

> Java treats text as all modern systems should: as a sequence of the open-ended Universal Character Set (UCS) unfortunately, Java treats text as data encoded in UCS-2 which uses a maximum of two bytes to represent a character and all the Java string APIs assume the UCS-2 encoding. The problem with UCS-2 is that it doesn't allow to encode characters outside of the Unicode BMP. In order to solve this, Java actually us…

"that get this (somewhat) right [...] and Swift" I don't know how Perl, Ruby and Python handle this, but with Unicode, I don't think we can get better than 'somewhat right'. For example, Swift removes some invariants on string operations that many people take for granted, such as: - appending to a string of length n doesn't change its first n characters. - appending a string of length m to one of length n gives you a…

The example you've provided is confusion/complexity related to what a "character" is.

If a language supports a string type and related functions that equate "character" with a grapheme they'll deliver your first two invariants.

With an appropriate approach to byte representations your third invariant can also be maintained.

Re: JEP254: proposal to represent Java Strings as ISO-8859-1

#49

Post by Someone »

raiph wrote: post #48
Someone wrote: post #42

Earlier quoted context omitted.

"that get this (somewhat) right [...] and Swift" I don't know how Perl, Ruby and Python handle this, but with Unicode, I don't think we can get better than 'somewhat right'. For example, Swift removes some invariants on string operations that many people take for granted, such as: - appending to a string of length n doesn't change its first n characters. - appending a string of length m to one of length n gives you a…

The example you've provided is confusion/complexity related to what a "character" is. If a language supports a string type and related functions that equate "character" with a grapheme they'll deliver your first two invariants. With an appropriate approach to byte representations your third invariant can also be maintained.

If you look at Swift, it uses what Apple calls "extended grapheme cluster" as its "Character" type.

That means:

   "e" + "`" = "è"
All three strings have 1 Swift Character in them; the first is not a prefix of the last.

And yes, you can normalize byte representations, but then you have to give up round-tripping through strings.

I maintain convinced that there is no way to do Unicode strings without drawbacks.

Re: JEP254: proposal to represent Java Strings as ISO-8859-1

#50

Post by needusername »

TheLoneWolfling wrote: post #46
needusername wrote: post #35

Earlier quoted context omitted.

I assume this is motivated by heap analysis done by Oracle on their could/SAAS/... applications. They may have quite a few large strings in old gen. To give you an example, for every application deployed Tomcat builds an retains a 200kb String. Other candidates are SQL queries, manifests or in-heap caches. But I agree with you on the performance side. My impression is that a lot of Java applications are simple data p…

Can this particular case not be solved by adding a constructor that doesn't compress the string? Edit: "There are no plans to add any new public APIs or other interfaces.". :(

> Can this particular case not be solved by adding a constructor that doesn't compress the string?

Presumably if you use one of the byte[] constructors and the encoding is already in the compression format or something compatible then yes.

Whether you'll be able to do that depends on very much on how you implemented your IO. We're still seeing way to many String#substring in our traces after it become slow in 1.7.0_06. Some of them can be fixed easily, others not so much.

Post reply on HN