[deleted]
How Python does Unicode
71–80 of 141 posts
Re: How Python does Unicode
#72Earlier quoted context omitted.
I think the devil is in the details of that opaque 'position' type. With integers you can do things like concatenate two strings and adjust the indexes referring to the second string by adding the length of the first one. If you invent a new position type you have to add support for several things like this. In any case I think the Python people were right to carry on using integers.
That would force a conversion from opaque type to integer, which would force creation of the rune to byte index. It doesn't have to be handled as a special case. The opaque type thing is an optimization hidden from the user. If you try to look at it, you get the integer value, expensively.
It's a shame to do all that work at runtime though, when the result is just going to be that the byte number in the opaque value is increased by the byte length of the first string.
I do think there's a great deal to be said for indexing and pattern matching returning opaque 'locations' (particularly if/when we have languages that let you verify at compile time that you're using the location with the right string).
But I fear doing it well would be distinctly fiddly.
Re: How Python does Unicode
#73Earlier quoted context omitted.
Critical rants that don't suggest a better alternative, or describe what a better alternative might look like even in outline, are rarely informative or persuasive.
Step One: Admit there's a problem. I heard, "Tell me more about what you think would be better." Here goes: For written languages that are well-served by a simple sequence of symbols (English, etc.) there is no problem: a catalog of the mappings from numbers to pictures is fine is all that is required. Put them in a sequence (anoint UTF-8 as the One True Encoding) and you're good-to-go. For languages that are NOT wel…
Is it possible to, based purely on the context of the symbols that the Unicode standard has added, determine the correct way of connecting the characters? Does the written language have such a formal mechanic?
Or is are tashkil extra distinctive modifications that might be like spices or sauces per (whatever you want to call a single display slot)?
Re: How Python does Unicode
#74UCS-4 is essentially never the right choice. It wastes space and thus messes up your cache. UCS-2 can be the right choice if the language you're encoding uses a lot of non-Latin glyphs (i.e. East Asian languages) but suffers from the same problem as UCS-2. UTF-8 is a good default: for most strings it's very compact, and for strings with a lot of multibyte codepoints it doesn't compare too unfavorably with UTF-16. Pyt…
Except in East Asia with a population of over one billion.
Re: How Python does Unicode
#75UCS-4 is essentially never the right choice. It wastes space and thus messes up your cache. UCS-2 can be the right choice if the language you're encoding uses a lot of non-Latin glyphs (i.e. East Asian languages) but suffers from the same problem as UCS-2. UTF-8 is a good default: for most strings it's very compact, and for strings with a lot of multibyte codepoints it doesn't compare too unfavorably with UTF-16. Pyt…
I've done the math for a few of the programs I've worked on, and the waste was negligible every time. A lot of strings like "reply" that end up being ten bytes longer in UCS-4 than UTF-8 once you add all the object and allocator overhead, progressively fewer long strings. Even the string-heavy code I worked on didn't spend much more than ten per cent of its total memory on strings, having the typical object be 40 instead of 30 bytes wasn't a big deal then.
Perhaps I should give an example. Suppose you're parsing and dealing with something. Say HTML since it's well-known. So you receive a long byte array starting "
Sometimes
". You parse the byte array and produce a number of objects, including up to four strings, namely "html", "body", "p" and "Sometimes", and by the time you've stored those in objects and allocated them, they occupy 32 bytes each on the heap. If you use UCS-4 the last may need 48 or 64, depending on your allocator's rounding and buckets. The byte array you for from the I/O subsystem may be 100k but most of the strings in the code are short, and the impact of using UCS-4 is moderate.A more interesting question is whether UCS-4's advantages are worth it. It provides an array of characters, but as the years pass, the code I see does ever less char-array processing on strings. 20-30 years ago the world was full of char pointers, now, not so much. Something like this looks more typical, and doesn't benefit much from UCS-4, if at all: foo.split(" ").each{|word| bar(word) }.
Re: How Python does Unicode
#76Earlier quoted context omitted.
Since a string doesn't have any font rendering metrics (in that it lacks a font or size), I'm not sure how you expect a language's String implementation to take it into account. Similarly, bytes will change based on encodings, which most people would expect a language's String type to abstract over. Do you really want UTF8 and UTF16 strings to behave differently and introduce even more complexity to a very complex sy…
It's not about encodings at all, actually. It's about the API that is presented to the programmer. And the way you take it all into account is by refusing to accept any defaults. So, for example, a string type should not have a "length" operation at all. It should have "length in code points", "length in graphemes" etc operations. And maybe, if you do decide to expose UTF-8 (which I think is a bad idea) - "length in…
This is exactly how Swift’s Strings work.
Re: How Python does Unicode
#77UCS-4 is essentially never the right choice. It wastes space and thus messes up your cache. UCS-2 can be the right choice if the language you're encoding uses a lot of non-Latin glyphs (i.e. East Asian languages) but suffers from the same problem as UCS-2. UTF-8 is a good default: for most strings it's very compact, and for strings with a lot of multibyte codepoints it doesn't compare too unfavorably with UTF-16. Pyt…
> I get that the idea was to maintain indexing via codepoint, but (again) in practice that's not great: usually you want to index via grapheme -- if you want to index at all. I definitely need indexes, and I don't really care about graphemes. I actually have only a vague idea what that is. I write parsers typically by using a global string and lots of indices. The important thing for me is to be able to extract chara…
The issue with your model is that’s graphemes ultimately don’t behave like you may expect a character to. For example, it may take multiple code points to make a grapheme, so getting the index of any particular one might require walking the string instead of a constant time access since any one code point could be “globbed” with its neighbors into a grapheme–in a way that is dependent on its neighbors.
Re: How Python does Unicode
#78UCS-4 is essentially never the right choice. It wastes space and thus messes up your cache. UCS-2 can be the right choice if the language you're encoding uses a lot of non-Latin glyphs (i.e. East Asian languages) but suffers from the same problem as UCS-2. UTF-8 is a good default: for most strings it's very compact, and for strings with a lot of multibyte codepoints it doesn't compare too unfavorably with UTF-16. Pyt…
> I get that the idea was to maintain indexing via codepoint, but (again) in practice that's not great: usually you want to index via grapheme -- if you want to index at all. I definitely need indexes, and I don't really care about graphemes. I actually have only a vague idea what that is. I write parsers typically by using a global string and lots of indices. The important thing for me is to be able to extract chara…
Formal parsers do without indexing, but those rolled by hand often do, for simplicity's sake. I think these cases can still be serviced by permitting indexes, but backing them by a lazily computed index table.
Re: How Python does Unicode
#79UCS-4 is essentially never the right choice. It wastes space and thus messes up your cache. UCS-2 can be the right choice if the language you're encoding uses a lot of non-Latin glyphs (i.e. East Asian languages) but suffers from the same problem as UCS-2. UTF-8 is a good default: for most strings it's very compact, and for strings with a lot of multibyte codepoints it doesn't compare too unfavorably with UTF-16. Pyt…
> Any benefit you get from using UTF-16 vanishes the moment you need to operate on it like a string, in other words.
So, don't decode to a string, and do all your character manipulation on the bytes.
> A better solution is to allow programmers to specify string encoding and default it to UTF-8.
Absolutely not: the internal representation of a string should be of no interest to a user of your language. The 'best' solution is to represent strings as a list of index lookups into a palette, and to update the palette as new graphemes are seen. This is similar to the approach Perl6 is using[0].
[0]: https://6guts.wordpress.com/2015/12/05/getting-closer-to-chr...
Re: How Python does Unicode
#80UCS-4 is essentially never the right choice. It wastes space and thus messes up your cache. UCS-2 can be the right choice if the language you're encoding uses a lot of non-Latin glyphs (i.e. East Asian languages) but suffers from the same problem as UCS-2. UTF-8 is a good default: for most strings it's very compact, and for strings with a lot of multibyte codepoints it doesn't compare too unfavorably with UTF-16. Pyt…
> Any benefit you get from using UTF-16 vanishes the moment you need to operate on it like a string, in other words. So, don't decode to a string, and do all your character manipulation on the bytes. > A better solution is to allow programmers to specify string encoding and default it to UTF-8. Absolutely not: the internal representation of a string should be of no interest to a user of your language. The 'best' solu…
> So, don't decode to a string, and do all your character manipulation on the bytes.
WHAT?!? I suppose that you've only ever worked with Latin characters. Please show a code example of changing European to African in this sentence in your language of choice, working on the bytes in any multi-byte encoding:
מהי מהירות האווירית של סנונית ארופאית ללא משא?
Yes, that is a Hebrew Monty Python quote. Now try it with a smiley somewhere in the string (HN filtered out my attempt to post the string with a smiley).
Is each application to maintain their own dictionary of code points? If the map is to be in a library, then why not have it in the language itself?