Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Python does Unicode

b-list.org

Post reply on HN
Print view View original

81–90 of 141 posts

Re: How Python does Unicode

#81

Post by dotancohen »

Arnt wrote: post #75
camgunz wrote: post #47

UCS-4 is essentially never the right choice. It wastes space and thus messes up your cache. UCS-2 can be the right choice if the language you're encoding uses a lot of non-Latin glyphs (i.e. East Asian languages) but suffers from the same problem as UCS-2. UTF-8 is a good default: for most strings it's very compact, and for strings with a lot of multibyte codepoints it doesn't compare too unfavorably with UTF-16. Pyt…

I've done the math for a few of the programs I've worked on, and the waste was negligible every time. A lot of strings like "reply" that end up being ten bytes longer in UCS-4 than UTF-8 once you add all the object and allocator overhead, progressively fewer long strings. Even the string-heavy code I worked on didn't spend much more than ten per cent of its total memory on strings, having the typical object be 40 ins…

> A more interesting question is whether UCS-4's advantages are worth it. It provides an array of characters, but as the years pass, the code I see does ever less char-array processing on strings. 20-30 years ago the world was full of char pointers, now, not so much. Something like this looks more typical, and doesn't benefit much from UCS-4, if at all: foo.split(" ").each{|word| bar(word) }.

You are looking at the issue from the perspective from a language user, not a language designer. 20 years ago we didn't have languages such as Python/Ruby which had internal multibyte support in their sting manipulation functions. 20 years ago string manipulation functions didn't even exist!

But this post is about the design of the language, not the application, and the language is still written in C/C++ and _internally_ stores strings as byte arrays that must be presented nicely to the programmer in that language's string manipulation functions.

Re: How Python does Unicode

#82

Post by falsedan »

dotancohen wrote: post #80
falsedan wrote: post #79

Earlier quoted context omitted.

> Any benefit you get from using UTF-16 vanishes the moment you need to operate on it like a string, in other words. So, don't decode to a string, and do all your character manipulation on the bytes. > A better solution is to allow programmers to specify string encoding and default it to UTF-8. Absolutely not: the internal representation of a string should be of no interest to a user of your language. The 'best' solu…

> So, don't decode to a string, and do all your character manipulation on the bytes. WHAT?!? I suppose that you've only ever worked with Latin characters. Please show a code example of changing European to African in this sentence in your language of choice, working on the bytes in any multi-byte encoding: מהי מהירות האווירית של סנונית ארופאית ללא משא?‏ Yes, that is a Hebrew Monty Python quote. Now try it with a smil…

I don't understand your complaints. You clearly have some task you have in mind that you wish to perform: why not tell me what it is?

> Please show a code example of changing European to African in this sentence in your language of choice, working on the bytes in any multi-byte encoding:

מהי מהירות האווירית של סנונית ארופאית ללא משא?‏

I don't see the string 'European' in that sentence, it seems to be solely comprised of Hebrew characters.

edit to attempt to answer your question:

    struct m {
        pos_t start;
        pos_t end;
    }

    int findsn(char* str, char* substr, match m) {
        next: for( int c_i = 0; c_i++; s[c_i] != '\0' ) {
            match.start = c_i;
            int s_i = 0;
            for( ; s_i++; substr[s_i] != '\0' ) {
                if( str[c_i] != substr[s_i] ) goto next;
            }
            match.end = c_i + s_i;
            return 1;
        }
        return 0;
    }

    char* replacesn(char* str, char* needle, char* rpl) {
        match m;
        if( findsn(str, needle, &m) ) {
            splicesn(str, m.start, m.end, rpl);
        }
        return str;
    }
splicesn should be obvious, and you normalise your strings before calling replacesn. This is just me crappily re-implementing a fraction of the wchar API without checking MSDN.

edit 2:

> Is each application to maintain their own dictionary of code points?

No, you use the system/standard library for composing/decomposing/normalising codepoints.

> If the map is to be in a library, then why not have it in the language itself?

Why not indeed? What a great idea.

Re: How Python does Unicode

#83

Post by [deleted] »

dotancohen wrote: post #81
Arnt wrote: post #75

Earlier quoted context omitted.

I've done the math for a few of the programs I've worked on, and the waste was negligible every time. A lot of strings like "reply" that end up being ten bytes longer in UCS-4 than UTF-8 once you add all the object and allocator overhead, progressively fewer long strings. Even the string-heavy code I worked on didn't spend much more than ten per cent of its total memory on strings, having the typical object be 40 ins…

> A more interesting question is whether UCS-4's advantages are worth it. It provides an array of characters, but as the years pass, the code I see does ever less char-array processing on strings. 20-30 years ago the world was full of char pointers, now, not so much. Something like this looks more typical, and doesn't benefit much from UCS-4, if at all: foo.split(" ").each{|word| bar(word) }. You are looking at the i…

[deleted]

Re: How Python does Unicode

#84

Post by masklinn »

camgunz wrote: post #47

UCS-4 is essentially never the right choice. It wastes space and thus messes up your cache. UCS-2 can be the right choice if the language you're encoding uses a lot of non-Latin glyphs (i.e. East Asian languages) but suffers from the same problem as UCS-2. UTF-8 is a good default: for most strings it's very compact, and for strings with a lot of multibyte codepoints it doesn't compare too unfavorably with UTF-16. Pyt…

> I get that the idea was to maintain indexing via codepoint, but (again) in practice that's not great

Of course not, but it was considered that breaking O(1) indexing guarantees were a bridge too far even in the breaky release of Python 3.

Re: How Python does Unicode

#85

Post by masklinn »

Animats wrote: post #51

Python took the obvious approach - they already had UTF-16 and UTF-32 builds, so this was just making that mechanism dynamic. Go and Rust expose UTF-8 at the byte level. This is something of a headache and may result in invalid string slices. It basically punts the problem back to the user. Here's an alternative: Use UTF-8 as the internal representation, but don't expose it to the user. If you're iterating over a str…

> Go and Rust expose UTF-8 at the byte level. This is something of a headache and may result in invalid string slices.

Rust will panic on invalid slices unless you first convert to raw bytes, and then it will not allow converting invalid slices back to a string in safe rust (in unsafe you're obviously on your own).

Safe Rust guarantees and requires[0] that strings are valid UTF8 at all times.

That aside, essentially all of your desires are part of Swift's string, you should check them out.

> Requests for s[0] to s[N], and s[-1] to s[-N], for small N, should be handled by working forwards or backwards through the UTF-8. (Yes, you can back up by rune in UTF-8. That's one of the neat features of the representation.)

Rust does that through the `chars()` iterator[1] which iterates through USVs (codepoints) and can be iterated from both ends. Sadly unlike Swift it does not ship with a grapheme cluster iterator. Happily there is a unicode_segmentation crate[2]. Swift also uses iterators but has more of them: the default iteration works on extended grapheme clusters, and alternate iterators are USV, UTF-16 and UTF-8.

If indexing is necessary for some reason Rust also has char_indices() which iterates on the USV and its (byte) position in the string.

> - Lookup functions such as "index" should return an opaque type which represents the position into that string. If such an object is used as a subscript, there's no need to build the index by rune. If you coerce this opaque type into an integer, the index table has to be built. Adding or subtracting small integers from this opaque type should be supported by working backwards or forwards in the string.

That is what Swift does. `String.index(of:String)` will return a String.Index: https://developer.apple.com/documentation/swift/string.index and indexed String methods will work based on that index type. This includes "reindexing" (offsetting) which is done using String.index(String.Index, offsetBy: String.IndexDistance). Furthermore String exposes two built-in indexes startIndex and endIndex as well as an "indices" iterator.

> This would maintain Python's existing semantics while reducing memory consumption.

It would not maintain O(1) USV indexing (especially in the C API), which was the reason for not just switching to UTF8.

In fact, FSR strings already contain a full UTF8 representation of the string[3], which the latin1 representation can share for pure ASCII strings.

[0] a non-utf8 str is one of Rust's 10 undefined behaviours, part of the "invalid primitive values" section alongside null references or invalid booleans: https://doc.rust-lang.org/nomicon/meet-safe-and-unsafe.html

[1] https://doc.rust-lang.org/std/primitive.str.html#method.char...

[2] https://kbknapp.github.io/clap-rs/unicode_segmentation/index...

[3] https://github.com/python/cpython/blob/49b2734bf12dc1cda80fd...

Re: How Python does Unicode

#86

Post by masklinn »

dotancohen wrote: post #81
Arnt wrote: post #75

Earlier quoted context omitted.

I've done the math for a few of the programs I've worked on, and the waste was negligible every time. A lot of strings like "reply" that end up being ten bytes longer in UCS-4 than UTF-8 once you add all the object and allocator overhead, progressively fewer long strings. Even the string-heavy code I worked on didn't spend much more than ten per cent of its total memory on strings, having the typical object be 40 ins…

> A more interesting question is whether UCS-4's advantages are worth it. It provides an array of characters, but as the years pass, the code I see does ever less char-array processing on strings. 20-30 years ago the world was full of char pointers, now, not so much. Something like this looks more typical, and doesn't benefit much from UCS-4, if at all: foo.split(" ").each{|word| bar(word) }. You are looking at the i…

> You are looking at the issue from the perspective from a language user, not a language designer. 20 years ago we didn't have languages such as Python/Ruby which had internal multibyte support in their sting manipulation functions.

20 years ago was 1997. I'm reasonably certain NSString has been unicode-aware for much longer than that.

> 20 years ago string manipulation functions didn't even exist!

What kind of absolute utter nonsense is that?

> But this post is about the design of the language, not the application, and the language is still written in C/C++ and _internally_ stores strings as byte arrays that must be presented nicely to the programmer in that language's string manipulation functions.

So?

Re: How Python does Unicode

#87

Post by masklinn »

mark-r wrote: post #4

I've always been curious on how this change in 3.3 impacts the C/C++ interface. I don't really know where to look it up, and since I haven't yet had to code a C++ library for Python I've had no burning need to answer the question.

https://www.python.org/dev/peps/pep-0393/ has details down to C API changes related to the FSR implementation.

Re: How Python does Unicode

#88

Post by dotancohen »

masklinn wrote: post #86
dotancohen wrote: post #81

Earlier quoted context omitted.

> A more interesting question is whether UCS-4's advantages are worth it. It provides an array of characters, but as the years pass, the code I see does ever less char-array processing on strings. 20-30 years ago the world was full of char pointers, now, not so much. Something like this looks more typical, and doesn't benefit much from UCS-4, if at all: foo.split(" ").each{|word| bar(word) }. You are looking at the i…

> You are looking at the issue from the perspective from a language user, not a language designer. 20 years ago we didn't have languages such as Python/Ruby which had internal multibyte support in their sting manipulation functions. 20 years ago was 1997. I'm reasonably certain NSString has been unicode-aware for much longer than that. > 20 years ago string manipulation functions didn't even exist! What kind of absol…

That should have been _30_ years ago string manipulation functions didn't exist.

NSString may have been Unicode-aware (I've never used Objective-C), and I believe that even the early Javas supported multibyte strings, but at that time most business and consumer desktop applications in the Windows world were still written in C/C++. Do you remember when the Euro symbol became common? I'm pretty sure that character alone was responsible for much of the push to support Unicode.

Re: How Python does Unicode

#89

Post by dotancohen »

falsedan wrote: post #82
dotancohen wrote: post #80

Earlier quoted context omitted.

> So, don't decode to a string, and do all your character manipulation on the bytes. WHAT?!? I suppose that you've only ever worked with Latin characters. Please show a code example of changing European to African in this sentence in your language of choice, working on the bytes in any multi-byte encoding: מהי מהירות האווירית של סנונית ארופאית ללא משא?‏ Yes, that is a Hebrew Monty Python quote. Now try it with a smil…

I don't understand your complaints. You clearly have some task you have in mind that you wish to perform: why not tell me what it is? > Please show a code example of changing European to African in this sentence in your language of choice, working on the bytes in any multi-byte encoding: מהי מהירות האווירית של סנונית ארופאית ללא משא?‏ I don't see the string 'European' in that sentence, it seems to be solely comprised…

You win on the string replace, that was a bad example. Try a regex replace! But I will also mention that seeing properly indented code with clear identifier names is refreshing where I work!

> Why not indeed? What a great idea.

It sounded to me that you were arguing that string manipulation functions do not need to be included in modern programming languages. You said: "don't decode to a string, and do all your character manipulation on the bytes"

Re: How Python does Unicode

#90

Post by simonh »

mjevans wrote: post #73
carapace wrote: post #34

Earlier quoted context omitted.

Step One: Admit there's a problem. I heard, "Tell me more about what you think would be better." Here goes: For written languages that are well-served by a simple sequence of symbols (English, etc.) there is no problem: a catalog of the mappings from numbers to pictures is fine is all that is required. Put them in a sequence (anoint UTF-8 as the One True Encoding) and you're good-to-go. For languages that are NOT wel…

Is it possible to, based purely on the context of the symbols that the Unicode standard has added, determine the correct way of connecting the characters? Does the written language have such a formal mechanic? Or is are tashkil extra distinctive modifications that might be like spices or sauces per (whatever you want to call a single display slot)?

You can't determine that purely from Unicode, you have to also know the conventions used in writing Arabic script. However Unicode is not intended to encode such conventions.

Suppose these conventions change, as they have throughout history? Or if there are different variations of these conventions in different regions or sub-dialects? Also for example in Arabic it's often possible to determine the pronunciation of a word from it's context in a sentence, but in other contexts it isn't and so the tashkil are added. There's no way for a system like Unicde to ddecide that for you. For example suppose you cut-and-paste the word from one sentence into another, should Unicode somehow automatically add or remove the tashkil? No, that's up to the author (e.g. performing the edit in a word processor) or the program performing the operation if it's being done programatically.

Unicode provides one layer in the stack. Fonts provide another layer. Program code or editorial sensibility provides another layer. Many criticisms of Unicode are premised on the expectation that it should be solving problems that belong to another layer. Not all criticisms, it's a complex system that has had to make many compromises and there have been a series of mistakes in it's history, but taken overall it's been unbelievably successful and useful.

I'm in awe of the way it solves such a huge range of problems in the space, that people picking nits about the gaps that remain are piss me off, especially when they're based on a fundamental misunderstanding of the problem it's actually solving. Cynicism is easy, solving hard problems is not. I know who gets my respect.

Post reply on HN