Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Python does Unicode

b-list.org

Post reply on HN
Print view View original

131–140 of 141 posts

Re: How Python does Unicode

#131

Post by int_19h »

Avernar wrote: post #42
int_19h wrote: post #36

Earlier quoted context omitted.

> That's why I just stored UTF-8 in a normal string and avoided the whole mess. This only works if every library that you use agrees with you on this, and treats all strings you pass to it as UTF-8 whenever encoding matters. OTOH, if you don't care about that, then you might as well just use bytes everywhere, and get the same thing. At least in Python 3, with bytes, if a library does try to use it as a string, you'll…

> This only works if every library that you use agrees with you on this, and treats all strings you pass to it as UTF-8 whenever encoding matters. Not exactly. The library just has to treat it as a string and not worry about the encoding (i.e. not try to encode it to/from the unicode type). Only ran into this issue once and the library had an option to return everything as string so not a problem. > At least in Pytho…

> Not exactly. The library just has to treat it as a string and not worry about the encoding (i.e. not try to encode it to/from the unicode type).

Or do anything else that implies encoding. Like measure length, index, slice, change case etc.

Re: How Python does Unicode

#132

Post by int_19h »

zlynx wrote: post #98
int_19h wrote: post #35

Earlier quoted context omitted.

The point is that Go lumps together byte arrays and strings. It's a common flaw, but it's really unfortunate to see it perpetrated in a language that was designed after this lesson was already learned. A byte array is a representation of a string, for sure. But strings themselves are higher-level abstractions. It shouldn't be that easy to mix the two. An equivalent situation would be if integers were byte arrays. So…

Easily moving from bytes to strings and back is the only way it makes sense for Go. It runs on POSIX for the most part, and every. single. POSIX. API. is done in bytes. Not Unicode. Bytes. Languages like Python 3 that try to be so Unicode-pure that they crash or ignore legal Linux filenames are insane.

I would dare say that the fact that Linux filenames don't have to be valid strings (i.e. they can be arbitrary byte sequences that cannot be meaningfully interpreted using the current locale encoding) is the insane part.

But does POSIX require support for arbitrary byte sequences in filenames, or does it merely use bytes (in locale encoding) as part of its ABI? I suspect the latter, since OS X is Unix-certified, and IIRC it does use UTF-16 for filenames on HFS - so presumably their POSIX API implementation maps to that somehow. If that's correct, then that's also the sane way forward - for the sake of POSIX compatibility, use byte arrays to pass strings around, but for the sake of sanity, require them to be valid UTF-8.

Re: How Python does Unicode

#133

Post by int_19h »

Avernar wrote: post #41
int_19h wrote: post #33

Earlier quoted context omitted.

It's not about encodings at all, actually. It's about the API that is presented to the programmer. And the way you take it all into account is by refusing to accept any defaults. So, for example, a string type should not have a "length" operation at all. It should have "length in code points", "length in graphemes" etc operations. And maybe, if you do decide to expose UTF-8 (which I think is a bad idea) - "length in…

What do you mean by "expose UTF-8"? Because nothing about UTF-8 requires that you give byte access to the string. As for indexing, strings shouldn't require indexing period. That's the ASCII way of thinking, especially fixed width columns and such. You should be thinking relatively. For example, find me the first space then using that point in the string the next character needs to be letter. When you build you're co…

By "expose UTF-8" I mean exposing the underlying UTF-8 representation of the string directly on the object itelf, instead of going through a separate byte array (or byte array view, to avoid copying)

Re: How Python does Unicode

#134

Post by int_19h »

Avernar wrote: post #121
ubernostrum wrote: post #111

Earlier quoted context omitted.

You're not really changing anything, though; you're basically saying that instead of indexing to position N, you're going to take an iterator and advance it N positions, and somehow say that's a completely different operation. It isn't a different operation, and doesn't change anything about what you're doing. If you want to argue that there should be ways to iterate over graphemes and index based on graphemes, then…

If the string is stored as ASCII characters or Unicode code points (UCS-16 or UCS-32) then you are correct that not much changes. But if the string is in UTF-8, UTF-16 or the string system uses graphemes then indexing goes from O(1) to O(N). Every index operation would have to start a linear scan from the beginning of the string to get to the correct spot. With iterators it would be a quick operation to access what i…

I agree that iterators generally make more sense with strings. But sometimes, you really do want to operate on code points - for example, because you're writing a lexer, and the spec that you're implementing defines lexemes as sequences of code points.

Re: How Python does Unicode

#135

Post by coldtea »

Avernar wrote: post #127
coldtea wrote: post #107

Earlier quoted context omitted.

> As to your second point it seems to me to be a little close minded. The whole point of a universal character set was that languages can be added to it whether they be textual, symbolic or pictographic. Representing all languages is ok as a goal -- adding klingon and BS emojis not so much (from a sanity perspective, if adding them meddled with having a logical and simple representation of characters). So, it comes t…

You might be correct and 32 bits could have been enough but Unicode has restricted code points to 21 bits. Why? Because of stupid UTF-16 and surrogate pairs. I'm curious why you think that UTF-8 requires complicated lookup tables.

>I'm curious why you think that UTF-8 requires complicated lookup tables.

Because in the end it's still a Unicode encoding, and still has to deal with BS like "equivalence", right?

Which is not mechanically encoded in the err, encoding (e.g. all characters with the same bit pattern there are equivalent) but needs external tables for that.

Re: How Python does Unicode

#136

Post by Avernar »

jstimpfle wrote: post #128
Avernar wrote: post #100

Earlier quoted context omitted.

> I definitely need indexes No you don't. You need iterators, which behave like pointers. Let's say you're hundreds or thousands of characters into a string at the start of some token. Now you want to scan from that position to the end of the token. With indexes it works fast only if it's by codepoint. in a language that properly supports graphemes this would mean it would have to scan from the beginning to get to th…

Well, I could roll my own iterator which encapsulates a string and some position information, but then I'd have to wrap a lot of different operations, like advance, advance by n, compare two iterators by position, test for end position, extract character, extract slice, etc. And the code would get a lot noiser, while the only advantage I see is graphemes support, which I have never needed so far. (And I hope grapheme…

I'm not advocating that the programmer needs to implement the iterators but that the language/runtime have built in support for them.

As for searching for ASCII, which is prevalent in parsing, the iterator function to find the next specified character can do a low level and fast byte search. That's one of the benefits of UTF-8, searching for ASCII characters is super fast.

You wouldn't have to do the character position on each advance. Just have a beginning of line iterator that's updated every time you see a newline character and on error you do call a function that gives you how many characters between the current position iterator and the start of line iterator.

Working with iterators is no more coplex than working with indexes. But it's the language that needs to provide them.

Re: How Python does Unicode

#137

Post by Avernar »

int_19h wrote: post #134
Avernar wrote: post #121

Earlier quoted context omitted.

If the string is stored as ASCII characters or Unicode code points (UCS-16 or UCS-32) then you are correct that not much changes. But if the string is in UTF-8, UTF-16 or the string system uses graphemes then indexing goes from O(1) to O(N). Every index operation would have to start a linear scan from the beginning of the string to get to the correct spot. With iterators it would be a quick operation to access what i…

I agree that iterators generally make more sense with strings. But sometimes, you really do want to operate on code points - for example, because you're writing a lexer, and the spec that you're implementing defines lexemes as sequences of code points.

That's why the search functions need to be more intelegent. If you pass the search function a grapheme it will do more work. If it notices you just passed in a grapheme that's just a code point it can do a code point scan. And if the internal representstion is UTF-8 and it sees you passed in an ASCII charzcter (very common in lexing/parsing) it will just do a fast byte scan for it.

Now if the spec thinks identifiers are just a collection of code points then it's being imprecise. But things would still work if the lexer/parser you wrote returns identifiers as a bunch of graphemes because ultimately they're just a bunch of code points strung together.

It's only in situations where you need to truncate identifiers to a certain length that graphemes become important. Also normalizing them when matching identifiers would also probably be a good idea.

Re: How Python does Unicode

#138

Post by Avernar »

coldtea wrote: post #135
Avernar wrote: post #127

Earlier quoted context omitted.

You might be correct and 32 bits could have been enough but Unicode has restricted code points to 21 bits. Why? Because of stupid UTF-16 and surrogate pairs. I'm curious why you think that UTF-8 requires complicated lookup tables.

> I'm curious why you think that UTF-8 requires complicated lookup tables. Because in the end it's still a Unicode encoding, and still has to deal with BS like "equivalence", right? Which is not mechanically encoded in the err, encoding (e.g. all characters with the same bit pattern there are equivalent) but needs external tables for that.

But that's the same for UTF-16 and UTF-32. That's why I was wondering why you singled UTF-8 out, implying it needed extra handling.

Re: How Python does Unicode

#139

Post by Avernar »

int_19h wrote: post #131
Avernar wrote: post #42

Earlier quoted context omitted.

> This only works if every library that you use agrees with you on this, and treats all strings you pass to it as UTF-8 whenever encoding matters. Not exactly. The library just has to treat it as a string and not worry about the encoding (i.e. not try to encode it to/from the unicode type). Only ran into this issue once and the library had an option to return everything as string so not a problem. > At least in Pytho…

> Not exactly. The library just has to treat it as a string and not worry about the encoding (i.e. not try to encode it to/from the unicode type). Or do anything else that implies encoding. Like measure length, index, slice, change case etc.

Change case, yes, that would require actually decoding the string to the unicode type. But that could be done when needed and not every time something from my databse needs to go out to the client.

Slicing works fine on a UTF-8 string as I'm slicing between ASCII characters which don't appear inside a non ASCII character. If I needed to slice between certain code points it would still be easy as I just look for the appropriate 2-4 byte sequence and slice before or after it. Python doesn't support graphemes so can't do much with those.

Measuring length is not something that comes up for me. And indexing to an absolute spot in a string never comes up at all.

But yes, if I did have to call a text processing library I'd have to then encode/decode to the Unicode type. But that's rare enough that I can keep everything UTF-8.

Re: How Python does Unicode

#140

Post by Avernar »

int_19h wrote: post #133
Avernar wrote: post #41

Earlier quoted context omitted.

What do you mean by "expose UTF-8"? Because nothing about UTF-8 requires that you give byte access to the string. As for indexing, strings shouldn't require indexing period. That's the ASCII way of thinking, especially fixed width columns and such. You should be thinking relatively. For example, find me the first space then using that point in the string the next character needs to be letter. When you build you're co…

By "expose UTF-8" I mean exposing the underlying UTF-8 representation of the string directly on the object itelf, instead of going through a separate byte array (or byte array view, to avoid copying)

Ah, I see. I agree that it would be a bad idea to give acess to the UTF-8 representstion.

As for length in bytes, a good way to handle most use cases regarding that is to have a function that truncates the string to fit into a certain number of bytes. That way you can make sure it fits into whatever fixed buffer and the truncation would happen on a grapheme level.

Post reply on HN