Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Python does Unicode

b-list.org

Post reply on HN
Print view View original

21–30 of 141 posts

Re: How Python does Unicode

#21

Post by [deleted] »

patrickthebold wrote: post #12
ekidd wrote: post #10

Earlier quoted context omitted.

> it's one big reason why I think UTF-8 is generally the wrong way to expose Unicode to a programmer, since it lets them think they can keep that cherished "one byte == one character" assumption right up until something breaks at 2AM on a weekend. Unfortunately, as long as you believe that you can index into a Unicode string, your code is going to break. The only question is how soon. I actually like UTF-8 because it…

I get the variable byte encodings. And I know that Unicode has things like U+0301 as you say, and so code points are not the same as characters/glyphs. But I don't understand why it was designed that way. Why is Unicode not simply an enumeration of characters.

[deleted]

Re: How Python does Unicode

#22

Post by Avernar »

ubernostrum wrote: post #8
wrs wrote: post #5

This was a pretty gutsy move on Python's part. The presence of a single emoji in an English string will blow up memory usage for the whole string by 4x. And because graphemes aren't 1:1 to code points, the O(1) indexing and length operations you bought with that trade-off will still confuse people who don't understand Unicode.

As I said in the article, I think the overhead of adding yet more weirdness in the form of quirks of the internal encoding (which could vary according to how the Python interpreter was compiled!) is a bad thing to do on top of how much people seem to struggle mentally just to get Unicode all on its own. Though I also think the struggle is mostly due to people being stuck in an everything-is-like-ASCII mindset, and th…

> I think UTF-8 is generally the wrong way to expose Unicode to a programmer, since it lets them think they can keep that cherished "one byte == one character" assumption right up until something breaks at 2AM on a weekend.

The solution to that is simple, don't let the programmer access individual bytes in a Unicode string.

Get rid of indexing into them and replace it with iterators. Make string handling functions work on code points at the very least but better yet on grapheme clusters. There's a little more to it than that but it's a good start.

Yes, people are still stuck in the ASCII mindset and can't seem to get away from thinking in bytes. But I belive it's the ability to index into strings is what's to blame and not the encoding used.

Re: How Python does Unicode

#23

Post by simonh »

carapace wrote: post #17

Unicode is a horrible scam, the worst thing to happen to digital language representation. This is all just so much turd polishing . (Also, that explanation of UTF-8 is crap. UTF-8 is beautiful quite apart from its utility, but you'd hardly know it from the article.) I've said it before: Unicode is a conflation of a good idea and an impossible idea. The good idea is a standard mapping from numbers to little pictures.…

Critical rants that don't suggest a better alternative, or describe what a better alternative might look like even in outline, are rarely informative or persuasive.

Re: How Python does Unicode

#24

Post by ubernostrum »

ekidd wrote: post #10
ubernostrum wrote: post #8

Earlier quoted context omitted.

As I said in the article, I think the overhead of adding yet more weirdness in the form of quirks of the internal encoding (which could vary according to how the Python interpreter was compiled!) is a bad thing to do on top of how much people seem to struggle mentally just to get Unicode all on its own. Though I also think the struggle is mostly due to people being stuck in an everything-is-like-ASCII mindset, and th…

> it's one big reason why I think UTF-8 is generally the wrong way to expose Unicode to a programmer, since it lets them think they can keep that cherished "one byte == one character" assumption right up until something breaks at 2AM on a weekend. Unfortunately, as long as you believe that you can index into a Unicode string, your code is going to break. The only question is how soon. I actually like UTF-8 because it…

Unfortunately, as long as you believe that you can index into a Unicode string, your code is going to break. The only question is how soon.

Depends on what you want to index into it for. I'll admit that once upon a time I opposed adding a "truncate at N characters" template helper to Django since there was a real risk it would cut in the middle of a grapheme cluster, and I don't particularly care for the compromise that ended up getting it added (it normalizes the string-to-truncate to a composed form first to try to minimize the chance of slicing at a bad spot).

But when you get right down to it, what I do for a living is write web applications, and sometimes I have to write validation that cares about length, or about finding specific things in specific positions, and so indexing into a string is something I have do to from time to time, and I'd rather have it behave as a sequence of code points than have it behave as a sequence of bytes in a variable-width encoding.

As to whether UTF-8 forces people to deal with Unicode up-front, I very strongly disagree; UTF-8 literally has as a design goal that it puts off your need to think about anything that isn't ASCII.

Re: How Python does Unicode

#26

Post by pseudalopex »

patrickthebold wrote: post #12
ekidd wrote: post #10

Earlier quoted context omitted.

> it's one big reason why I think UTF-8 is generally the wrong way to expose Unicode to a programmer, since it lets them think they can keep that cherished "one byte == one character" assumption right up until something breaks at 2AM on a weekend. Unfortunately, as long as you believe that you can index into a Unicode string, your code is going to break. The only question is how soon. I actually like UTF-8 because it…

I get the variable byte encodings. And I know that Unicode has things like U+0301 as you say, and so code points are not the same as characters/glyphs. But I don't understand why it was designed that way. Why is Unicode not simply an enumeration of characters.

Some languages use multiple marks on a single base character. The combining character system is more flexible, uses fewer code points, and doesn't require lookup tables to search or sort by base character.

Re: How Python does Unicode

#27

Post by ubernostrum »

captaincrowbar wrote: post #13
ekidd wrote: post #10

Earlier quoted context omitted.

> it's one big reason why I think UTF-8 is generally the wrong way to expose Unicode to a programmer, since it lets them think they can keep that cherished "one byte == one character" assumption right up until something breaks at 2AM on a weekend. Unfortunately, as long as you believe that you can index into a Unicode string, your code is going to break. The only question is how soon. I actually like UTF-8 because it…

This sort of thing is why Swift treats grapheme clusters, rather than code points or bytes or "characters", as the fundamental unit of text. When I first started learning Swift I thought that was a weird choice that would just get in the way, but these days I'm coming around to their way of thinking.

I would love it if Python at least would support the '\X' regex metacharacter in its own built-in regex module. Right now you have to turn to a third-party implementation to get that.

I also wish Python would expose more of the Unicode database than it does; I've had to turn to third-party modules that basically build their own separate database for some Unicode stuff Python doesn't provide (like access to the Script property).

Re: How Python does Unicode

#28

Post by Avernar »

patrickthebold wrote: post #12
ekidd wrote: post #10

Earlier quoted context omitted.

> it's one big reason why I think UTF-8 is generally the wrong way to expose Unicode to a programmer, since it lets them think they can keep that cherished "one byte == one character" assumption right up until something breaks at 2AM on a weekend. Unfortunately, as long as you believe that you can index into a Unicode string, your code is going to break. The only question is how soon. I actually like UTF-8 because it…

I get the variable byte encodings. And I know that Unicode has things like U+0301 as you say, and so code points are not the same as characters/glyphs. But I don't understand why it was designed that way. Why is Unicode not simply an enumeration of characters.

One reason is because it would take a lot more code points to describe all the possible combinations.

Take the country flag emoji. They're actually two seperate code points. The 26 code points used are just special country code letters A to Z. The pair of letters is the country code and shows up as a flag. So just 26 codes to make all the flags in the world. Plus new ones can be added easily without having to add more code points.

Another example is the new skin tone emoji. The new codes are just the colour and are put in front of the existing emoji codes. Existing software just shows the normal coloured emoji but you may see a square box or question mark symbol in front of it.

Re: How Python does Unicode

#29

Post by ubernostrum »

Avernar wrote: post #18

I'm not a fan of how Python 3 stores Unicode strings internally. In my opinion they should have went with UTF-8. The extra scanning and conversion puts more preassure on the processor and caches under load. I agree that Python 2's Unicode handling is broken. That's why I just stored UTF-8 in a normal string and avoided the whole mess. The only thing I have to do is validate any input from the outside world is really…

Since the high-level API is supposed to let you treat a string as a sequence of code points, a correct implementation (which Python didn't have until 3.3!) would've imposed the overhead of conversion to something resembling a fixed-width encoding whenever a programmer invoked certain operations.

And the vast majority of strings in real-world Python contain only code points also present in latin-1, which means they can be stored in one byte per code point with this approach. And for strings which can't be stored in one byte per code point, you were similarly going to pay the price sooner or later.

Re: How Python does Unicode

#30

Post by Retra »

captaincrowbar wrote: post #13
ekidd wrote: post #10

Earlier quoted context omitted.

> it's one big reason why I think UTF-8 is generally the wrong way to expose Unicode to a programmer, since it lets them think they can keep that cherished "one byte == one character" assumption right up until something breaks at 2AM on a weekend. Unfortunately, as long as you believe that you can index into a Unicode string, your code is going to break. The only question is how soon. I actually like UTF-8 because it…

This sort of thing is why Swift treats grapheme clusters, rather than code points or bytes or "characters", as the fundamental unit of text. When I first started learning Swift I thought that was a weird choice that would just get in the way, but these days I'm coming around to their way of thinking.

Pfft, that is just as bad. There is no 'fundamental unit of text'. There are different units of text that are appropriate to different tasks.

If I want to know how much memory to allocate, bytes are it. If I want to know how much screen space to allocate, font rendering metrics are it. If I want to do word-breaking, grapheme clusters are it.

None of these are fundamental.

Post reply on HN