Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Python does Unicode

b-list.org

Post reply on HN
Print view View original

51–60 of 141 posts

Re: How Python does Unicode

#51

Post by Animats »

Python took the obvious approach - they already had UTF-16 and UTF-32 builds, so this was just making that mechanism dynamic.

Go and Rust expose UTF-8 at the byte level. This is something of a headache and may result in invalid string slices. It basically punts the problem back to the user.

Here's an alternative: Use UTF-8 as the internal representation, but don't expose it to the user.

If you're iterating over a string one rune or one grapheme at a time, the UTF-8 substructure is hidden from the user. Only if the user uses an explicit numeric subscript do you need to know a rune's position in string. When a request by subscript comes in, scan the string and build an index of rune subscript->byte position. This is expensive, but no worse than UTF-32 in space usage or expansion to UTF-32 in time.

Optimizations:

- Requests for s[0] to s[N], and s[-1] to s[-N], for small N, should be handled by working forwards or backwards through the UTF-8. (Yes, you can back up by rune in UTF-8. That's one of the neat features of the representation.)

- Lookup functions such as "index" should return an opaque type which represents the position into that string. If such an object is used as a subscript, there's no need to build the index by rune. If you coerce this opaque type into an integer, the index table has to be built. Adding or subtracting small integers from this opaque type should be supported by working backwards or forwards in the string.

- Regular expression processing has to be UTF-8 aware. It shouldn't need an index by rune.

This would maintain Python's existing semantics while reducing memory consumption.

Some performance measurement tool that finds all the places where an index by rune has to be built is useful. It's rare that you really need this, but sometimes you do.

Re: How Python does Unicode

#52

Post by carapace »

ubershmekel wrote: post #39
carapace wrote: post #34

Earlier quoted context omitted.

Step One: Admit there's a problem. I heard, "Tell me more about what you think would be better." Here goes: For written languages that are well-served by a simple sequence of symbols (English, etc.) there is no problem: a catalog of the mappings from numbers to pictures is fine is all that is required. Put them in a sequence (anoint UTF-8 as the One True Encoding) and you're good-to-go. For languages that are NOT wel…

> Unicode is a horrible scam, the worst thing to happen to digital language representation. > They baldly admit that Unicode is not good for drawing Arabic > Consider the "Base-4 fractions in Telugu" [...] any software that wants to use them properly will require some code to translate to and from numbers in the computer to Telugu sequences of those graphics. Written language is hard to represent, encode and draw. Yo…

When I'm trolling you'll know it. I have a point, I believe it's a good point, and I'm making it.

For languages that can be represented as a sequence of little pictures Unicode is a little better than ASCII. For the rest, it's a scam: We tell people that we have a way of dealing with human languages in computers but it's half-baked, born in ignorance, and all the grotty details are papered over, but you can write PIZZA SLICE or POOP now, so fuck it, ship it.

Represent: "Astral Plane"? What does that have to do with a standard catalog mapping numbers to pictures? I feel Unicode messes that up.

Encoding: UTF-8 is near perfect, 'nuff said. Ken Thompson and Rob Pike doing their thing.

Drawing: Doesn't even begin to touch it really.

Unicode is a nasty little black hole that's sucking up time and other resources and not really solving the problem.

Re: How Python does Unicode

#53

Post by Animats »

ubernostrum wrote: post #24
ekidd wrote: post #10

Earlier quoted context omitted.

> it's one big reason why I think UTF-8 is generally the wrong way to expose Unicode to a programmer, since it lets them think they can keep that cherished "one byte == one character" assumption right up until something breaks at 2AM on a weekend. Unfortunately, as long as you believe that you can index into a Unicode string, your code is going to break. The only question is how soon. I actually like UTF-8 because it…

Unfortunately, as long as you believe that you can index into a Unicode string, your code is going to break. The only question is how soon. Depends on what you want to index into it for. I'll admit that once upon a time I opposed adding a "truncate at N characters" template helper to Django since there was a real risk it would cut in the middle of a grapheme cluster, and I don't particularly care for the compromise t…

Yes, while "back up one UTF-8 rune" is a well defined operation, "back up one grapheme" is tough. Forward is easy, though.

I had the need to write grapheme-level word wrap in Rust. Here it is. It assumes all graphemes have the same visible width. This is used mostly for debug output, not for general text rendering.

[1] https://github.com/John-Nagle/rust-rssclient/blob/master/src...

Re: How Python does Unicode

#54

Post by ubernostrum »

carapace wrote: post #37
ubernostrum wrote: post #31

Earlier quoted context omitted.

that explanation of UTF-8 is crap. UTF-8 is beautiful quite apart from its utility, but you'd hardly know it from the article My goal was not to judge UTF-8 aesthetically, but to explain how it works and point out that it's a variable-width encoding which emphasizes its compatibility with ASCII for strings containing only code points Unicode Consortium et. al. are absurdly arrogant. I would agree that Unicode as it e…

Oh, Hi there. Apologies for being cranky. You did a great job explaining how Python now handles Unicode! To me it was strange reading about UTF-32 first and then getting to UTF-8 from that context. It seemed to obscure the coolth and beauty of the format. Overall a great article, sorry again for being so negative.

That section was written for people who know little to nothing about Unicode and the ways Unicode can be encoded to bytes. So it starts with the obvious approach -- just spit out a sequence of bytes whose integer values are the code points, which is near enough as makes no difference to how UTF-32 works -- then introduces variable-width encoding through the history of UCS-2 and UTF-16, then gets to UTF-8 and what motivated it.

The advantages/disadvantages of the various encodings is something that could eat up several pieces just as long as the entire post, and for fun I'd probably throw in weird stuff like the attempt to do EBCDIC-compatible UTF instead of ASCII-compatible, etc.

Re: How Python does Unicode

#55

Post by hsivonen »

Animats wrote: post #51

Python took the obvious approach - they already had UTF-16 and UTF-32 builds, so this was just making that mechanism dynamic. Go and Rust expose UTF-8 at the byte level. This is something of a headache and may result in invalid string slices. It basically punts the problem back to the user. Here's an alternative: Use UTF-8 as the internal representation, but don't expose it to the user. If you're iterating over a str…

> Go and Rust expose UTF-8 at the byte level. This is something of a headache and may result in invalid string slices. It basically punts the problem back to the user.

In Go, yes. In Rust, no. UTF-8 in Go is garbage in, garbage out. Rust, however, won't let you materialize an invalid &str without "unsafe".

Re: How Python does Unicode

#56

Post by hackits »

hsivonen wrote: post #55
Animats wrote: post #51

Python took the obvious approach - they already had UTF-16 and UTF-32 builds, so this was just making that mechanism dynamic. Go and Rust expose UTF-8 at the byte level. This is something of a headache and may result in invalid string slices. It basically punts the problem back to the user. Here's an alternative: Use UTF-8 as the internal representation, but don't expose it to the user. If you're iterating over a str…

> Go and Rust expose UTF-8 at the byte level. This is something of a headache and may result in invalid string slices. It basically punts the problem back to the user. In Go, yes. In Rust, no. UTF-8 in Go is garbage in, garbage out. Rust, however, won't let you materialize an invalid &str without "unsafe".

> Go and Rust expose UTF-8 at the byte level.

Or you can take the C++/C approach and have a character 1 byte, 2 bytes, or a multi-byte. It's a pain in the ass to constantly in C/C++ having to interface between two libraries that one decided to use char and another w_char!

Re: How Python does Unicode

#57

Post by hsivonen »

captaincrowbar wrote: post #13
ekidd wrote: post #10

Earlier quoted context omitted.

> it's one big reason why I think UTF-8 is generally the wrong way to expose Unicode to a programmer, since it lets them think they can keep that cherished "one byte == one character" assumption right up until something breaks at 2AM on a weekend. Unfortunately, as long as you believe that you can index into a Unicode string, your code is going to break. The only question is how soon. I actually like UTF-8 because it…

This sort of thing is why Swift treats grapheme clusters, rather than code points or bytes or "characters", as the fundamental unit of text. When I first started learning Swift I thought that was a weird choice that would just get in the way, but these days I'm coming around to their way of thinking.

Treating grapheme clusters as fundamental is slightly problematic in the sense that then the fundamentals change as Unicode adds more combining characters. A reasonable programming environment should still provide iteration by grapheme cluster as a library feature whose exact behavior is expected to change over time as the library tracks new Unicode versions.

Depending on the task at hand, iterating by UTF-8 byte or by code point can make sense, too. And the definition of these is frozen regardless of Unicode version, which makes these safer candidates for "fundamental" operations. There is no right unit of iteration for all tasks.

Re: How Python does Unicode

#58

Post by hsivonen »

hackits wrote: post #56
hsivonen wrote: post #55

Earlier quoted context omitted.

> Go and Rust expose UTF-8 at the byte level. This is something of a headache and may result in invalid string slices. It basically punts the problem back to the user. In Go, yes. In Rust, no. UTF-8 in Go is garbage in, garbage out. Rust, however, won't let you materialize an invalid &str without "unsafe".

> Go and Rust expose UTF-8 at the byte level. Or you can take the C++/C approach and have a character 1 byte, 2 bytes, or a multi-byte. It's a pain in the ass to constantly in C/C++ having to interface between two libraries that one decided to use char and another w_char!

The way the C and C++ committees approach Unicode is even worse than Python breaking away from UTF-16 in the wrong direction (UTF-32 being the wrong direction and UTF-8 being the right direction).

The first rule of reasonably happy C and C++ Unicode programming is not to use wchar_t for any purpose other than immediate interaction with the Win32 API.

The second rule of reasonably happy C and C++ Unicode programming is not to use the standard library facilities (which depend on the execution environment) for text processing but using some other library where the UTF-* interpretation of inputs and outputs doesn't shift depending on the execution environment or compilation environment.

Re: How Python does Unicode

#59

Post by ghayes »

Animats wrote: post #51

Python took the obvious approach - they already had UTF-16 and UTF-32 builds, so this was just making that mechanism dynamic. Go and Rust expose UTF-8 at the byte level. This is something of a headache and may result in invalid string slices. It basically punts the problem back to the user. Here's an alternative: Use UTF-8 as the internal representation, but don't expose it to the user. If you're iterating over a str…

If you look over Elixir's doc [0] on binary strings, they take the best of both worlds. The APIs are specifically crafted for least surprise, e.g. with `String.length()`, `byte_size`, `Strings.graphemes()` and `String.codepoints()` functions.

[0] https://hexdocs.pm/elixir/String.html

Re: How Python does Unicode

#60

Post by hackits »

hsivonen wrote: post #58
hackits wrote: post #56

Earlier quoted context omitted.

> Go and Rust expose UTF-8 at the byte level. Or you can take the C++/C approach and have a character 1 byte, 2 bytes, or a multi-byte. It's a pain in the ass to constantly in C/C++ having to interface between two libraries that one decided to use char and another w_char!

The way the C and C++ committees approach Unicode is even worse than Python breaking away from UTF-16 in the wrong direction (UTF-32 being the wrong direction and UTF-8 being the right direction). The first rule of reasonably happy C and C++ Unicode programming is not to use wchar_t for any purpose other than immediate interaction with the Win32 API. The second rule of reasonably happy C and C++ Unicode programming i…

This is where we run into a little bit of a problem. You have a char pointer that can be either a multiple byte encoded (depending on the code page window is using). It also can be UTF-8 encoded. Then when you move onto windows wchar_t that is originally defined as (UCS-2) then was later renamed to UTF-16, due to surrogate pair's.

So in the windows world with COM/DCOM you're basically nugged into using UTF-16 wchar_t or it becomes a hell of a lot of pain. So it is easier just simply to accept to use UTF-16 and do all the conversion from UTF-8, UTF-32, code pages to a single encoding standard.

Post reply on HN