Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Python does Unicode

b-list.org

Post reply on HN
Print view View original

121–130 of 141 posts

Re: How Python does Unicode

#121

Post by Avernar »

ubernostrum wrote: post #111
Avernar wrote: post #105

Earlier quoted context omitted.

I don't see why this would be hard with iterators. You have an iterstor to the start of the HICN, either at the start of a or deep in the string. Take a second iterator and set it to the first. Loop six times advancing that iterator checking to see if it's a digit. Then check if the next position is a space. For the prefix and suffix and how many characters between them you do the above but use the second iterator to…

You're not really changing anything, though; you're basically saying that instead of indexing to position N, you're going to take an iterator and advance it N positions, and somehow say that's a completely different operation. It isn't a different operation, and doesn't change anything about what you're doing. If you want to argue that there should be ways to iterate over graphemes and index based on graphemes, then…

If the string is stored as ASCII characters or Unicode code points (UCS-16 or UCS-32) then you are correct that not much changes. But if the string is in UTF-8, UTF-16 or the string system uses graphemes then indexing goes from O(1) to O(N). Every index operation would have to start a linear scan from the beginning of the string to get to the correct spot. With iterators it would be a quick operation to access what it's pointing to and very quick to advance it.

My argument is that iterators are far superior to indexing when using graphemes (or code points stored as UTF-8 but grapheme support is superior). And they don't hurt when used on ASCII or fixed width strings either so the code will work with either string format. No hairs, split or otherwise here.

Re: How Python does Unicode

#122

Post by carapace »

mixmastamyk wrote: post #120
carapace wrote: post #52

Earlier quoted context omitted.

When I'm trolling you'll know it. I have a point, I believe it's a good point, and I'm making it. For languages that can be represented as a sequence of little pictures Unicode is a little better than ASCII. For the rest, it's a scam: We tell people that we have a way of dealing with human languages in computers but it's half-baked, born in ignorance, and all the grotty details are papered over, but you can write PIZ…

I think that's too negative. Is Unicode perfect? Of course not, but it's the best we've got for now. Just as Morse, Baudot, or ASCII were the best approximations at one point in time. It's a hard problem and will take decades to for the right solutions/implementations to present themselves. Surely one day there will be an improved successor to Unicode. Things are a lot better than they were even ten years ago, howeve…

Yeah, sorry, I was pretty cranky last night. Please see my reply to simonh in this thread a few minutes ago. (I'm basically agreeing with you.)

Re: How Python does Unicode

#123

Post by carapace »

mnarayan01 wrote: post #94
carapace wrote: post #34

Earlier quoted context omitted.

Step One: Admit there's a problem. I heard, "Tell me more about what you think would be better." Here goes: For written languages that are well-served by a simple sequence of symbols (English, etc.) there is no problem: a catalog of the mappings from numbers to pictures is fine is all that is required. Put them in a sequence (anoint UTF-8 as the One True Encoding) and you're good-to-go. For languages that are NOT wel…

> I feel like it's a huge scam and a kind of cultural imperialism from us hacker types to the folks who are late to the party and for whom ASCII++ isn't going to really cut it. It's more pay-to-play than "cultural imperialism". Arabic does seem to suffer due to no primarily-Arabic country being a member of the consortium (IIRC and it hasn't changed in the last five years). If someone was willing to absorb that cost t…

"Hey Arabs, we'll computerize your language if you pay for it or show up otherwise we'll do it anyway, poorly, because it's fun for us and it makes us feel like we're helping. Hope that works for you 'cause it's what you're going to get whether you want it or not."

Yeah, I don't have a lot of respect for that.

Re: How Python does Unicode

#124

Post by gpvos »

chucksmash wrote: post #46
millstone wrote: post #14

Earlier quoted context omitted.

In practice UTF-8 has done more to enable the wrong thing, rather than forcing programmers to do the right thing. > You can't really index Unicode characters like ASCII strings But then why do strings-are-UTF8 languages like Go or D make it so easy? Why optimize your syntax with `len(txt)` and `txt[0]` when, as you point out, you can't do that? Why make it trivial to split code points or composed character sequences,…

> Why optimize your syntax with `len(txt)` and `txt[0]` when, as you point out, you can't do that? I like Rust's approach. It's a strings-are-UTF8 language but strings (both str and String): - are not directly indexable - force you to be explicit when iterating: you iterate over either `s.chars()` or `s.bytes()` - are called out in the docs as being a vector of unsigned 8-bit integers internally - support a len() met…

> - support a len() method that is called out as returning the length of that vector

They should have called that one bytelen() then.

And how do you get a proper offset for slicing? Do you then have to interpret the UTF-8 bytes yourself, or can you somehow get it via the chars() iterator or something similar?

Re: How Python does Unicode

#125

Post by carapace »

ubernostrum wrote: post #115
WorldMaker wrote: post #112

Earlier quoted context omitted.

Someone should write up EBCDIC-based UTF as an RFC. I'm sure that there's at least one COBOL programmer out there that has been waiting for that for decades. ETA: Mostly a joke, but it would also fit right in with things like WTF-8 ( https://simonsapin.github.io/wtf-8/ )

It wasn't a joke. UTF-EBCDIC is a Unicode Technical Report: http://www.unicode.org/reports/tr16/

aw, now i'm cranky again. lol

Re: How Python does Unicode

#126

Post by Avernar »

ubernostrum wrote: post #113
mjevans wrote: post #69

Earlier quoted context omitted.

int_19h's approach is still valid for this; you're asking for whole displayed characters which are combined of some (you don't need to know) number of bits in memory across several units of the memory segment(s) that hold the string. Based on your description, the correct solution is probably to use a structure or class of a more regular format to store the decoded HICN in pre-broken form. If they really only allow n…

It's more that I get tired of people declaring that indexing and length operations need to be completely and utterly and permanently forbidden and removed, and then proposing that they be replaced by operations which are equivalent to indexing and length operations. Doing these operations on sequences of code points can be perfectly safe and correct, and in 99.99%+ of real-world cases probably will be perfectly safe…

Semantically equivalent yes, access time equivalent for variable width strings no. One of the reasons for Python 3's odd internal string format is because they wanted to keep indexing and have indexing be O(1). The reason why I think replacing indexing with iterators is that it removes this restriction and they could have made the internal format UTF-8 and/or easily added support for graphemes.

I prefer to have a system where 100% of the cases are valid and teaching people corner cases is not required. We all know how well teaching people about surrogate pairs went. And we're not forbidding the 99.99% case but providing an alternative way to accomplish the exact same thing. The vast majority of code uses index variables as a form of iterator anyways so it's not that big of a change.

The main reason people keep clinging to indexing strings is that's all they know. Most high level languages don't provide another way of doing it. People who program in C quickly switch from indexing to pointers into strings. Give a C programmer an iterator into strings and they'll easily handle it.

Re: How Python does Unicode

#127

Post by Avernar »

coldtea wrote: post #107
Avernar wrote: post #106

Earlier quoted context omitted.

It didn't have emoji but it did have other combining characters. While some langages it's feasable to normalize them to single code points but other langagues it would not be. Plus the fact that some visible characters are made up of many graphemes the number of single code points would be huge. As to your second point it seems to me to be a little close minded. The whole point of a universal character set was that l…

> As to your second point it seems to me to be a little close minded. The whole point of a universal character set was that languages can be added to it whether they be textual, symbolic or pictographic. Representing all languages is ok as a goal -- adding klingon and BS emojis not so much (from a sanity perspective, if adding them meddled with having a logical and simple representation of characters). So, it comes t…

You might be correct and 32 bits could have been enough but Unicode has restricted code points to 21 bits. Why? Because of stupid UTF-16 and surrogate pairs.

I'm curious why you think that UTF-8 requires complicated lookup tables.

Re: How Python does Unicode

#128

Post by jstimpfle »

Avernar wrote: post #100
jstimpfle wrote: post #70

Earlier quoted context omitted.

> I get that the idea was to maintain indexing via codepoint, but (again) in practice that's not great: usually you want to index via grapheme -- if you want to index at all. I definitely need indexes, and I don't really care about graphemes. I actually have only a vague idea what that is. I write parsers typically by using a global string and lots of indices. The important thing for me is to be able to extract chara…

> I definitely need indexes No you don't. You need iterators, which behave like pointers. Let's say you're hundreds or thousands of characters into a string at the start of some token. Now you want to scan from that position to the end of the token. With indexes it works fast only if it's by codepoint. in a language that properly supports graphemes this would mean it would have to scan from the beginning to get to th…

Well, I could roll my own iterator which encapsulates a string and some position information, but then I'd have to wrap a lot of different operations, like advance, advance by n, compare two iterators by position, test for end position, extract character, extract slice, etc.

And the code would get a lot noiser, while the only advantage I see is graphemes support, which I have never needed so far. (And I hope graphemes are actually designed with a similar sensibility for technical concerns as is UTF-8, where I can simply parse with indexes at the byte level, looking only for ASCII characters, without headaches and with maximum performance.)

As for getting line/character from a byte or codepoint offset, that's no problem if I do the calculation only in case of an error. The alternative would be to do it on each advance, which again means ADT wrapping, thus line noise and slower performance.

Re: How Python does Unicode

#129

Post by Retra »

coldtea wrote: post #95
Retra wrote: post #30

Earlier quoted context omitted.

Pfft, that is just as bad. There is no 'fundamental unit of text'. There are different units of text that are appropriate to different tasks. If I want to know how much memory to allocate, bytes are it. If I want to know how much screen space to allocate, font rendering metrics are it. If I want to do word-breaking, grapheme clusters are it. None of these are fundamental.

> If I want to know how much memory to allocate, bytes are it. If I want to know how much screen space to allocate, font rendering metrics are it. If I want to do word-breaking, grapheme clusters are it. Size in memory/bytes you could get trivially for any string (and this doesn't change with whether you chose bytes, graphemes or code points or whatever to iterate). Screen space is irrelevant/orthogonal to encoding -…

>Screen space is irrelevant/orthogonal to encoding

Exactly. That's why measurements of string length shouldn't ever assume I'm looking for a unit-of-offset for a monospaced font.

The problem is that most naive programmers think that's what a string length is and should be.

Re: How Python does Unicode

#130

Post by masklinn »

Avernar wrote: post #103
ubernostrum wrote: post #63

Earlier quoted context omitted.

Bytes in Python 3 don't support string operators. Slight nitpick: `bytes` objects in Python 3 do not share all of the operations and methods available on `str`, but do share quite a few. Notably, `bytes` will never implement format(), but it does implement printf()-style formatting via the modulo operator. The `bytes` and `bytearray` types implement the following methods which also exist on `str` (in some cases, with…

I didn't realize the modulo operator for bytes was added. most information I've run across said it didn't work. Unfortunately most libraries for 3 will be using str so using bytes with UTF-8 inside will become more and more difficult.

> I didn't realize the modulo operator for bytes was added. most information I've run across said it didn't work.

It was added in Python 3.5 (IIRC that's the last backwards compatibility feature added, I don't remember 3.6 adding any, or any being planned for 3.7).

Post reply on HN