Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

hsivonen.fi

Post reply on HN
Print view View original

271–280 of 315 posts

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#271

Post by hnfong »

jodrellblank wrote: post #201
2h wrote: post #25

this one one of those things that people point to when comparing languages, but in reality rarely matters. with Go, you just get the number of bytes, which the the correct default thing to do: https://godocs.io/builtin#len if the language default was anything other than this, THAT WOULD BE WRONG and unexpected. I would prefer the default to be the dumb, fast thing. then if I want the slow, fancy thing, I can import s…

For anyone wondering what Go does, it looks like Python2's way[1]; strings are byte sequences with no guarantees of UTF{anything} correctness. Go's source code is specified to be UTF8 so string literals in source code will become valid UTF8 encoded strings, but any string from any library call or code you didn't write might contain invalid Unicode text, or mixed encodings, or anything. That feels a bit "pit of despai…

The sad thing is returning unicode code points is probably not going properly do what you wanted to do either... sliding down the slippery slope, you'd end up needing a text layout renderer and a language model to do what you thought you wanted to do. (and then there'll be a thousand bugs and edge cases that your libraries didn't handle properly)

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#272

Post by GoblinSlayer »

arcticbull wrote: post #24
grayhatter wrote: post #5

> Python 3’s approach is unambiguously the worst one, though. Did I miss the part where he explains this take? It's made up of 5 valid unicode code units. For a language where you're not supposed to need to know the byte size semantics, the correct length should be 5. What am I missing? The close second being 17, because length in bytes. Is another fine way to represent this data, e.g. what a successful write of some…

It is wrong that "{emoji}".length == 7 -- but it's wrong because there's no such thing as the 'length' of a string out of context. A string should be viewed as an opaque data type with views into it depending on what you're trying to do. You can have its length in the context of storage/retrieval/transmission (UTF-8 byte count), its length in the context of parsing (code points), its length in the context of editing…

>It's particularly bad because so often (esp. for western programmers) 'countOfCodePoints' == 'countOfBytesInUTF8' == 'countOfGraphemeClusters' == """length""" so it's hella easy to accidentally write buggy software.

Then programmers will pick a random view and assume its length equals the number of characters and bytes. Also the grapheme view will introduce an OS-dependent bug.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#273

Post by chrismorgan »

pierrebai wrote: post #265
chrismorgan wrote: post #147

Earlier quoted context omitted.

> For a language where you're not supposed to need to know the byte size semantics, the correct length should be 5. What am I missing? In the words of the article: “The choice of UTF-32 (or Python 3-style code point sequences) arises from wanting the wrong thing.” “Not needing to know the byte size semantics” seems reasonable, but it simply isn’t a useful goal. The things it makes easier or faster (knowing how many c…

... but maybe it simplifies and speed-up the internal processing? I haven't looked at Python 3 C implementation of strings, but that is a guess. Also, IIRC, Python 3 has the ability to keep different internal representations of strings and uses the most compact one. If all character are 7-bit ASCII, it uses bytes representation. That's what I remember from Python dev discussions long ago. But the overall tone of the…

No. Python internally would be made much faster by working on pure UTF-8. Absolutely nothing internal to the language uses the operations that code point semantics speeds up.

Since you mention the varying internal representation of strings: that’s PEP 393 https://peps.python.org/pep-0393/>, which landed in CPython 3.3, and it generally made things slower by introducing a lot of branching and reallocating and such, though it does speed up some cases due to having to touch less memory, and some methods due to being able to quickly rule out possibilities (e.g. str.isascii can immediately return False for a canonical UCS-2 or UCS-4 string, since if they were ASCII they’d have been of the Latin-1 kind).

PEP 393 was done because people were complaining about how much memory their UCS-4 encoding had been using.

Note also how PEP 393 retains code point semantics: Latin-1 (Unicode values 0–255), UCS-2 or UCS-4; all fixed-width encodings of code point sequences. PEP 393 does also allow a string to cache UTF-8 representation (see PyCompactUnicodeObject.{utf8, utf8_length}), choosing “UTF-8 as the recommended way of exposing strings to C code”, but I gather this isn’t used very much.

(Related: PyPy 7.1 shifted to using UTF-8 exclusively internally, and according to https://www.pypy.org/posts/2019/03/pypy-v71-released-now-use... got a “nice speed bump” out of it.)

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#274

Post by lxgr »

tialaramex wrote: post #260
lxgr wrote: post #248

Earlier quoted context omitted.

> UTF-16 length is only useful if you are moving UTF-16 [...] Remember to pass on your condolences and look forward to a day when we don't do that any more. Java/the JVM says hi! Arguably, "how many bytes does this string occupy in memory/on disk (e.g. in class files)" is a pretty useful thing to be able to ask.

> Arguably, "how many bytes does this string occupy in memory/on disk (e.g. in class files)" is a pretty useful thing to be able to ask. Sure. On disk those strings are (modified) UTF-8 of course, is that what you meant ?

On disk yes, at runtime it's still UTF-16, I believe.

This is still relevant for many databases as well, as far as I remember.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#275

Post by WorldMaker »

ThunderSizzle wrote: post #177
tremon wrote: post #162

Earlier quoted context omitted.

Why is it ambiguous? The Python documentation is pretty clear about what type of elements a string contains: > Strings are immutable sequences of Unicode code points (from https://docs.python.org/3/library/stdtypes.html#text-sequenc... )

According to python. Every language can have a different definition. C#, for example, defines the blocks to be Char objects, which is based on UTF 16: > The Length property of a string represents the number of Char objects it contains, not the number of Unicode characters. https://learn.microsoft.com/en-us/dotnet/csharp/programming-...

Though C# also recommends the Rune APIs for more modern/better code point handling. The Rune APIs have a bit more in common with Python 3's unicode treatment than the classic (and sometimes wrong) UTF-16 approach.

https://learn.microsoft.com/en-us/dotnet/api/system.text.run...

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#276

Post by dang »

avgcorrection wrote: post #246
dang wrote: post #37

Earlier quoted context omitted.

Ok, we've remojied the title above. Edit: there are always exceptions - https://news.ycombinator.com/item?id=34460417

dang is definitely wrong about what “plain text” means in that thread.

Happy to wrong but when saying something like that, you should explain why and how, so we can all learn something!

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#277

Post by Dylan16807 »

Joker_vD wrote: post #128
Dylan16807 wrote: post #30

Earlier quoted context omitted.

That means 7 is also a measure of bytes, just slightly more awkward. So it's roughly on par with 17. For 5, the idea is that while you might want to iterate code points, the total number of code points is less useful than either grapheme count or byte count. I think that argument makes sense.

7 cannot be a measure of bytes because a UTF-16 point takes 2 bytes, so the number has to be even. Did you mean 14?

I meant to write 7. That's why I said "measure of" instead of "number of", because you need to multiply by 2.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#278

Post by arcticbull »

cameldrv wrote: post #82
arcticbull wrote: post #64

Earlier quoted context omitted.

Characters in context are printable or non-printable/formatting marks right? I agree they probably meant grapheme clusters, but grapheme clusters can vary dramatically in width so the point of the conversation was to explain why a bounding box was a better approximation of their goals.

They do very in width, but with a proportional font that’s true even with ASCII text. What grapheme clusters tells you is how many times you have to press the arrow key/backspace to get to the beginning of the string.

You know it's been a long time since this conversation but I think, reflecting, it has to do with grapheme clusters not being particularly consistent across operating systems and over time. The article even has an example where one Unicode spec encodes the same 5 USVs and either 1 or 2 graphemes.

Re: It’s not wrong that "🤦🏼‍♂️".length == 7 (2019)

#279

Post by avgcorrection »

dang wrote: post #276
avgcorrection wrote: post #246

Earlier quoted context omitted.

dang is definitely wrong about what “plain text” means in that thread.

Happy to wrong but when saying something like that, you should explain why and how, so we can all learn something!

Plain text is often contrasted with markup or rich text. UTF-8 text is neither markup nor rich text.
Post reply on HN