[..." "].length === 5 // in JSIt’s not wrong that "🤦🏼♂️".length == 7 (2019)
281–290 of 315 posts
Re: It’s not wrong that "🤦🏼♂️".length == 7 (2019)
#282Earlier quoted context omitted.
> I'm basing this all on the idea that it's almost always a mistake to confuse how a program manages some data, vs how a drawing lib might. Your language shouldn't concern it self with how many glyphs it needs to draw... until you actually try to draw them. Well, why not? There are a lot of things that people would want to call string.length for — drawing little equals signs under text in a terminal, for a frivolous…
> Well, why not? Separation of concerns?
I’m not sure what separation of concerns is served by not providing a method to show how many glyphs are in a given string.
Re: It’s not wrong that "🤦🏼♂️".length == 7 (2019)
#283> Python 3’s approach is unambiguously the worst one, though. Did I miss the part where he explains this take? It's made up of 5 valid unicode code units. For a language where you're not supposed to need to know the byte size semantics, the correct length should be 5. What am I missing? The close second being 17, because length in bytes. Is another fine way to represent this data, e.g. what a successful write of some…
As far as I can tell, you're only missing two things:
1. It's five "Unicode scalars," that's the name for the top-level logical unit. The term "code points" technically refers to a lower-level concept, one that varies across encodings, just not as much as the number of bytes. I didn't know that, and it's the helpful thing I learned from this article. UPDATE: And it's also not true, sorry. "code units" are the lower-level concept from the article, "code points" are a more expansive category at the same level: https://www.unicode.org/versions/Unicode10.0.0/ch03.pdf#G740...
2. The author takes it as an unstated assumption that top-level logical structure is useless because any specific usage either ignores all structure or has a point at which low-level structure comes into play. (That assumption is false: Top-level structure is useful for keeping track of what you are doing and as a sort of "common currency" for translating between different low level representations. For example, see the very first table in the article.)
Re: It’s not wrong that "🤦🏼♂️".length == 7 (2019)
#284Earlier quoted context omitted.
The python doc says "str" are immutable sequences of unicode code points. Since it implements __getitem__, its fair to call it an array (it has a length, and allows indexing). I couldn't find out in the documentation whether the __getitem__ is O(1), which I consider a deficiency -- this should definitely be well documented. It doesn't really matter how some people think "how people understand" something, the document…
> Since it implements __getitem__, its fair to call it an array (it has a length, and allows indexing) Well, weren't we talking about things being "ambiguous"? In Python we call what you describe a list. An array is something different. And people would expect something like the C (or the Java) data structure. In Python that would match the "array" lib package. And that's just discussing the meaning of array - before…
Tell me a language where a string isnt an ordered sequence of elements of some atomic text-like data type. Those may have different types - like utf8 bytes, bytes, unicode code points, grapheme clusters etc. But these are all some sort of representation of text at some level. Which one a programming language uses depends on the language, and should be checked in the documentation. Its not like some obscure “check page 2000” of the doc type small print, implying that you need to read 18 tomes of language doc before u can work with the language —- no, but if u want to work with strings in any programming language, u should know what type the elements consist of.
Btw, python my try to overload the meaning of the words array and list, but the word “array” has a generic meaning in this branch of math called computer science (an ordered sequence of elements indexible in O(1)), which is how I used it here.
Re: It’s not wrong that "🤦🏼♂️".length == 7 (2019)
#285Earlier quoted context omitted.
The python doc says "str" are immutable sequences of unicode code points. Since it implements __getitem__, its fair to call it an array (it has a length, and allows indexing). I couldn't find out in the documentation whether the __getitem__ is O(1), which I consider a deficiency -- this should definitely be well documented. It doesn't really matter how some people think "how people understand" something, the document…
Except a sequence is not an array. So OP’s definition still does not apply to python’s definition of a string in an unambiguous manner, which was the claim they were making. In fact, using the OP’s “unambiguous” definition leads to the conclusion that strings shouldn’t have a length function at all since it’s not an array.
I didnt say say a sequence is an array. I said its fair to call str objects in python arrays: https://en.m.wikipedia.org/wiki/Array_(data_type)
Re: It’s not wrong that "🤦🏼♂️".length == 7 (2019)
#286Earlier quoted context omitted.
Is having a hole from U+D800 to U+DFFF such a big deal? The parent comment was specifically talking about surrogate pairs. That to me looks more like buggy implementation issue rather than standards issue.
As a hole , it would only be annoying and a performance penalty for validation. But by its very design, it will leak, and it does in such ways that it became the worst thing to ever happen to Unicode. I don’t know of a single language or library that uses UTF-16 for strings that validates strings: every last one actually uses sequences of UTF-16 code units, potentially ill-formed, and has APIs that guarantee this wil…
Let say A is an ill formed utf-16 string with unmatched surrogates.
The problem comes when trying to convert A to utf-8. Is this the leak you are talking about?
Re: It’s not wrong that "🤦🏼♂️".length == 7 (2019)
#287Earlier quoted context omitted.
> Code point length is the most useful for people who are actually writing string algorithms based upon Unicode. What algorithms would you be writing against code points?
Codepoints is best for collaborative text editing / CRDTs (diamond types, automerge, etc). We generally model the document as a big list of unicode codepoints. We could use grapheme clusters, but the grapheme cluster boundary points change as unicode evolves, and not all systems update at the same time. Separating strings based on grapheme cluster boundaries also requires a big lookup table to be embedded in every ap…
Using code points (or scalar values, I hope) just means that it’s inefficient for everyone, because now everyone has to convert indexes (well, except Python, but it has other problems), instead of only half the people.
Going UTF-8 is fairly clearly superior: it will be the wire format, even if it’s not the language’s string format, so now environments that use UTF-8 strings never need any conversions (apart from decoding escape sequences, most likely).
Much as I hate UTF-16, I would even be inclined to argue that UTF-16 was a better choice than code points, as it will reduce the amount of extra work UTF-16 environments have to do, without changing how much UTF-8 environments have to do at all; but it also has the disadvantage that validation is vanishingly rare in UTF-16, so you’re sure to end up with lone surrogate trouble at some point, whereas UTF-8 tooling has a much stronger culture of validation, so you’re much less likely to encounter it directly and can much more comfortably just declare “valid Unicode only”.
Yes, code points is a purer concept to use. I don’t care: it’s less efficient than choosing UTF-8, which adds negative-to-negligible complexity. Please, just abandon code point indexing and embrace the UTF-8.
Re: It’s not wrong that "🤦🏼♂️".length == 7 (2019)
#288Earlier quoted context omitted.
Treating Unicode strings as a sequence of code points is a completely valid thing to do, but is usually not what you actually care about when dealing with text. Really, are code points any less of an implementation detail?
Code points are what you care about when you do any kind of text-based format encoding or decoding. Any of JSON, XML, HTML, YAML or whatever is defined by sequence of code points. There is no reason to complicate these with visual representation-specific concepts. If you have to care about the visual representation of text then you probably need to be familiar with other concepts as well.
But, given the root ancestor of this comment, it’s worth clarifying that Python’s approach to strings doesn’t help at all with things like decoding JSON/XML/HTML/YAML; what Python gives you is random access by code point index, which you won’t ever need to use in such tasks.
Re: It’s not wrong that "🤦🏼♂️".length == 7 (2019)
#289Earlier quoted context omitted.
In ruby you have " ".codepoints.size == 5 and " ".bytes.size == 17 (It also has `length` which equals codepoints.size)
JavaScript is a weird one. To count UTF-16 bytes you write: " ".length For unicode character count you write: [..." "].length And for grapheme count (or language aware word/sentence count) you write: [...new Intl.Segmenter('en-US', { granularity: "grapheme" }).segment(" ")].length For word/sentence count you swap out the granularity option.
> [..." "].length
Mind you, this is inefficient due to unnecessarily constructing an array. Here’s a more efficient version, though the difference will normally be fairly slight:
function codePointLength(str) {
let len = 0;
for (const c of str) {
len++;
}
return len;
}
> [...new Intl.Segmenter('en-US', { granularity: "grapheme" }).segment(" ")].length
Caution: Intl.Segmenter may not be available, so be sure to have a fallback if you want to use it. Chromium shipped it 2½ years ago, Safari 2 years ago, and Firefox hasn’t shipped it yet. (No idea why and I haven’t looked. It’s not always the case: I know of other Intl things that Firefox has shipped first.)
Re: It’s not wrong that "🤦🏼♂️".length == 7 (2019)
#290Until reading this I had never heard of UTF-32. It doesn't seem like a good way to encode strings.
It's useful if you want array-like semantics (e.g. O(1) lookup) on Unicode text strings, because you have a fixed size for every codepoint, unliked UTF-8. Python for example uses it internally.
Except code point indexing simply isn’t useful.
In the words of the article: “The choice of UTF-32 (or Python 3-style code point sequences) arises from wanting the wrong thing.”