Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

hsivonen.fi

Post reply on HN
Print view View original

181–190 of 287 posts

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#181

Post by jlarocco »

DavidPiper wrote: post #11

I think that string length is one of those things that people (including me) don't realise they never actually want. In a production system, I have never actually wanted string length. I have wanted: - Number of bytes this will be stored as in the DB - Number of monospaced font character blocks this string will take up on the screen - Number of bytes that are actually being stored in memory "String length" is just a…

It's definitely worth thinking about the real problem, but I wouldn't say it's never helpful.

The underlying issue is unit conversion. "length" is a poor name because it's ambiguous. Replacing "length" with three functions - "lengthInBytes", "lengthInCharacters", and "lengthCombined" - would make it a lot easier to pick the right thing.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#182

Post by torstenvl »

account42 wrote: post #63
arcticbull wrote: post #35

Earlier quoted context omitted.

Taking this one step further -- there's no such thing as the context-free length of a string. Strings should be thought of more like opaque blobs, and you should derive their length exclusively in the context in which you intend to use it. It's an API anti-pattern to have a context-free length property associated with a string because it implies something about the receiver that just isn't true for all relevant usage…

> Number of code points when parsing. You shouldn't really ever care about the number of code points. If you do, you're probably doing something wrong.

I really wish people would stop giving this bad advice, especially so stridently.

Like it or not, code points are how Unicode works. Telling people to ignore code points is telling people to ignore how data works. It's of the same philosophy that results in abstraction built on abstraction built on abstraction, with no understanding.

I vehemently dissent from this view.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#183

Post by ryandrake »

raverbashing wrote: post #177
bigstrat2003 wrote: post #152

Earlier quoted context omitted.

> in the global international connected computing world it doesn’t fit at all. Most people aren't living in that world. If you're working at Amazon or some business that needs to interact with many countries around the globe, sure, you have to worry about text encoding quite a bit. But the majority of software is being written for a much narrower audience, probably for one single language in one single country. There…

This is naive at best Here's a better analogy, in the 70s "nobody planned" for names with 's in then. SQL injections, separators, "not in the alphabet", whatever. In the US. Where a lot of people with 's in their names live... Or double-barrelled names. It's a much simpler problem and still tripped a lot of people And then you have to support a user with a "funny name" or a business with "weird characters", or you ex…

Yea, I cringe when I hear the phrase "special characters." They're only special because you, the developer, decided to treat them as special, and that's almost surely going to come back to haunt you at some point in the form of a bug.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#184

Post by dhosek »

westurner wrote: post #165
craftkiller wrote: post #146

Earlier quoted context omitted.

> Notably Rust did the correct thing In addition to separate string types, they have separate iterator types that let you explicitly get the value you want. So: String.len() == number of bytes String.bytes().count() == number of bytes String.chars().count() == number of unicode scalar values String.graphemes().count() == number of graphemes (requires unicode-segmentation which is not in the stdlib) String.lines().cou…

String.graphemes().count() That's a real nice API. (Similarly, python has @ for matmul but there is not an implementation of matmul in stdlib. NumPy has a matmul implementation so that the `@` operator works.) ugrapheme and ucwidth are one way to get the graphene count from a string in Python. It's probably possible to get the grapheme cluster count from a string containing emoji characters with ICU?

Any correctly designed grapheme cluster handles emoji characters. It’s part of the spec (says the guy who wrote a Unicode segmentation library for rust).

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#185

Post by dhosek »

flohofwoe wrote: post #131
0x000xca0xfe wrote: post #123

Earlier quoted context omitted.

I'm not advocating for ASCII-everywhere, I'm for bytes-everywhere. And bytes can conveniently fit both ASCII and UTF-8. If you want to restrict your programming language to ASCII for whatever reason, fine by me. I don't need "let wohnt_bei_Böckler_STRAẞE = ..." that much. But if you allow full 8-bit bytes, please don't restrict them to UTF-8. If you need to gracefully handle non-UTF-8 sequences graphically show the a…

> "let wohnt_bei_Böckler_STRAẞE" This string cannot be encoded as ASCII in the first place. > But if you allow full 8-bit bytes, please don't restrict them to UTF-8 UTF-8 has no 8-bit restrictions... You can encode any 21-bit UNICODE codepoint with UTF-8. It sound's like you're confusing ASCII, Extended ASCII and UTF-8: - ASCII: 7-bits per "character" (e.g. not able to encode international characters like äöü) but ma…

I’d go farther and say that extended ASCII was an unmitigated disaster of compatibility issues (not to mention that more than a few scripts still don’t fit in the available spaces of an 8-bit encoding). Those of us who were around for the pre-Unicode days understand what a mess it was (not to mention the lingering issues thanks to the much vaunted backwards compatibility of some operating systems).

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#186

Post by capitainenemo »

zahlman wrote: post #157
xigoi wrote: post #32

Earlier quoted context omitted.

I prefer languages where strings are simply sequences of bytes and you get to decide how to interpret them.

Such languages do not have strings. Definitionally a string is a sequence of characters, and more than 256 characters exist. A byte sequence is just an encoding; if you are working with that encoding directly and have to do the interpretation yourself, you are not using a string. But if you do want a sequence of bytes for whatever reason, you can trivially obtain that in any version of Python.

My experience personally with python3 (and repeated interactions with about a dozen python programmers, including core contributors) is that python3 does not let you trivially work with streams of bytes, esp if you need to do character set conversions, since a tiny python2 script that I have used for decades for conversion of character streams in terminals has proved to be repeated unportable to python3. The last attempt was much larger, still failed, and they thought they could probably do it, but it would require far more code and was not worth their effort.

I'll probably just use rust for that script if python2 ever gets dropped by my distro. Reminds me of https://gregoryszorc.com/blog/2020/01/13/mercurial%27s-journ...

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#187

Post by arp242 »

bigstrat2003 wrote: post #152
baq wrote: post #17

Earlier quoted context omitted.

ASCII is very convenient when it fits in the solution space (it’d better be, it was designed for a reason), but in the global international connected computing world it doesn’t fit at all. The problem is all the tutorials, especially low level ones, assume ASCII so 1) you can print something to the console and 2) to avoid mentioning that strings are hard so folks don’t get discouraged. Notably Rust did the correct th…

> in the global international connected computing world it doesn’t fit at all. Most people aren't living in that world. If you're working at Amazon or some business that needs to interact with many countries around the globe, sure, you have to worry about text encoding quite a bit. But the majority of software is being written for a much narrower audience, probably for one single language in one single country. There…

No one is "obsessing" over anything. Reality is there are very few cases where you can use a single 8-bit character set and not run in to problems sooner or later. Say your software is used only in Greece so you use ISO-8859-7 for Greek. That works fine, but now you want to talk to your customer Günther from Germany who has been living in Greece for the last five years, or Clément from France, or Seán from Ireland and oops, you can't.

Even plain English text can't be represented with plain ASCII (although ISO-8859-1 goes a long way).

There are some cases where just plain ASCII is okay, but there are quite few of them (and even those are somewhat controversial).

The solution is to just use UTF-8 everywhere. Or maybe UTF-16 if you really have to.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#188

Post by dcrazy »

torstenvl wrote: post #182
account42 wrote: post #63

Earlier quoted context omitted.

> Number of code points when parsing. You shouldn't really ever care about the number of code points. If you do, you're probably doing something wrong.

I really wish people would stop giving this bad advice, especially so stridently. Like it or not, code points are how Unicode works. Telling people to ignore code points is telling people to ignore how data works. It's of the same philosophy that results in abstraction built on abstraction built on abstraction, with no understanding. I vehemently dissent from this view.

You’re arguing against a strawman. The advice wasn’t to ignore learning about code points; it’s that if your solution to a problem involves reasoning about code points, you’re probably doing it wrong and are likely to make a mistake.

Trying to handle code points as atomic units fails even in trivial and extremely common cases like diacritics, before you even get to more complicated situations like emoji variants. Solving pretty much any real-world problem involving a Unicode string requires factoring in canonical forms, equivalence classes, collation, and even locale. Many problems can’t even be solved at the _character_ (grapheme) level—text selection, for example, has to be handled at the grapheme _cluster_ level. And even then you need a rich understanding of those graphemes to know whether to break them apart for selection (ligatures like fi) or keep them intact (Hangul jamo).

Yes, people should learn about code points. Including why they aren’t the level they should be interacting with strings at.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#189

Post by zahlman »

capitainenemo wrote: post #186
zahlman wrote: post #157

Earlier quoted context omitted.

Such languages do not have strings. Definitionally a string is a sequence of characters, and more than 256 characters exist. A byte sequence is just an encoding; if you are working with that encoding directly and have to do the interpretation yourself, you are not using a string. But if you do want a sequence of bytes for whatever reason, you can trivially obtain that in any version of Python.

My experience personally with python3 (and repeated interactions with about a dozen python programmers, including core contributors) is that python3 does not let you trivially work with streams of bytes, esp if you need to do character set conversions, since a tiny python2 script that I have used for decades for conversion of character streams in terminals has proved to be repeated unportable to python3. The last att…

> a tiny python2 script that I have used for decades for conversion of character streams in terminals has proved to be repeated unportable to python3.

Show me.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#190

Post by capitainenemo »

zahlman wrote: post #189
capitainenemo wrote: post #186

Earlier quoted context omitted.

My experience personally with python3 (and repeated interactions with about a dozen python programmers, including core contributors) is that python3 does not let you trivially work with streams of bytes, esp if you need to do character set conversions, since a tiny python2 script that I have used for decades for conversion of character streams in terminals has proved to be repeated unportable to python3. The last att…

> a tiny python2 script that I have used for decades for conversion of character streams in terminals has proved to be repeated unportable to python3. Show me.

Heh. It always starts this way... then they confidently send me something that breaks on testing it, then half a dozen more iterations, then "python2 is doing the wrong thing" or, "I could get this working but it isn't worth the effort" but sure, let's do this one more time. Could be they were all missing something obvious - wouldn't know, I avoid python personally, apart from when necessary like with LLM glue. https://pastebin.com/j4Lzb5q1

This is a script created by someone on #nethack a long time ago. It works great with other things as well like old BBS games. It was intended to transparently rewrite single byte encodings to multibyte with an optional conversion array.

Post reply on HN