71–80 of 247 posts
This is a nice dive into limitations in Python's unicode handling and at the end, how to work around some problems. But you could use languages with proper unicode support like Swift or Elixir (weirdly HN is fighting flags in comment code which makes examples header to demonstrate).
If you think the Unicode flag emoji take a lot of bytes, then consider the family emoji! ( https://unicode.org/emoji/charts/full-emoji-list.html#family ) I'm in the process of designing a scripting language and implementing it in C++. I plan to put together a YouTube series about it. (Doesn't everyone want to see Bison and Flex mixed with proper unit tests and C++20 code?) Due to my future intended use case, I needed…
I think the trap Unicode got in to is technically they can have infinite emoji so they just don’t ever have a way to say no to new proposals.
This misses the real problem with flag emoji in that they are composed of codepoints that can be in any order. With other emoji you get a base codepoint with potential combining characters. Using a table of combining character ranges you can skip over them and isolate the logical glyph sequences. You don't need surrounding context to parse them out like flags need.
But then again, flags seem to be not only Unicode-hard but post-Unicode-hard.
Earlier quoted context omitted.
Thanks for that interesting detail! If such re-purposing continues, it might be easier to go straight to utf-32 for some use cases.
Nope, because the repurposing is independent of how the Unicode is represented. There's absolutely no advantage to having a string in UTF-32 over UTF-8 since you'll still need to examine every character and the added overhead for converting byte strings in UTF-8 to 32-bit code points is by far offset by the huge memory increase necessary to store UTF-32. What's more, it's really not that difficult to start at the end…
To expand, if the most-significant-bit is a 0, it's an ASCII codepoint. If the top two are '10', it's a continuation byte, and if they're '11', it's the start of a multibyte codepoint (the other most-significant-bits specify how long it is to facilitate easy codepoint counting).
So a naive codepoint reversal algorithm would start at the end, and move backwards until it sees either an ASCII codepoint or the start of a multibyte one. Upon reaching it, copy those 1-4 bytes to the start of a new buffer. Continue until you reach the start.
Earlier quoted context omitted.
There's a specification problem here. I like to say that a "string" isn't a data structure, it's the absence of one. Discussing "strings" is pointless. It follows that comparing programming languages by their "string" handling is likewise pointless. Case in point: a "struct" in languages like C and Rust is literally a specification of how to treat segments of a "string" of contiguous bytes.
Even the most basic ASCII string is still a data structure. Is it a PASCAL string (length byte followed by data) or a C string (arbitrary run of bytes terminated by a null character)?
Of course, various programming languages have primitives and concepts which they may label "string". But you still need to specify that context, drawing in the additional specification those languages provide. Plus, traditionally and in practice, such concepts often serve the function of importing or exporting unstructured data. So even in the context of a specific programming language, the label "string" is often used to elide details necessary to understanding the content and semantics of some particular chunk of data.
Earlier quoted context omitted.
In my browser (Firefox on Windows), the thing between the quotes in the first block of code looks like a picture of the US flag cropped to a circle, not like the characters "us".
Ah I see, I just opened it in firefox. It looks like some JS library is not getting loaded in Edge. The author was talking about "us", "so", etc. looking like one character and I thought I was going crazy, lol.
If you think the Unicode flag emoji take a lot of bytes, then consider the family emoji! ( https://unicode.org/emoji/charts/full-emoji-list.html#family ) I'm in the process of designing a scripting language and implementing it in C++. I plan to put together a YouTube series about it. (Doesn't everyone want to see Bison and Flex mixed with proper unit tests and C++20 code?) Due to my future intended use case, I needed…
- Counting, rendering and collapsing grapheme clusters (like the flag emoji)
- Converting between legacy encodings (shiftjis, ko8, etc) and UTF-8 / UTF-16
- Canonicalization
If all you need is to deal with utf8 byte buffers, you don't need all that stuff. And your code can stay simple, small and fast.
IIRC the rust standard library doesn't bother supporting any of the hard parts in unicode. The only real unicode support in std is utf8 validation for strings. All the complex aspects of unicode are delegated to 3rd party crates.
By contrast, nodejs (and web browsers) do all of this. But they implement it in the same way you're suggesting - they simply call out to libicu.
Earlier quoted context omitted.
> imagine handing a Unicode string to a human. They could without any knowledge look at the characters they see and produce the correct string reversal. I really highly doubt it. How do you reverse this?: مرحبًا ، هذه سلسلة. Can you do it without any knowledge about whether what looks like one character is actually a special case joiner between two adjacent codepoints that only happens in one direction? Can you do it…
I mean no but only because I don’t understand the characters. Someone who reads Arabic (I assume based on the shape) would have no trouble. You’re nitpicking cases where for some readers visual characters might be hard to distinguish but it doesn’t change the fact that there exists a correct answer for every piece of text that will be obvious to readers of that text which is the definition of a grapheme cluster.
No, I insist there is not a single "correct answer," even if a reader has perfect knowledge of the language(s) involved. Now remember, this is already moving the goalposts, since it was claimed that a human needed "no knowledge" to get to this allegedly "correct answer."
You already admit that people who don't speak Arabic will have trouble finding the "grapheme clusters," but even two people who speak Arabic may do your clustering or not, depending on some implicit feeling of "the right way to do it" vs taking the question literally and pasting the smallest highlight-able selection of the string in reverse at a time.
Anyway, take a string like this: "here is some Arabic text: And back to English"
Whether you discard the ordering mark[0], keep them, or inverse them is an implementation decision that already produces three completely different strings. Unless we want to write a rulebook for the right way to reverse a string, it remains an impossibility to declare anything the correct answer, and because there is no reason to reverse such a string outside of contrived interview questions and ivory tower debates, it is also meaningless.
[0]: https://en.m.wikipedia.org/wiki/Right-to-left_mark https://en.m.wikipedia.org/wiki/Left-to-right_mark
Earlier quoted context omitted.
Even the most basic ASCII string is still a data structure. Is it a PASCAL string (length byte followed by data) or a C string (arbitrary run of bytes terminated by a null character)?
You qualified "string" with "ASCII", and also tacitly admitted you still need more information than the octets themselves--the length. Of course, various programming languages have primitives and concepts which they may label "string". But you still need to specify that context , drawing in the additional specification those languages provide. Plus, traditionally and in practice, such concepts often serve the functio…
Shifting definitions to yours, I agree.