111–120 of 247 posts
(1) https://en.wikipedia.org/wiki/Emoji#Joining
edit: forgot HN doesn't render emojis. Better read it directly on Wikipedia i guess.
If you think the Unicode flag emoji take a lot of bytes, then consider the family emoji! ( https://unicode.org/emoji/charts/full-emoji-list.html#family ) I'm in the process of designing a scripting language and implementing it in C++. I plan to put together a YouTube series about it. (Doesn't everyone want to see Bison and Flex mixed with proper unit tests and C++20 code?) Due to my future intended use case, I needed…
Handling unicode can be fine, depending on what you're doing. The hard parts are: - Counting, rendering and collapsing grapheme clusters (like the flag emoji) - Converting between legacy encodings (shiftjis, ko8, etc) and UTF-8 / UTF-16 - Canonicalization If all you need is to deal with utf8 byte buffers, you don't need all that stuff. And your code can stay simple, small and fast. IIRC the rust standard library does…
Rust's core library gives char methods such as is_numeric which asks whether this Unicode codepoint is in one of Unicode's numeric classes such as the letter-like-numerics and various digits. (Rust does provide char with is_ascii_digit and is_ascii_hexdigit if that's all you actually cared about)
So yes, the Rust standard library is carrying around the entire Unicode standard class rule list among other things, of course Rust's library isn't built to a vast binary, so if you never use these features your binary doesn't get that code.
This is a bit of a hobby horse, but imagine if every time you read an article in English on your phone some of the letters were replaced with "equivalent" Greek or Cyrillic one and you can get an idea of the annoyance. Yeah, you can still read it with a bit of thought, but who wants to read that way?
Earlier quoted context omitted.
They do say no though. Frequently too. The problem with Unicode is simply that it’s trying to solve a very hard problem.
Exactly this. Humans have incredibly complicated writing systems, and all Unicode wants to do is encode them all. Keep in mind that the trivial toy system we're more familiar with, ASCII, already has some pretty strange features because even to half-arse one human writing system they needed those features. Case is totally wild, it only applies to like 5% of the symbols in ASCII, but in the process it means they each…
Not only that, there isn't even agreement about what's correct all the time!
>it doesn't understand how to spell a bunch of common English words like naïve or café, pretty disappointing.
A perfect example of this, since I would argue English doesn't have any diacritics at all. So the use of café is code switching. :)
Earlier quoted context omitted.
> the fact that there exists a correct answer for every piece of text that will be obvious to readers of that text which is the definition of a grapheme cluster. No, I insist there is not a single "correct answer," even if a reader has perfect knowledge of the language(s) involved. Now remember, this is already moving the goalposts, since it was claimed that a human needed "no knowledge" to get to this allegedly "cor…
You added the requirement that it be a single correct answer. I just asserted that there existed a correct answer. You're being woefully pedantic -- a human who can read the text presented to them but no knowledge of unicode was my intended meaning. Grapheme clusters are language dependent and chosen for readers of languages that use the characters involved. There's no implicit feeling, this is what the standards bod…
It is impossible to "correctly reverse a string" because "reverse a string" is not well defined. We explored many different potential definitions of it, to show that there is no meaningful singular answer.
> You added the requirement that it be a single correct answer.
Your original post says "they could produce the correct string reversal"?
Earlier quoted context omitted.
Exactly this. Humans have incredibly complicated writing systems, and all Unicode wants to do is encode them all. Keep in mind that the trivial toy system we're more familiar with, ASCII, already has some pretty strange features because even to half-arse one human writing system they needed those features. Case is totally wild, it only applies to like 5% of the symbols in ASCII, but in the process it means they each…
>Humans have incredibly complicated writing systems Not only that, there isn't even agreement about what's correct all the time! >it doesn't understand how to spell a bunch of common English words like naïve or café, pretty disappointing. A perfect example of this, since I would argue English doesn't have any diacritics at all. So the use of café is code switching. :)
I'm trying to find the best combination of UTS-46, UTS-51, UTS-39, and prior work on IDN resolution w/r/t confusables: https://adraffy.github.io/ens-normalize.js/test/report-confu...
Personally, I found the Unicode spec very messy. Critical information is all over the place. You can see the direct effect of this when you compare various packages across different languages and discover that every library disagrees in multiple places. Even JS String.normalize() isn't consistent in the latest version of most browsers: https://adraffy.github.io/ens-normalize.js/test/report-nf.ht... (fails in Chrome, Safari)
The major difference between ENS and DNS is emoji are front and center. ENS resolves by computing a hash of a name in a canonicalized form. Since resolution must happen decentralized, simply punting to punycode and relying custom logic for Unicode-handling isn't possible. On-chain records are 1:1, so there's no fuzzy matching either. Additionally, ENS is actively registering names, so any improvement to the system must preserve as many names as possible.
At the moment, I'm attempting to improve upon the confusables in the Common/Greek/Latin/Cyrillic scripts, and will combine these new grouping with the mixed-script limitations similar to IDN handling in Chromium.
Interactive Demo: https://adraffy.github.io/ens-normalize.js/test/resolver.htm...
Also this emoji report is pretty cool: https://adraffy.github.io/ens-normalize.js/test/report-emoji...
There's another word that comes to mind when thinking about those two: metastasis.