111–120 of 572 posts
Earlier quoted context omitted.
To save other people the google: Tengwar is probably not in unicode because it is a fictional script from a book.
I would wonder how many people are here who have never seen Tengwar. I would bet that's a minuscule minority.
> “I know, I’ll use a library to do strlen()!” — nobody, ever. The standard library provided by languages like C, C++ is a library. Features like character strings are present and it's a totally reasonable expectation for the length to give you the cluster count.
Earlier quoted context omitted.
Be the change you want to see in the world. If we're going to make huge breaking changes, might as well do it sooner rather than later.
With something as large as a end user language format for input, this is a change we ourselves cannot make, just as using another calendar for dates. Just because I want to use the year 2002023 calendar with 29.5 days per month, doesn't make it useful to others or myself really.
The problems with Unicode are mostly to do with internal inconsistencies and churn, problems that usually only affect programmers.
1. Different ways to encode the same visually indistinguishable set of characters as code points leading to normal forms, text that compares unequal even when it appears to be identical, the disastrous "grapheme clusters" concept and so on.
2. Many different ways to encode the same sequence of code points as bytes. Not only UTF-32/16/8 but also curiousities like "modified UTF-8".
3. Emoji. A fractal of disasters:
3.a. Updates frequently. Neither Unicode nor software in general was built on the assumption that something as basic as the alphabet changes every year. If you send someone an emoji, can their device draw it? Who knows! In practice this means messaging apps can't rely on the OS system fonts or text handling libraries anymore which is a drastic regression in basic functionality.
3.b. (Ab)uses composition so much it's practically a small programming language, e.g. flags are composed of the two letter country code spelled using special characters. People are represented as as generic person plus skin color patch, families are represented using composed individual people etc.
3.c. Meaning of a character is theoretically specified but can subtly depend on the font used, e.g. people use a fruit emoji in visual puns because of how it looks specifically on Apple devices, so a "sentence" can make no sense if it's rendered with a different font.
3.d. Unbounded in scope. There's no reason the Unicode committee won't just keep adding new pictograms forever.
3.e. Encoded beyond the BMP which in theory every correct program should handle but in practice some don't because nobody except a few academics used characters beyond it much until emoji came along.
3.f. Disagreement over single vs double width chars, can only know this via hard-coded tables, matters for terminals and code editors.
Some of these can potentially be cleaned up outside of the Unicode consortium in backwards compatible ways. You could have a programming language that automatically normalized strings to fully composed form when deserializing from bytes, and then automatically folded semantically identical code points together (this would be a small efficiency win for some languages too). You could campaign to build a consensus around a specific normal form, like how UTF-8 gained consensus as a transfer encoding. You could also define a fork of Unicode (using private use areas?) that allocates a single code point to the characters that are unnecessarily using composition today but don't yet have one and then just subset out the concept of composition entirely.
Emoji are a big problem. It's tempting to say that these should not be encoded as characters at all. Instead there could be a set of code points that define bounds that contain a tiny binary subset of SVG, enough to recreate the Apple pixel art somewhat closely. Emoji would always be transmitted as inlined vector art. Text rendering libraries would call out to a little renderer for each encoded glyph, using a fast fingerprinting algorithm to deduplicate the bytes to an internal notion of a character. To avoid wire bloat, text can simply be compressed with a pre-agreed zstd or Brotli dictionary that contains whatever images happen to be popular in the wild. At a stroke this would avoid backwards compat problems with new emoji, enabling programs working with text to be upgraded once and then never again, eliminate all the ridiculous political committee bike-shedding over what gets added, let apps go back to using system text support and get rid of the bajillion edge cases that emoji have spewed all over the infrastructure.
> what to you think "ẇ͓̞͒͟͡ǫ̠̠̉̏͠͡ͅr̬̺͚̍͛̔͒͢d̠͎̗̳͇͆̋̊͂͐".length should be? This is a nice example of the kind of thing we need to think about when defining a measure of length for Unicode strings.
The more interesting question is whether the Unicode rules actually give that answer.
EDIT: Just checked it using the first online tool [1] that came up and it indeed says four. So all is good.
[1] https://onlinetools.com/unicode/extract-unicode-graphemes
What on EARTH is that mouse cursor thing all about? Why would you even bother writing this, then making it impossible to read properly?
Earlier quoted context omitted.
> Why would because it shows that they don't understand important design aspects while it doesn't really show off their technical skills because it could be some plugin or copy pasted code, only someone who looks at the code would know better. But if someone care enough about you to look at your code you don't need to show of that skill on you normal web-site and can have some separate tech demo. > okay if we let peo…
Not every website, even technical ones, need to have an eye towards professional advancement. Sometimes they're just for fun. I welcome it, as it's a thing that gets more rare on the web as time goes by.
This is quite a good write up. An answer to one of the author's questions: > Why does the fi ligature even have its own code point? No idea. On of the principles of Unicode is round trip compatibility. That is you should be able to read in a file encoded with some obsolete coding system and write it out again properly. Maybe frob it a bit with your unicode-based tools first. This is a good principle, though less usefu…
Is it just me, or is anyone else seeing what looks like the mouse pointer of everyone else reading the page, like 1,000 little ants on the screen