141–150 of 166 posts
Earlier quoted context omitted.
This is from a couple of weeks ago, there's a few things broken still, but what languages do have full support out of the box? http://blog.honeybadger.io/ruby-s-unicode-support/
Go
Earlier quoted context omitted.
> You are not a high-level language if your standard library struggles with Unicode So C++, Lisp, Java, Python, Ruby, PHP, and JS are not high-level languages. HN teaches me something new every day.
What would you say Java is missing? Sure, it does have the "oops, we implemented Unicode when they said we only needed 16 bits problem" but unlike, say, JS, it actually handles astral plane characters well (e.g., the regex implementation actually says that . matches an astral plane code point rather than half of one). It does have all the major Unicode annexes--normalization (java.text.Normalizer), grapheme clusters…
Earlier quoted context omitted.
That is certainly true if you only look at the word sizes at different layers. But any implementation will at least logically start with a sequence of bytes, then turn them into code units according to the encoding scheme, group code units into minimal well-formed code unit subsequences according to the encoding form, and finally turn them into code points. While different layers may use words of the same size, there…
> But any implementation will at least logically start with a sequence of bytes, then turn them into code units according to the encoding scheme, group code units into minimal well-formed code unit subsequences according to the encoding form, and finally turn them into code points. Not at all. I've never seen people using UTF-8 deal with a code unit stage. They parse directly from bytes to code points. > While for ex…
Well, that's probably because in UTF-8 code unit is byte :)
Quoting https://en.wikipedia.org/wiki/UTF-8:
> The encoding is variable-length and uses 8-bit code units.
By definition, code unit is a bit sequence of a fixed size which can form code points. In UTF-8 you form code points using 8-bit bytes, therefore in UTF-8 code unit is byte. In UTF-16 it is a sequence of two bytes. In UTF-32 it is a sequence of four bytes.
Earlier quoted context omitted.
Some limits are technical (and in that case the hard limit is often bytes, but sometimes code units or code points, or broken if you told MySQL utf8 instead of bytes or utf8mb4), but in many cases, the limits are for aesthetic purposes: a post title or a username often is often required to be fairly short to look nice; in an ascii or latin1 world, those limits are usually expressed in terms of characters, but graphem…
"Your username must be 1-4cm when printed with 12pt Times New Roman." I kind of like the idea of minimum length in cm as a password requirement.
or, "Your username must not take more than 0.001ml of ink when printed at 12pt"
Earlier quoted context omitted.
Backspace is typically not one grapheme at a time, though it is for emoji. For scripts such as Arabic, it typically deletes ḥarakāt when they are composed on top of a base character. For a bit more discussion of how I hope to handle this in xi-editor, as well as links to the logic in Android, see https://github.com/google/xi-editor/issues/159
clarification: It is for some emoji, e.g. backspace on a family emoji will eliminate family members one by one. (on most browsers and platforms afaict). But flag emoji will be deleted as a group. IIRC handling of multicodepoint profession emoji is inconsistent.
Unexpectedly sinister.
Earlier quoted context omitted.
That is certainly true if you only look at the word sizes at different layers. But any implementation will at least logically start with a sequence of bytes, then turn them into code units according to the encoding scheme, group code units into minimal well-formed code unit subsequences according to the encoding form, and finally turn them into code points. While different layers may use words of the same size, there…
> But any implementation will at least logically start with a sequence of bytes, then turn them into code units according to the encoding scheme, group code units into minimal well-formed code unit subsequences according to the encoding form, and finally turn them into code points. Not at all. I've never seen people using UTF-8 deal with a code unit stage. They parse directly from bytes to code points. > While for ex…
Surrogate codepoints are indeed valid codepoints. It's just that valid UTF-8 is not allowed to encode surrogate codepoints, so the space of codepoints supported by UTF-8 is actually a subset of all Unicode codepoints. This subset is known as the set of Unicode scalar values. ("All codepoints except for surrogates.")
Earlier quoted context omitted.
Why do you have a limit on the length of a field? So it can fit in a database, i.e. with a certain number of bytes?
Without a limit on password length, an attacker can DOS you by forcing you to run your KDF on gigabyte-sized strings.
Oh, no. That doesn't make sense. You need to limit by Giga grapheme strings.
Earlier quoted context omitted.
> You are not a high-level language if your standard library struggles with Unicode So C++, Lisp, Java, Python, Ruby, PHP, and JS are not high-level languages. HN teaches me something new every day.
How does Ruby struggle with unicode?
> "\xAA".split ''
That works on a platform where my platform is UTF-32, but not one where it is UTF-8.
Earlier quoted context omitted.
> But any implementation will at least logically start with a sequence of bytes, then turn them into code units according to the encoding scheme, group code units into minimal well-formed code unit subsequences according to the encoding form, and finally turn them into code points. Not at all. I've never seen people using UTF-8 deal with a code unit stage. They parse directly from bytes to code points. > While for ex…
> I've never seen people using UTF-8 deal with a code unit stage. They parse directly from bytes to code points. Well, that's probably because in UTF-8 code unit is byte :) Quoting https://en.wikipedia.org/wiki/UTF-8 : > The encoding is variable-length and uses 8-bit code units. By definition, code unit is a bit sequence of a fixed size which can form code points. In UTF-8 you form code points using 8-bit bytes, ther…
Code units may 'exist' on all three through the fiat of their definition, but they only have a visible function and require you to process an additional layer in UTF-16.