91–100 of 166 posts
> The current largest codepoint? Why that would be a cheese wedge at U+1F9C0. How did we ever communicate before this? Sounds cute, but inaccurate. If we count the last two planes that are reserved for private use (aka, applications/users can use them for whatever domain problems they like), that would be U+10FFFD. If we count the variation selector codepoints (used for things like changing skin tone, or the look of…
Earlier quoted context omitted.
They're not supposed to render, it's purely for describing text. As are the interlinear ruby annotations.
I'm curious. Are interlinear ruby annotation codepoints actually used for their intended purpose anywhere? And what are you supposed to do when you encounter one? I know that they appear on Unicode's shitlist in [UTR#20], a proposed tech report that contained a table of codepoints that should not be used in text meant for public consumption. UTR#20 suggested things you could do when you encounter these codepoints, bu…
Yes
> And what are you supposed to do when you encounter one?
Nothing. Don't display them, or display some symbolic representation. You probably shouldn't make ruby happen here; if your text is intended to be rendered correctly use a markup language.
----------
Unicode is ultimately a system for describing text. Not all stored text is intended to be rendered. This is why it has things like lacuna characters and other things.
So when you come across some text using ruby, or some text with an unencodable glyph, what do you do? You use ruby annotations or IDS respectively. It lets you preserve the nature of the text without losing info.
(Ruby is inside unicode instead of being completely deferred to markup since it is used often enough in Japanese text, especially whenever an irregular (not out of the "common" list) kanji is used. You're supposed to use markup if you actually want it rendered, but if you just wanted to store the text of a manuscript you can use ruby annotations)
func main() {
shit := "\U0001f4a9"
fmt.Printf("len of %s is %d\n", shit, utf8.RuneCountInString(shit))
}
Though I can't say that this is all that intuitive either...
The Unicode standard describes in Annex 29 [1] how to properly split strings into grapheme clusters. And here [2] is a JavaScript implementation. This is a solved problem. [1] http://www.unicode.org/reports/tr29/ [2] https://github.com/orling/grapheme-splitter
> This is a solved problem. Not … really. Yes, we "know" the solution, but the terrible APIs that compose so many language's standard string type goads the programmer into choosing the wrong method or type. JavaScript has — to an extent — the excuse of age. But the language still really (to my knowledge) lacks an effective way to deal with text that doesn't involve dragging in third-party libraries. You are not a hig…
So C++, Lisp, Java, Python, Ruby, PHP, and JS are not high-level languages.
HN teaches me something new every day.
There are multiple ways of counting "length" of a string. Number of UTF-8 bytes, number of UTF-16 code units, number of codepoints, number of grapheme clusters. These are all distinct yet valid concepts of "length." For the purpose of allocating buffers, I can see the obvious use in knowing number of bytes, UTF-16 code units, or the number of codepoints. I also see the use in being able to iterate through grapheme cl…
> Perhaps someone can shed light on a compelling use case for knowing the number of grapheme clusters in a particular string, because I haven't been able to think of one. If you have a limit on the length of a field, it helps to tell the user what it is in a way they understand. For non-technical users, bytes (and the embedded issue of encoding) and code points are both pretty esoteric, but number of symbols is less…
So it can fit in a database, i.e. with a certain number of bytes?
Earlier quoted context omitted.
> This is a solved problem. Not … really. Yes, we "know" the solution, but the terrible APIs that compose so many language's standard string type goads the programmer into choosing the wrong method or type. JavaScript has — to an extent — the excuse of age. But the language still really (to my knowledge) lacks an effective way to deal with text that doesn't involve dragging in third-party libraries. You are not a hig…
> You are not a high-level language if your standard library struggles with Unicode So C++, Lisp, Java, Python, Ruby, PHP, and JS are not high-level languages. HN teaches me something new every day.
How can that entire article never mention the term UTF-16?
Earlier quoted context omitted.
> You are not a high-level language if your standard library struggles with Unicode So C++, Lisp, Java, Python, Ruby, PHP, and JS are not high-level languages. HN teaches me something new every day.
Python 3 does pretty good.
Before emoji, fonts and colors were independent. Combining the two creates a mess. Try using emoji in an editor with syntax coloring. We got into this because some people thought that single-color emoji were racist.[1] So now there are five skin tone options. The no-option case is usually rendered as bright yellow, which comes from the old AOL client. They got it from the happy-face icon of the 1970s. Here's the curr…
Why test this at all? It's not as if a website should ever need to render a user's password as text. Is there another use case for excluding this "weird stuff" that I'm not seeing?