Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What every software developer must know about Unicode in 2023

tonsky.me

Post reply on HN
Print view View original

531–540 of 572 posts

Re: What every software developer must know about Unicode in 2023

#531

Post by kzrdude »

Findecanor wrote: post #350
mostlylurks wrote: post #294

Earlier quoted context omitted.

Do the danes not have the mechanism that is found on Finnish keyboard layouts, where pressing AltGr+Ö yields Ø and AltGr+Ä yields Æ, except in reverse?

Those mappings are not universal. They are present under Linux but not on MS-Windows. I don't know about Mac, but the layout has in the past been slightly different there from Windows also.

They are there on mac too, use option-Ä to get the Æ or the other way around. What's more, it has worked like that since system 7.x times or so, it's just a good idea.

Re: What every software developer must know about Unicode in 2023

#532

Post by nerdbert »

jcranmer wrote: post #142

There's one part of this document that I would push extremely hard against, and that's the notion that "extended grapheme clusters" are the one true, right way to think of characters in Unicode, and therefore any language that views the length in any other way is doing it wrong. The truth of the matter is that there are several different definitions of "character", depending on what you want to use it for. An extende…

> for this reason, String iteration should be based on codepoints

This leads you to the problem where you'll get different results iterating over

n a ï v e

vs

n a ̈ i v e

And I can't see how that's ever going to be a useful outcome.

If you normalize everything first, then you can sidestep this to some degree, but then in effect your normalization has turned codepoint iteration into grapheme iteration for most common Latin-script text characters.

Re: What every software developer must know about Unicode in 2023

#533

Post by davidmurdoch »

mike_hearn wrote: post #519
kalleboo wrote: post #485

Earlier quoted context omitted.

There are basically 2 places where programmers mostly want the "length" of a string: 1. To save storage space or avoid pathological input, they want to limit the "length" of text input fields. E.g., not allow a name to be 4 KB long 2. To fit something on screen Developers mostly used to western languages can approximate both with "number of letters", but the correct answers are For 1. Limit to bytes to avoid people b…

Presizing buffers, initializing for loop counts ...

Unrelated to this post, but you suggested (https://news.ycombinator.com/item?id=37381390) I use your company hydraulic.dev for my electron build. I ultimately gave up on it, then someone from node-gyp comment on an issue I opened about it, and provided the solution:

https://github.com/electron-userland/electron-builder/issues...

Just wanted to let you know in case it's a gotcha you might not be aware of that might help you out if you run into similar problems with some of your customer builds.

Re: What every software developer must know about Unicode in 2023

#534

Post by nerdbert »

gumby wrote: post #263
WorldMaker wrote: post #255

Earlier quoted context omitted.

> So the fi ligature was in a legacy encoding system and thus must be in Unicode. Most of the pre-composed latin ligatures are generally from EBCDIC codepages. People in the ancient Mainframe era wanted nice typesetting too, but computer fonts with ligature support were a much later invention. You can see fi and several others directly in EBCDIC code page 361: https://en.wikibooks.org/wiki/Character_Encodings/Code_Tabl…

Thanks. Some alphabets have precomposed ligatures that aren't really letters, like old German alphabets with tz, ch, ss (I only know how to type the last one, ß, because the others have died out over the last hundred years). Actually in German (at least) ä, ö and ü really are actually ligatures for ae, oe, and ue -- the scribes started to write the E's on their sides above the base letters, and over time the superscr…

> old German alphabets with tz, ch, ss (I only know how to type the last one, ß, because the others have died out over the last hundred years)

They still ꜩ on some German street signs. I can't find ch in Unicode though (could just be my old eyes).

Re: What every software developer must know about Unicode in 2023

#535

Post by slipperlobster »

throwaway290 wrote: post #464
toastercat wrote: post #419

Earlier quoted context omitted.

I'm having a hard time reconciling "he knows what he is doing" with him making his site practically unusable without a reader mode, which by the way, not every browser supports (especially on mobile).

Don't even think of switching on the dark (night) mode with that attitude! :D I really enjoyed the tongue in cheek design. I think every modern browser either allows you to turn on reader mode (especially on mobile) or just turn off CSS. This particular article works excellently even in w3m.

Fine, sure. Cute - turn on reader mode. Now the images that are supposed to be sitting over yellow background are dark gray images over a slightly less dark gray background.

The decision to design a serious (read: not-tongue-in-cheek) topic with these "quirky" tricks sucks, JMHO.

Re: What every software developer must know about Unicode in 2023

#536

Post by daveslash »

gpvos wrote: post #331
tonsky wrote: post #185

Earlier quoted context omitted.

Hi, author here. In case you really want to know: no, it’s custom-made and works exactly as intended. There are two main reasons: 1. Fun. Modern internet is boring, most blog posts are just black text on white background. Hard to remember where you read what. And you can’t really have fun without breaking some expectations. 2. Sense of community. Internet is a lonely place, and I don’t necessarily like that. I like t…

2. I only understood that it was actual other people's mouse cursors when I read that here. So it didn't really engender a sense of community, although after some time I did think you are very good at modelling actual human mouse movements. Now that I know it, it's pretty neat though.

Same. I didn't realize it was other people until I came here. Then... I went back to the page and had fun trying to follow other people's cursors.

Re: What every software developer must know about Unicode in 2023

#537

Post by NikolaNovak »

kergonath wrote: post #443
NikolaNovak wrote: post #375

Earlier quoted context omitted.

I am assuming that means "on French keyboard", not "in French". I have a usa keyboard and live in Canada...Every now and then it thinks I'm typing French and keyboard indeed behaves in a way that some vowel plus some quotation mark indeed gives me some other character (that I don't need :)

That would feel very strange to me. The Canadian layout probably behaves differently, but à is a modifier letter, not two letters. I’d expect backspace to remove the whole letter, including the accent.

Oh, it's strange to me too! I curse every time I accidentally hit that setting and my computer seems possessed :)

It appears there's a few keyboard layouts and language options:

French (Canada) - Canadian French

French (Canada) - Canadian Multilingual Standard

French (France)

etc

And then I think it intersects with what keyboard you actually have.

Some of these layouts maybe are designed to enable you to type in French characters on a non-French keyboard? Not sure.

Re: What every software developer must know about Unicode in 2023

#538

Post by FoeNyx »

gumby wrote: post #408
cyxxon wrote: post #300

Earlier quoted context omitted.

That's sounds a bit false to me. The Umlaute (ä,ö, ü) and the "eszett" ß are actually part of the German alphabet[1]. Also it is kinda weird to describe them as ligatures of the original letters and the diaeresis, because while this is what they started out as a long time ago, they are just their own letters now (as opposed to "real" stylistic ligatures like combining fi into one glyph). The advice your kid was told…

C'mon that page is highly technical, really just listing the letters or glyphs used for forming printed text. In reality, if you walk into any first grade classroom you see pictures of the letters A-Z with pictures (Apfel, Bär, uwv) and then after the end maybe around the corner, what, Öl? I can't even remember. When the kids recite the letters they don't recite äöüß. TBH I really only remember this because when kidd…

> When the kids recite the letters they don't recite äöüß.

On a somehow related side note, I read that "&", which is derived from a ligature of the Latin conjunction "Et" (meaning "and"), was named "ampersand" in English as a mondegreen for "and per se and" as it used to be placed at the end of the English alphabet recitation.

Re: What every software developer must know about Unicode in 2023

#539

Post by [deleted] »

toastercat wrote: post #119
davidham wrote: post #10

Is it just me, or is anyone else seeing what looks like the mouse pointer of everyone else reading the page, like 1,000 little ants on the screen

Anytime tonsky's site gets posted here, I'm reminded by how awful it is, which is ironic given his UI/UX background. The site's lightmode is a blinding saturated yellow, and if you switch into darkmode, it's an even less readable "cute" flashlight js trick. I don't know why he thought this was a good idea. Thank god for Firefox reader mode.

[deleted]

Re: What every software developer must know about Unicode in 2023

#540

Post by grumpyprole »

archgoon wrote: post #475
grumpyprole wrote: post #457

Earlier quoted context omitted.

> And for this reason, String iteration should be based on codepoints Why not offer both and be clear about it? Rather than just "length", why not call them code points? The Python docs for "len" which can be called on a unicode string say "Return the length (the number of items) of an object.". It doesn't look like a clear and easy to use API to me.

If you insist that `len` shouldn't be defined on strings, and the default iterator should be undefined in python then: for c in "Hello": pass should throw an exception. Also if word[0] == 'H': pass should throw an exception. This would have been an extremely controversial suggestion when python3 came out to say the least. Codepoints is a natural way of defining unicode strings in python, and it mostly works the way y…

I would suggest that len works as the article suggests; and "Hello".codepoints gives the behaviour you want.
Post reply on HN