Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Charset="WTF-8"

wtf-8.xn--stpie-k0a81a.com

Post reply on HN
Print view View original

381–390 of 463 posts

Re: Charset="WTF-8"

#381

Post by jrochkind1 »

kristopolous wrote: post #197
simonh wrote: post #179

Earlier quoted context omitted.

No, no, no, no, no… So then we’d get ‘the same’ character with potentially infinite different encodings. Lovely. Unicode is a coding system, not a glyph system or font.

Fonts are already in there and proto-glyphs are too as generalized dicritics. There's also a large variety of generic shapes, lines, arrows, circles and boxes in both filled and unfilled varieties. Lines even have different weights. The absurdity of a custom alphabet can already be partially actualized. Formalism is merely the final step This conversation was had 20 years ago and your (and my) position lost. Might as…

The diacritics are there because they were in legacy encodings, and it was decided at some point that encodings should be round-trippable between legacy encodings and unicode.

The fact that hardly anyone cares any longer are about going to any legacy non-unicode encoding is, of course, a testament to the success of unicode, a success that required not only technical excellence but figuring out what would actually work for people to actually adopt practically. It worked. It's adopted.

I have no idea if the diacritics choice was the right one or not, but I guarantee if it had been made differently people would be complaining about how things aren't round-trippable to unicode encoding and back from some common legacy encoding, and that's part of it's problem.

I think some combining diacritics are also necessary for some non-latin scripts, where it is (or was) infeasible to have a codepoint for every possible combination.

The choices in Unicode are not random. The fact that it has become universal (so many attempts at standards have not) is a pretty good testatement to it's success at balancing a bunch of competing values and goals.

Re: Charset="WTF-8"

#382

Post by hobs »

Muromec wrote: post #369
hobs wrote: post #368

Earlier quoted context omitted.

No, it allows an exact representation of your name, it doesn't do anything to your life. If you dont like your name, either change it or go complain to your parents. They might tell you that you cultural reference point is more important than some person being able to read your name off of a computer screen. If you want to store a phonetic name for the destination speaker that's not a bad idea, but a name is a name i…

But it does affect my life in a way you refuse to understand. That's the problem -- there isn't a true canonical representation of a name (any name really) that fits all practical purposes. Storing a bag of bytes to display back to user is the easiest of practical purposes and suggesting the practice that solve that is worse than rejecting Stępień, it's refusal to understand complexities, that leads to eventually doi…

There's no such thing as a data structure that fits "all practical purposes" that is correct.

There's no wrong thing - this is the best representation we can make given the system of record for the person's name.

They are definitely mutable, context dependent, and effectively data you cannot make assumptions about because of all those things.

If you want to do more than that you need a highly constrained use case, and its going to fail for "all practical purposes".

Re: Charset="WTF-8"

#383

Post by throw_a_grenade »

Muromec wrote: post #110
Diti wrote: post #9

Earlier quoted context omitted.

What are “invalid names” in this context? Because, depending on the country the person was born in, a name can be literally anything, so I’m not sure what an invalid name looks like (unless you allow an `eval` of sorts).

The non-joke answer for Europe is extened Latin, dashes, spaces and apostrophe sign, separated into two (or three) distinct ordered fields. Just because it's written in a different script originally, doesn't mean it will printed only with that on your id in the country of residence or travel document issued at home. My name isn't written in Latin characters and it's fine. I know you can't even try to pronounce them,…

Non-joke answer for Europe is at least Latin, Greek or Cyrillic (български is already one of the official EU languages!). No reason to treat them differently, just don't allow for mixing them so you won't get homoglyphs. EURid (.eu-NIC) gets it mostly right I believe.

Re: Charset="WTF-8"

#384

Post by hobs »

Muromec wrote: post #378
soco wrote: post #371

Earlier quoted context omitted.

Then where is the problem? If the support can read Polish they will pronounce your name properly, if they're from India they will mess it up, why should we have different expectations? Nobody will identify you by name anyway, they will ask how to call you (chatbots do this already) and then use for proper identification all kind of ids and pins and whatnot. So we are talking here about a complexity that nobody actual…

>if they're from India they will mess it up Or not able to read at all. >Then where is the problem? Since you don't indicate for what purpose my name is stored, which may actually be display only, any of the following can happen: A name as entered in your system is compared to a name entered in a different system or when you interface (maybe indirectly and unknowingly) with a system using different constrains or a di…

Managing the canonical representation of your name in my system is one of the few things you are responsible for.

The number of times I have had people ask me to customize name rendering, capitalize things, trying to build phonetic maps, all of these things to avoid data entry or confusion and all they do is prove out that you can't have a general solution to human names, you can hit a big percentage in a cultural context, but there's always exceptions and edge cases to the problem we're solving which can be described as "please tell me your name when you call or whatever so I can pronounce it right"

Re: Charset="WTF-8"

#385

Post by poizan42 »

lxgr wrote: post #84
poizan42 wrote: post #10

I have an 'æ' in my middle name (formally secondary first name because history reasons). Usually I just don't use it, but it's always funny when a payment form instructs me to write my full name exactly as written on my credit card, and then goes on to tell me my name is invalid.

Did you actually get banks to print that on your credit card? I’m impressed, most I know struggle with any kind of non-[A-Z]!

In Denmark? I don't think they legally are allowed not to print my legal name.

Would be weird if our banks couldn't handle our own alphabet.

Re: Charset="WTF-8"

#386

Post by wruza »

n2d4 wrote: post #351
wruza wrote: post #299

Earlier quoted context omitted.

How many direction marks can we see in this hidden text?

None — it's tag characters instead, which are used to represent emojis. But there's no difference! Either you can smuggle text in Unicode, or you can't. It's quite binary, and you don't gain advantages from having "fewer ways" to smuggle text, but you certainly gain advantages from having emojis in your characterset.

This yesman attitude is honestly unnerving.

We make things worse, because they were worse! Sort of not, but were anyway. That’s an advantage, nothing to see here!

Instead of praising the advantages of going insane, let us better make (or at least strive for) a charset that makes subj work in practice, not on paper.

Re: Charset="WTF-8"

#387

Post by ajsnigrutin »

jeroenhd wrote: post #185
ajsnigrutin wrote: post #114

Earlier quoted context omitted.

Yeah, that'll work great.. https://en.wikipedia.org/wiki/%C4%8Celje echo "Čelje" | uconv -f "UTF-8" -t "UTF-8" -x "Latin-ASCII" > "Celje" https://en.wikipedia.org/wiki/Celje (i mean... we do have postal numbers just for problems like this, but both Štefan and Stefan are not-so-uncommon male names over here, so are Jozef and Jožef, etc.)

If you're dealing with a bad API that only takes ASCII, "Celje" is usually better than "ÄŒelje" or "蒌elje". If you have control over the encoding on the input side and on the output side, you should just use UTF-8 or something comparable. If you don't, you have to try to get something useful on the output side.

This depends.

Everyone over here would know that "ÄŒelje" (?elje) is either čelje, šelje or želje. Maybe even đelje or ćelje if it's a name or something else. So, special attention would be taken to 'decypher' what was meant here.

But if you see "Celje", you assume it's actually Celje (a much larger city than Čelje) and not one of those variants above. And noone will bother with figuring out if part of a letter is missing, it'll just get sent to Celje.

Re: Charset="WTF-8"

#388

Post by int_19h »

shaky-carrousel wrote: post #367
arkh wrote: post #309

Earlier quoted context omitted.

You sanitize at the frontier of what your code controls. Sending data to a database: parametrized queries to sanitize as it is leaving your control. Sending to display to the user: sanitized for a browser Sending to an API: sanitize for whatever rules the API has Sending to a legacy system: sanitize for it Writing a file to the system: sanitize the path The common point is you don't sanitize before you have to send i…

Be liberal in what you accept, and conservative in what you send.

Please don't. This is how standards die.

https://datatracker.ietf.org/doc/html/rfc9413

Re: Charset="WTF-8"

#389

Post by int_19h »

tugu77 wrote: post #292
ddulaney wrote: post #241

Earlier quoted context omitted.

There's at least one major exception to this: Unicode normalization. It's possible for the same logical character to have two different sets of code points (for example, a-with-umlaut as a single character, vs a followed by umlaut combining diacritic). Related, distinguishing between the "a" character in Latin, Greek, Cyrillic, and the handful of other times it shows up throughout Unicode. This comes up in at least 3…

Type system ftw? As long as it's a blob (unnormalized), it should have a blob type which can do very little besides storing and retrieving, perhaps printing. Only the normalized version should be even comparable.

Why wouldn't blobs be comparable? A blob is just a byte array, and those have fairly natural equality semantics. They're wrong for Unicode strings, sure, but this is akin to complaining about string "1" not being equal to "01".

Re: Charset="WTF-8"

#390

Post by int_19h »

swiftcoder wrote: post #296
dgfitz wrote: post #293

Earlier quoted context omitted.

In this specific case, it seems like your concerns are a hypothetical, no?

Not really, no. A lot of us only really have to deal with English-adjacent input (i.e. European languages that share the majority of character forms with English, or cultures that explicitly Anglicise their names when dealing with English folks). As soon as you have to deal with users with a radically different alphabet/input-method, the wheels tend to come off. Can your CSR reps pronounce names written in Chinese lo…

For one thing, this concern applies equally to names written entirely in Latin script. Can your CSR reps correctly pronounce a French name? How about Polish? Hungarian?

In any case, the proper way to handle this is to store the name as originally written, and have the app that CSRs use provide a phonetic transcription. Coincidentally, this kind of stuff is something that LLMs are very good at already (but I bet you could make it much more efficient by training a dedicated model for the task).

Post reply on HN