Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Charset="WTF-8"

wtf-8.xn--stpie-k0a81a.com

Post reply on HN
Print view View original

421–430 of 463 posts

Re: Charset="WTF-8"

#421

Post by account42 »

dcow wrote: post #323
afiori wrote: post #310

Earlier quoted context omitted.

Forbidding users to use your service to propagate "litte bobby tables" pseudo-pranks is likely a good choice. The choice is different if like most apps you are almost only a data sink, but if you are also a data source for others it pays to be cautious.

I think it’s more of an ethical question than anything. There will always be pranksters and there will never be perfect input validation for names. So who do you oppress? The people with uncommon names? Or the pranksters? I happen to think that if you do your job right, the pranksters aren’t really a problem. So why oppress those with less common names?

No one is being oppressed. Having to use an ASCII version of your name is literally a non-issue unless you WANT to be offended.

Maybe also think of the other humans that will need to read and retype the name. Do you expect everyone to understand and be able to type all characters? That's not reasonable. The best person to normalize the name to something interoperable is the user himself, so make him do it at data entry.

Re: Charset="WTF-8"

#422

Post by account42 »

dcow wrote: post #329
afiori wrote: post #328

Earlier quoted context omitted.

I am not saying to only allow [a-zA-Z ]+ in names, what I am Saying is that it is ok to block names like "'; drop table users;" or " " rel="nofollow">https://bad.site.net/> " if part of your business is to distribute that data to other consumers.

And I’m arguing, rhetorically, what if your name produces a syntax error—or worse means something semantically devious—in the query language I’m using? Not all problems look like script tags and semicolons.

This is getting really absurd. Are you also going to complain that Unicode is too restrictive or are you going to demand being able to use arbitrary bytes as names. Images? If Unicode is enough, then which version.

There is always a somewhat arbitrary restriction. It's not unreasonable to also take other people into account besides the user wanting to enter his special snowflake name.

Re: Charset="WTF-8"

#423

Post by account42 »

rob74 wrote: post #300
rafram wrote: post #272

Earlier quoted context omitted.

Sanitizing your strings immediately before display is all well and good until you need to pass them to some piece of third-party software that is very dumb and doesn’t sanitize them. You’ll argue that it’s the vendor’s fault, but the vendor will argue that nobody else allows characters like that in their name inputs! See the Companies House XSS injection situation, where their rationale for forcing a business to chan…

> but the vendor will argue that nobody else allows characters like that in their name inputs ...and maybe they will even link to this page to support that statement! But, seeing that most of the pages are German, I bet they do accept the usual German "special" letters (ÄÖÜß) in names?

So? Have you considered that the names may need to be eventually processed by people who understand the German alphabet but not all French accents (and certinly won't be able to type Hanzi or arabic or whatever else you expect everyone to support)? Will every system they interact with be able to deal with arbitrary symbols. Does the font of their letterhead support every script?

It's reasonable to expect a German company to deal with German script, less reasonable to expect them to deal with literally every script that someone once thought would be funny to include in Unicode.

Re: Charset="WTF-8"

#424

Post by account42 »

ACS_Solver wrote: post #316
Muromec wrote: post #273

Earlier quoted context omitted.

Quite the opposite actually. I want it stored correctly and in a way that both me and CSR can understand and so it can be used to interface with other systems. I don’t however know which unicode subset to use, because you didn’t tell me in the signup form. I have many options, all of them correct, but I don’t know whether your CSR can read Ukrainian Cyrillic and whether you can tell what vocative case is and not use…

I think you're touching on another problem, which is that we as users rarely know why the form wants a name. Is it to be used in emails, or for sending packages, or for talking to me? My language also has a separate vocative case, but I live in a country that has no concept of it and just vestiges of a case system. I enter my name in the nominative, which then of course looks weird if I get emails/letters from them l…

The thing is "printable ASCII letters" is something usable for all of those cases. It may not be 100% perfect for the user's feelings but it just works.

Re: Charset="WTF-8"

#425

Post by account42 »

int_19h wrote: post #390
swiftcoder wrote: post #296

Earlier quoted context omitted.

Not really, no. A lot of us only really have to deal with English-adjacent input (i.e. European languages that share the majority of character forms with English, or cultures that explicitly Anglicise their names when dealing with English folks). As soon as you have to deal with users with a radically different alphabet/input-method, the wheels tend to come off. Can your CSR reps pronounce names written in Chinese lo…

For one thing, this concern applies equally to names written entirely in Latin script. Can your CSR reps correctly pronounce a French name? How about Polish? Hungarian? In any case, the proper way to handle this is to store the name as originally written, and have the app that CSRs use provide a phonetic transcription. Coincidentally, this kind of stuff is something that LLMs are very good at already (but I bet you c…

This situation is not the same at all. The CSR might mangle a name in latin script but can at least attempt to pronounce it and will end up doing so in a way that the user can understand.

Add to that that natives of non-latin languages are already used to this.

For better or worse, English and therefore the basic latin script is the lingua franca of the computing age. Having something universal for internation communication is very useful.

Re: Charset="WTF-8"

#426

Post by account42 »

arghwhat wrote: post #322
Muromec wrote: post #254

Earlier quoted context omitted.

You can treat names as byte blobs for as long as you don't use them for their purpose -- naming people. Suppose you have a unicode blob of my name in your database and there is a problem and you need to call me and say hi. Would your customer representative be able to pronounce my name somewhat correctly? >I think there're very few exceptions to this, probably something law-related, or if you have to interact with so…

> Suppose you have a unicode blob of my name in your database and there is a problem and you need to call me and say hi. Would your customer representative be able to pronounce my name somewhat correctly? You cannot pronounce the name regardless of whether it is written in ASCII. Pronouncing a name requires at the very least knowledge of the language it originated in, and attempts at reading it with an English pronun…

> You cannot pronounce the name regardless of whether it is written in ASCII. Pronouncing a name requires at the very least knowledge of the language it originated in, and attempts at reading it with an English pronunciation can range from incomprehensible to outright offensive.

Offensive if you WANT to be offended perhaps but definitely understandable, which is the main thing that matters.

Re: Charset="WTF-8"

#427

Post by account42 »

soco wrote: post #311
Muromec wrote: post #277

Earlier quoted context omitted.

That’s the most basic consideration for names, unless you only show it to the user themselves — other people have to be able to read it at least somehow. Which one is why the bag of unicode bytes approach is as wrong as telling Stęphań he has an invalid name.

And here we go again, engineers expecting the world should behave fitting their framework du jour. Unfortunately, the real world doesn't care about our engineering bubble and goes on with life - where you can be called !xóõ Kxau or ꦱꦭꦪꦤ or X Æ A-12.

Your examples are a great argument why we should not allow people to use arbitrary characters. Special snowflakes will be able to cope, I assure you.

Re: Charset="WTF-8"

#428

Post by account42 »

red_admiral wrote: post #346
Muromec wrote: post #254

Earlier quoted context omitted.

You can treat names as byte blobs for as long as you don't use them for their purpose -- naming people. Suppose you have a unicode blob of my name in your database and there is a problem and you need to call me and say hi. Would your customer representative be able to pronounce my name somewhat correctly? >I think there're very few exceptions to this, probably something law-related, or if you have to interact with so…

> Would your customer representative be able to pronounce my name somewhat correctly? If the user is Chinese and the CSR is not - probably no, and that's not a Unicode issue.

Yet the CSR will be able to adequately pronounce a romanized ASCII-only version of the chinese name. And that's an entirely reasonable thing to do for western organizations and governments just like you might need to get a chinese name to interact with the chinese bureaucracy.

Re: Charset="WTF-8"

#429

Post by ACS_Solver »

account42 wrote: post #424
ACS_Solver wrote: post #316

Earlier quoted context omitted.

I think you're touching on another problem, which is that we as users rarely know why the form wants a name. Is it to be used in emails, or for sending packages, or for talking to me? My language also has a separate vocative case, but I live in a country that has no concept of it and just vestiges of a case system. I enter my name in the nominative, which then of course looks weird if I get emails/letters from them l…

The thing is "printable ASCII letters" is something usable for all of those cases. It may not be 100% perfect for the user's feelings but it just works.

This is patently wrong and it's the sort of thinking that still causes inconvenience to people using non-ASCII languages, years after it's technically justifiable.

The most typical problem scenario is getting some package or document with names transformed to ASCII and then being unable to actually receive the package or use the document because the name isn't your name. Especially when a third party is involved that doesn't speak the language that got mangled either.

Åke Källström is not the same name as Ake Kallstrom. Domestically the latter just looks stupid but then you get a hotel booking with that name, submit it as part of your visa application and the consulate says it's invalid because that's not your name.

Or when Rūta Lāse gets some foreign document or certificate, nobody in her country treats is authentic because the name written is Ruta Lase, which is also a valid and existing name - but a different one. She ends up having to request another document that establishes the original one is issued to her, and paying for an apostille on that so the original ASCII document is usable. While most languages have a standard way of changing arbitrary text to ASCII, the conversion function is often not bijective even for Latin-based alphabets.

These are real examples of real problems people still encounter because lots of English-speaking developers insist everyone should deal with an ASCII-fied version of their language. In the past I could certainly understand the technical difficulties, but we're some 20-25 years past the point where common software got good Unicode support. ASCII is no longer the only simple solution.

Re: Charset="WTF-8"

#430

Post by account42 »

Y_Y wrote: post #166
asddubs wrote: post #154

Earlier quoted context omitted.

or just double all the numbers and use DM

Weirdly the old Deutsch Mark doesn't seem to have its own code point in the block start U+20A0, whereas the Spanish equivalent (Peseta, ₧, not just Pt) does.

It's not a Unicode issue, there just isn't a dedicated symbol for it, everyone just used the letters DM. Unicode (at least back then) was mostly a superset of existing character sets and then distinct glyphs.
Post reply on HN