Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Charset="WTF-8"

wtf-8.xn--stpie-k0a81a.com

Post reply on HN
Print view View original

241–250 of 463 posts

Re: Charset="WTF-8"

#241

Post by ddulaney »

kgeist wrote: post #232

My rule of thumb is to treat strings as opaque blobs most of the time. The only validation I'd always enforce is some sane length limit, to prevent users from shoving entire novels inside. If you treat your strings as opaque blobs, and use UTF8, most of internationalization problems go away. Imho often times, input validation is an attempt to solve a problem from the wrong side. Say, when XSS or SQL injections are fo…

There's at least one major exception to this: Unicode normalization.

It's possible for the same logical character to have two different sets of code points (for example, a-with-umlaut as a single character, vs a followed by umlaut combining diacritic). Related, distinguishing between the "a" character in Latin, Greek, Cyrillic, and the handful of other times it shows up throughout Unicode.

This comes up in at least 3 ways:

1. A usability issue. It's not always easy to predict which identical-looking variant is produced by different input methods, so users enter the identical-looking characters on different devices but get an account not found error.

2. A security issue. If some of your backend systems handle these kinds of characters differently, that can cause all kinds of weird bugs, some of which can be exploited.

3. An abuse issue. If it's possible to create accounts with the same-looking name as others that aren't the same account, there can be vectors for impersonation, harassment, and other issues.

So you have to make a policy choice about how to handle this problem. The only things that I've seen work are either restricting the allowed characters (often just to printable ASCII) or being very clear and strict about always performing one of the standard Unicode transformations. But doing that transformation consistently across a big codebase has some real challenges: in particular, it can change based on Unicode version, and guaranteeing that all potential services use the same Unicode version is really non-trivial. So lots of people make the (sensible) choice not to deal with it.

But yeah, agreed that parenthesis should be OK.

Re: Charset="WTF-8"

#242

Post by mmoskal »

extraduder_ire wrote: post #217
bjackman wrote: post #159

Earlier quoted context omitted.

I believe this is what Rust OsStrings are under the hood on Windows.

Which I assume stands for "Windows-Transformation-Format-8(bits)".

Abstract

WTF-8 (Wobbly Transformation Format − 8-bit) is a superset of UTF-8 that encodes surrogate code points if they are not in a pair.

Re: Charset="WTF-8"

#243

Post by Muromec »

lmm wrote: post #236
Muromec wrote: post #142

Earlier quoted context omitted.

>But you can compare the transliterated version to make sure it matches No you can't. Add: Okay, you need to know why. I'm right here a living breathing person with a government id that has the same name scribed in two scripts side by side. There is an algorithm (blessed by the same government that issued said it) which defines how to transliterate names from one to another, published on the parliament web site and i…

Your example fails to explain any problem with GPs proposal. They would show you a transliteration of your name and ask you to confirm it. You would confirm it or not. It might match one or other of your IDs (in which case you would presumably say yes) or not (in which case you would presumably say no). What's the issue?

You will compare the transliterated version I provided with the one you have already, it will not match and then what? Either you tell me I have invalid name or you just ignore it.

Re: Charset="WTF-8"

#244

Post by Muromec »

layer8 wrote: post #224
junek wrote: post #196

Earlier quoted context omitted.

The fundamental mistake is in trying to take input for one purpose and transform it for another purpose. Just have the user fill in an additional field for their name as it appears on bank statements, or whatever the second purpose is. Trying to be clever about this stuff never works out.

What you call the second purpose is often the only purpose. Or you have to talk to half a dozen other systems each of which have different constraints. You wouldn’t want to present the user half a dozen fields just so that they can choose the nicest representation of their name for each system. That being said, in Japan it’s actually common to have two fields, one for the name in kanji (the “nice” name), and one in k…

You usually don't have a dozen, just two or three and if you do have a dozen, there is a certain pattern, or at least the common denominator for the half of them to be ASCII, a another half using some kind of local convention you totally know how to encode.

Re: Charset="WTF-8"

#245

Post by 77pt77 »

kgeist wrote: post #232

My rule of thumb is to treat strings as opaque blobs most of the time. The only validation I'd always enforce is some sane length limit, to prevent users from shoving entire novels inside. If you treat your strings as opaque blobs, and use UTF8, most of internationalization problems go away. Imho often times, input validation is an attempt to solve a problem from the wrong side. Say, when XSS or SQL injections are fo…

> If you treat your strings as opaque blobs, and use UTF8, most of internationalization problems go away

This is laughably naive.

So many things can go wrong.

Strings are not arrays of bytes.

There is a price to pay if someone doesn't understand that or chooses to ignore it.

Re: Charset="WTF-8"

#246

Post by 77pt77 »

hyeonwho4 wrote: post #229
nicbou wrote: post #209

Earlier quoted context omitted.

We had problems with a Ukrainian refugee we helped because certified translations of her documents did not match. Her name was transliterated the German way in one place and the English way in another. Those are translations coming from professionals who swore an oath. Don’t try to do it with code.

In the US, you can generally specify to your certified translators how you want proper names and place names written. I would suggest you or your friend talk to the translators again so that everything matches. It will also minimize future pains. Also, USCIS usually has an "aliases" field on their forms, which would be a good place to put German government misspellings.

USCIS is a mess.

I know someone that still doesn't know whether they have a middle name as far as american authorities are concerned.

Coupled with "two last names" and it gets really messy, really quickly.

Purchases names don't match the CC name.

Bank statements are actually "for another person".

Border crossings are now extra spicy.

And "pray" that your name doesn't resemble a name in some blacklist.

Re: Charset="WTF-8"

#247

Post by hughesjj »

77pt77 wrote: post #245
kgeist wrote: post #232

My rule of thumb is to treat strings as opaque blobs most of the time. The only validation I'd always enforce is some sane length limit, to prevent users from shoving entire novels inside. If you treat your strings as opaque blobs, and use UTF8, most of internationalization problems go away. Imho often times, input validation is an attempt to solve a problem from the wrong side. Say, when XSS or SQL injections are fo…

> If you treat your strings as opaque blobs, and use UTF8, most of internationalization problems go away This is laughably naive. So many things can go wrong. Strings are not arrays of bytes. There is a price to pay if someone doesn't understand that or chooses to ignore it.

RTL go brrr

Re: Charset="WTF-8"

#248

Post by lmm »

zarzavat wrote: post #15
croes wrote: post #3

Earlier quoted context omitted.

Is name validation even possible?

Presumably there aren't any people with control characters in their name, for example.

> Presumably there aren't any people with control characters in their name, for example.

Of course there are. If you commit to supporting everything anyone wants to do, people will naturally test the boundaries.

The biggest fallacy programmers believe about names is that getting name support 100% right matters. Real engineers build something that works well enough for enough of the population and ship it, and if that's not US-ASCII only then it's usually pretty close to it.

Re: Charset="WTF-8"

#249

Post by lmm »

crazygringo wrote: post #14
croes wrote: post #3

Earlier quoted context omitted.

Is name validation even possible?

If you just use the {Alphabetic} Unicode character class (100K code points), together with a space, hyphen, and maybe comma, that might get you close. It includes diacritics. I'm curious if anyone can think of any other non-alphabetic characters used in legal names around the world, in other scripts? I wondered about numbers, but the most famous example of that has been overturned: "Originally named X Æ A-12, the chi…

> I'm curious if anyone can think of any other non-alphabetic characters used in legal names around the world, in other scripts?

Some Japanese names are written with Japanese characters that do not have Unicode codepoints.

(The Unicode consortium claims that these characters are somehow "really" Chinese characters just written in a different font; holders of those names tend to disagree, but somehow the programmer community that would riot if someone suggested that people with ø in their name shouldn't care when it's written as o accepts that kind of thing when it comes to Japanese).

Re: Charset="WTF-8"

#250

Post by Muromec »

josephcsible wrote: post #133

What would be wrong with "enter your name as it appears in the machine-readable zone of your passport" (or "would appear" for people who have never gotten one)? Isn't that the one standard format for names that actually is universal?

The problem is, people exists that have É in their name and will go to court when you spell it as E, the court will also say that 1 ) you have the technical ability to write it as É and 2) they have a right to have their name spelled correctly. Also it's not nice and bad for business to be like this.
Post reply on HN