Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Charset="WTF-8"

wtf-8.xn--stpie-k0a81a.com

Post reply on HN
Print view View original

221–230 of 463 posts

Re: Charset="WTF-8"

#221

Post by crooked-v »

blahedo wrote: post #177
epcoa wrote: post #69

Earlier quoted context omitted.

The thing is though that payment networks do in fact do instant verification and it is interesting what gets verified and when. At gas stations it is very common to ask for a zip code (again US), and this is verified immediately to allow the transaction to proceed. I’ve found that when a street address is asked for there is some verification and often a match on the house number is sufficient. Zip codes are verified…

Funny thing about house numbers: they have their own validation problems. For a while I lived in a building whose house number was of the form 123½ and that was an ongoing source of problems. If it just truncated the ½ that was basically fine (the house at 123 didn't have apartment numbers and the postal workers would deliver it correctly) but validating in online forms (twenty-ish years ago) was a challenge. If they…

Around here, there used to be addresses like "0100 SW Whatever Ave" that were distinct from "100 SW Whatever Ave". And we've still got various places that have, for example, "NW 21st Avenue" and "NW 21st Place" as a simple workaround for a not-entirely-regular street grid optimized for foot navigation.

Re: Charset="WTF-8"

#222

Post by crooked-v »

josephcsible wrote: post #133

What would be wrong with "enter your name as it appears in the machine-readable zone of your passport" (or "would appear" for people who have never gotten one)? Isn't that the one standard format for names that actually is universal?

From experience, it's not actually universal. Visa applications will often ask for name information not encoded in the MRZ.

Re: Charset="WTF-8"

#223

Post by layer8 »

numpad0 wrote: post #182
wruza wrote: post #127

I'll say it again: this is the consequence of Unicode trying to be a mix of html and docx, instead of a charset. It went too far for an average Joe DevGuy to understand how to deal with it, so he just selects a subset he can handle and bans everything else. HN does that too - special symbols simply get removed. Unicode screwed itself up completely. We wanted a common charset for things like latin, extlatin, cjk, cyri…

IMO, the sin of Unicode is that they didn't just pick local language authorities and gave them standardized concepts like lines and characters, and start-of-language and end-of-language markers. Lots of Unicode issues come from handling languages that the code is not expecting, and codes currently has no means to select or report quirk supports. I suppose they didn't like getting national borders involved in technica…

I’m working with Word documents in different languages, and few people take the care to properly tag every piece of text with the correct language. What you’re proposing wouldn’t work very well in practice.

The other historical background is that when Unicode was designed, many national character sets and encodings existed, and Unicode’s purpose was to serve as a common superset of those, as otherwise you’d need markers when switching between encodings. So the existing encodings needed to be easily convertible to Unicode (and back), without markers, for Unicode to have any chance of being adopted. This was the value proposition of Unicode, to get rid of the case distinctions between national character sets as much as possible. As a sibling comment notes, originally there were also optional language markers, which however nobody used.

Re: Charset="WTF-8"

#224

Post by layer8 »

junek wrote: post #196
powersnail wrote: post #122

As someone who really think name field should just be one field with any printable unicode characters, I do wonder what the hell would I need to do if I take customer names in this form, and then my system has to interact with some other service that requires first/last name split, and/or [a-zA-Z] validation, like a bank or postal service. Automatic transliteration seems to be very dangerous (wrong name on bank accou…

The fundamental mistake is in trying to take input for one purpose and transform it for another purpose. Just have the user fill in an additional field for their name as it appears on bank statements, or whatever the second purpose is. Trying to be clever about this stuff never works out.

What you call the second purpose is often the only purpose. Or you have to talk to half a dozen other systems each of which have different constraints. You wouldn’t want to present the user half a dozen fields just so that they can choose the nicest representation of their name for each system.

That being said, in Japan it’s actually common to have two fields, one for the name in kanji (the “nice” name), and one in katakana (restricted, 8-bit phonetic alphabet, which earlier/older computer systems used and some probably still use).

Re: Charset="WTF-8"

#225

Post by ks2048 »

josephcsible wrote: post #188
ks2048 wrote: post #167

Earlier quoted context omitted.

There's the problem that "appears" is a visible phenomenon and unicode strings can contain non-visible characters and multiple ways to represent the same visible information. Normalization is supposed to help here, but some sites may fail to do this or do incorrectly, etc.

But the MRZ of a passport doesn't contain any of those problem characters.

But on some random website, with people copy-pasting from who-knows-what, they will have to normalize or validate, etc. to deal with such characters.

Re: Charset="WTF-8"

#226

Post by gwervc »

Y_Y wrote: post #160
meew0 wrote: post #148

Earlier quoted context omitted.

The “invisible symbols” are necessary to correctly represent human language. For instance, one of the most infamous Unicode control characters — the right-to-left override — is required to correctly encode mixed Latin and Hebrew text [1], which are both scripts that you mentioned. Besides, ASCII has control characters as well. The “colorful icons” are not part of Unicode. Emoji are just characters like any other. The…

I'm happy to discriminate against those damn ancient Sumerians and anyone still using goddamn Linear B.

People who should use Sumerian characters don't even use them, sadly. First probably because of habit with their transcription, but also because missing variants of characters mean lot of text couldn't be accurately represented. Also I'm downvoting you for discriminating me.

Re: Charset="WTF-8"

#228

Post by philipwhiuk »

xyst wrote: post #117

Software has been gaslighting generations of people around the world. Side note: not a bad way to skirt surveillance though. A name like “stępień” will without a doubt have many ambiguous spellings across different intelligence gathering systems (RUMINT, OSINT, …). Americans will probably spell it as “Stefen” or “Steven” or “Stephen”, especially once communicated over phone.

Turns out just foreign document is enough for that sometimes ;) http://news.bbc.co.uk/1/hi/northern_ireland/7899171.stm

Re: Charset="WTF-8"

#229

Post by hyeonwho4 »

nicbou wrote: post #209
powersnail wrote: post #122

As someone who really think name field should just be one field with any printable unicode characters, I do wonder what the hell would I need to do if I take customer names in this form, and then my system has to interact with some other service that requires first/last name split, and/or [a-zA-Z] validation, like a bank or postal service. Automatic transliteration seems to be very dangerous (wrong name on bank accou…

We had problems with a Ukrainian refugee we helped because certified translations of her documents did not match. Her name was transliterated the German way in one place and the English way in another. Those are translations coming from professionals who swore an oath. Don’t try to do it with code.

In the US, you can generally specify to your certified translators how you want proper names and place names written. I would suggest you or your friend talk to the translators again so that everything matches. It will also minimize future pains.

Also, USCIS usually has an "aliases" field on their forms, which would be a good place to put German government misspellings.

Re: Charset="WTF-8"

#230

Post by marcus_holmes »

tomtomtom777 wrote: post #207
perching_aix wrote: post #157

Earlier quoted context omitted.

In certain cultures yes. Where I live, you can only select from a central, though frequently updated, list of names when naming your child. So theoretically only (given) names that are on that list can occur. Family names are not part of this, but maybe that exists too elsewhere. I don't know how people whose name has been given to them before this list was established is handled however. An alternative method, which…

This assumes every resident is born and registered in said country which is a silly assumption. Surely, any service only catered only to "naturally born citizen" is discriminatory and illegal?

I read that Iceland asks people to change their names if they naturalise there (because of the -sson or -dottir surname suffix).

But your point stands - not everyone in the system will follow this pattern.

Post reply on HN