Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Charset="WTF-8"

wtf-8.xn--stpie-k0a81a.com

Post reply on HN
Print view View original

331–340 of 463 posts

Re: Charset="WTF-8"

#331

Post by hooby »

MrJohz wrote: post #82
throw310822 wrote: post #35

Earlier quoted context omitted.

I know multiple developers who would just say "well it's their fault, they have to change name then".

I worked with an office of Germans who insisted that ASCII was sufficient. The German language uses letters that cannot be represented in ASCII. In fairness, they mostly wanted stuff to be in English, and when necessary, to transliterate German characters into their English counterparts (in German there is a standardised way of doing this), so I can understand why they didn't see it was necessary. I just never unders…

There are some valid reasons to use software in English as a German speaker. Main among those is probably translations.

If you can speak English, you might be better of using the software in English, as having to deal with the English language can often be less of hassle, than having to deal with inconsistent, weird, or outright wrong translations.

Even high quality translations might run into issues, where the same thing is translated once as "A" and then as "B" in another context. Or run into issues where there is an English technical term being used, that has no prefect equivalent in German (i.e. a translation does exist, but is not a well-known, clearly defined technical term). More often than not though, translations are anything but high quality. Even in expensive products from big international companies.

Re: Charset="WTF-8"

#332

Post by smatija »

rmrfchik wrote: post #330

Well, the labels of input fields are written in English yet user enters his name in native language. What's the reason of having name at all? You can call the person by this name. But if I write you my name in my language, what you (not knowing how to read it) can do? Only "hey, still-don't-know-you, here is your info". In my foreign passport I have name __transliterated__ to Latin alphabet. Shouldn't this be the cas…

w3c recommends adding a separate field for pronounciation - see e.g. paragraph after second image in https://www.w3.org/International/questions/qa-personal-names...

Re: Charset="WTF-8"

#333

Post by JodieBenitez »

kgeist wrote: post #232

My rule of thumb is to treat strings as opaque blobs most of the time. The only validation I'd always enforce is some sane length limit, to prevent users from shoving entire novels inside. If you treat your strings as opaque blobs, and use UTF8, most of internationalization problems go away. Imho often times, input validation is an attempt to solve a problem from the wrong side. Say, when XSS or SQL injections are fo…

> or if you have to interact with some legacy service.

Which happens almost every day in the real world.

Re: Charset="WTF-8"

#334

Post by wvh »

There's little more you can do to validate a name internationally than to provide one textbox and check if it's a valid encoding of Unicode. Maybe you can exclude some control and graphical ranges at best.

Of course there are valid concerns that international names should pass through e.g. local postal services, which would require at least some kind of Latinized representation of name and address. I suppose the Latin alphabet is the most convenient minimal common denominator across writing systems, even though I admit being Euro-centric.

Re: Charset="WTF-8"

#335

Post by rmrfchik »

smatija wrote: post #332
rmrfchik wrote: post #330

Well, the labels of input fields are written in English yet user enters his name in native language. What's the reason of having name at all? You can call the person by this name. But if I write you my name in my language, what you (not knowing how to read it) can do? Only "hey, still-don't-know-you, here is your info". In my foreign passport I have name __transliterated__ to Latin alphabet. Shouldn't this be the cas…

w3c recommends adding a separate field for pronounciation - see e.g. paragraph after second image in https://www.w3.org/International/questions/qa-personal-names...

Wow, that's neat.

Re: Charset="WTF-8"

#336

Post by wvh »

kgeist wrote: post #232

My rule of thumb is to treat strings as opaque blobs most of the time. The only validation I'd always enforce is some sane length limit, to prevent users from shoving entire novels inside. If you treat your strings as opaque blobs, and use UTF8, most of internationalization problems go away. Imho often times, input validation is an attempt to solve a problem from the wrong side. Say, when XSS or SQL injections are fo…

Because you don't want to ever store bad data. There's not point to that, it will just create annoying situations and potential security risks. And the best place to catch bad data is when the user is still present so they can be made aware of the issue (in case they care and are able to solve it). Once they're gone, it becomes nearly impossible and/or very expensive to check what they meant.

Re: Charset="WTF-8"

#337

Post by beagle3 »

kgeist wrote: post #232

My rule of thumb is to treat strings as opaque blobs most of the time. The only validation I'd always enforce is some sane length limit, to prevent users from shoving entire novels inside. If you treat your strings as opaque blobs, and use UTF8, most of internationalization problems go away. Imho often times, input validation is an attempt to solve a problem from the wrong side. Say, when XSS or SQL injections are fo…

You do need to use a canonical representation, or you will have two distinct blobs that look exactly the same, tricking other users of the data (other posters in a forum, customer service people in a company, etc)

Re: Charset="WTF-8"

#338

Post by jlhwung »

crazygringo wrote: post #14
croes wrote: post #3

Earlier quoted context omitted.

Is name validation even possible?

If you just use the {Alphabetic} Unicode character class (100K code points), together with a space, hyphen, and maybe comma, that might get you close. It includes diacritics. I'm curious if anyone can think of any other non-alphabetic characters used in legal names around the world, in other scripts? I wondered about numbers, but the most famous example of that has been overturned: "Originally named X Æ A-12, the chi…

https://en.wikipedia.org/wiki/Perri_6

Re: Charset="WTF-8"

#339

Post by speleding »

ddulaney wrote: post #241
kgeist wrote: post #232

My rule of thumb is to treat strings as opaque blobs most of the time. The only validation I'd always enforce is some sane length limit, to prevent users from shoving entire novels inside. If you treat your strings as opaque blobs, and use UTF8, most of internationalization problems go away. Imho often times, input validation is an attempt to solve a problem from the wrong side. Say, when XSS or SQL injections are fo…

There's at least one major exception to this: Unicode normalization. It's possible for the same logical character to have two different sets of code points (for example, a-with-umlaut as a single character, vs a followed by umlaut combining diacritic). Related, distinguishing between the "a" character in Latin, Greek, Cyrillic, and the handful of other times it shows up throughout Unicode. This comes up in at least 3…

Something we just ran in to: There are two UTF-8 codepoints for the @ character, the normal one and "Full width At Sign U+FF20". It took a lot of head scratching to understand why several Japanese users could not be found with their email address when I was seeing their email right there in the database.

Re: Charset="WTF-8"

#340

Post by Y_Y »

gwervc wrote: post #226
Y_Y wrote: post #160

Earlier quoted context omitted.

I'm happy to discriminate against those damn ancient Sumerians and anyone still using goddamn Linear B.

People who should use Sumerian characters don't even use them, sadly. First probably because of habit with their transcription, but also because missing variants of characters mean lot of text couldn't be accurately represented. Also I'm downvoting you for discriminating me.

I know you're being funny, but that's sort of the point. There's an important "use-mention" distinction when it comes to historical character sets. You surely could try to communicate in authentic Unicode Akkadian (𒀝𒅗𒁺𒌑(𒌝) but what's much more likely is that you really just want to refer to characters or short strings thereof while communicating anything else in a modern living language like English. I don't want to stop someone from trying to revive the language for fun or profit, but I think there's an important distinction between cases of primarily historical interest like that, and cases that are awkward but genuine like Inuktut.
Post reply on HN