Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Charset="WTF-8"

wtf-8.xn--stpie-k0a81a.com

Post reply on HN
Print view View original

411–420 of 463 posts

Re: Charset="WTF-8"

#411

Post by hyeonwho4 »

77pt77 wrote: post #246
hyeonwho4 wrote: post #229

Earlier quoted context omitted.

In the US, you can generally specify to your certified translators how you want proper names and place names written. I would suggest you or your friend talk to the translators again so that everything matches. It will also minimize future pains. Also, USCIS usually has an "aliases" field on their forms, which would be a good place to put German government misspellings.

USCIS is a mess. I know someone that still doesn't know whether they have a middle name as far as american authorities are concerned. Coupled with "two last names" and it gets really messy, really quickly. Purchases names don't match the CC name. Bank statements are actually "for another person". Border crossings are now extra spicy. And "pray" that your name doesn't resemble a name in some blacklist.

That's exactly the kind of mess we're hoping to avoid.

I think these name problems are pretty common everywhere, though: even in Korea, where bureaucratic tasks are quite streamlined compared to Europe, foreigners whose names are too long get dashes in their official ID. Which official name goes into bank accounts and stuff. Pity the poor souls whoese bank ID doesn't match the telecom ID used for online and mobile ID verification during transactions...and during taxes.

Re: Charset="WTF-8"

#412

Post by 77pt77 »

shakna wrote: post #286
77pt77 wrote: post #245

Earlier quoted context omitted.

> If you treat your strings as opaque blobs, and use UTF8, most of internationalization problems go away This is laughably naive. So many things can go wrong. Strings are not arrays of bytes. There is a price to pay if someone doesn't understand that or chooses to ignore it.

> Strings are not arrays of bytes. That very much depends on the language that you are using. In some, they are.

No.

Those languages don't have strings.

Re: Charset="WTF-8"

#413

Post by kristopolous »

simonh wrote: post #407
kristopolous wrote: post #380

Earlier quoted context omitted.

You can say things like the different "styles" that exploit Unicode on a myriad of websites such as https://qaz.wtf/u/convert.cgi?text=Hello are not technically "fonts" but it's a distinction without a meaningful difference. You have script, fraktur, bold, monospace, italic...

Fraktur is interesting because it’s more a writing style, verging in a character set in its own right. However Unicode doesn’t directly support all of its ligatures and such. None of this is in any way justification for turning Unicode into something like SVG. Even the pseudo-drawing capabilities it does have are largely for legacy reasons.

Fraktur at one point was genuinely a different script

You can find texts in the late 1500-early 1900s at least that will switch to a fraktur style when quoting or using German.

ANSI escape codes even accommodates for it. Codepoint 20: https://en.m.wikipedia.org/wiki/ANSI_escape_code#Select_Grap...

Don't ask me why, I only work here.

See also https://en.wikipedia.org/wiki/Antiqua%E2%80%93Fraktur_disput...

I also don't find any of my predictions defensible as much as I believe they're inevitable. Again I've got no agency here.

Re: Charset="WTF-8"

#414

Post by soco »

Muromec wrote: post #378
soco wrote: post #371

Earlier quoted context omitted.

Then where is the problem? If the support can read Polish they will pronounce your name properly, if they're from India they will mess it up, why should we have different expectations? Nobody will identify you by name anyway, they will ask how to call you (chatbots do this already) and then use for proper identification all kind of ids and pins and whatnot. So we are talking here about a complexity that nobody actual…

>if they're from India they will mess it up Or not able to read at all. >Then where is the problem? Since you don't indicate for what purpose my name is stored, which may actually be display only, any of the following can happen: A name as entered in your system is compared to a name entered in a different system or when you interface (maybe indirectly and unknowingly) with a system using different constrains or a di…

>Or not able to read at all.

"Hello, how should we address you?". Not everything must be done in code.

>when you interface (maybe indirectly and unknowingly) with a system using different constrains

I have yet to encounter a system recognizing assets and making automatic decisions based on name. It would fail already if the user switched first/last name.

>people exist that actually get it right

You could have started by explaining this right way and we'd be all smarter.

Re: Charset="WTF-8"

#415

Post by shakna »

77pt77 wrote: post #412
shakna wrote: post #286

Earlier quoted context omitted.

> Strings are not arrays of bytes. That very much depends on the language that you are using. In some, they are.

No. Those languages don't have strings.

So Lua doesn't have strings? The type is called a string. The documentation calls it a string. It's certainly not a buffer.

Re: Charset="WTF-8"

#416

Post by numpad0 »

tadfisher wrote: post #391
numpad0 wrote: post #288

Earlier quoted context omitted.

That's not "mistake", that's the reality. They don't exchange, and they're not the same. "Same or trivially derived" is just a completely false statement that solely exist to justify Han Unification, or maybe something that made sense in the 80s, it doesn't make literal sense.

> "Same or trivially derived" is just a completely false statement You'd have to ignore a lot of reality to believe this. It's even in the names of the writing systems: Kanji, Hanja, Chữ Hán. Of course they don't exchange, because they don't carry the same meaning, just as the word "chat" means completely different things in French and English. But it is literally the same script, albeit with numerous stylistic diffe…

CJK native speakers can't read or write other "trivially derived" versions of Hanzi. I don't understand why this has to be reiterated ad infinitum.

We can't actually read Simplified Chinese as a native Japanese just like French speakers can't exactly read Cyrillic, only recognize some of it. Therefore those are different alphabet sets. Simple as that.

The "trivially derived different styles" justification assumes that to be false, that native users of all 3 major styles of Hanzi can write, at least read, the other two styles without issues. That is not true.

Итъс а реал проблем то бе cонстантлй пресентед wитҳ чараcтерс тҳат И жуст cанът реад он тҳе гроунд тҳат тҳейъре "саме".

I hope you don't get offended by the line before this, because that's "same" latin, isn't it?

Re: Charset="WTF-8"

#417

Post by numpad0 »

anonymoushn wrote: post #213
kevin_thibedeau wrote: post #201

Earlier quoted context omitted.

> start-of-language and end-of-language markers Unicode used to have language tagging but they've been (mostly) deprecated: https://en.wikipedia.org/wiki/Tags_(Unicode_block) https://www.unicode.org/reports/tr7/tr7-1.html

The lack of such markers prevents Unicode from encoding strings of mixed Japanese and Chinese text correctly. Or in the case of a piece of software that must accept both Chinese and Japanese names for different people, Unicode is insufficient for encoding the written forms of the names.

Just in case this has to be said: the reason this hasn't been a problem in the past is because you could solve this problem by picking a team and completely breaking support for the others.

With rapidly improving single-image i18n in OS and apps, "breaking support for the others" slowly became non-ideal or hardly viable solution, and the problem surfaced.

Re: Charset="WTF-8"

#418

Post by barryrandall »

croes wrote: post #3
jtvjan wrote: post #2

A coworker once implemented a name validation regex that would reject his own name. It still mystifies me how much convincing it took to get him to make it less strict.

Is name validation even possible?

Anything is possible with enough qualifiers and caveats.

Re: Charset="WTF-8"

#419

Post by account42 »

1) The "real" WTF-8 charset [0] is a cool and useful encoding.

2) This may be an unpopular opinion but I think restricting name input to latin is an OK thing to do, especially if the entered names are being viewed/used by humans who cannot be expected to be know all scripts.

3) Similarly, internationalized domain names were a mistake. If your business card tells me to go to stępień.com then chances are I won't bother to try end remember how to enter those accents on my keyboard layout. Most users won't even be able to enter them. Worse are letters that are visually indistinguishible - and no, registries preventing confusable names is not enough when I still won't know which letter to enter. This makes IDN domains less useful while retaining all the security issues they bring.

Most languages were already forced to deal with ASCII and have developed standardized ways to "romanize" names and other words to that character set. This solution achieves peak interopability - not only between computer systems but also between the fleshy components operating them.

[0] https://simonsapin.github.io/wtf-8/

Re: Charset="WTF-8"

#420

Post by account42 »

arkh wrote: post #309
rafram wrote: post #272

Earlier quoted context omitted.

Sanitizing your strings immediately before display is all well and good until you need to pass them to some piece of third-party software that is very dumb and doesn’t sanitize them. You’ll argue that it’s the vendor’s fault, but the vendor will argue that nobody else allows characters like that in their name inputs! See the Companies House XSS injection situation, where their rationale for forcing a business to chan…

You sanitize at the frontier of what your code controls. Sending data to a database: parametrized queries to sanitize as it is leaving your control. Sending to display to the user: sanitized for a browser Sending to an API: sanitize for whatever rules the API has Sending to a legacy system: sanitize for it Writing a file to the system: sanitize the path The common point is you don't sanitize before you have to send i…

What if the legacy API doesn't support escaping? Just drop characters? Implement your own ad-hoc transform? What if you need to interoperate with other API users.

Limting the character set at name input gives the user the chance to use the same ASCII-encoding of their name in all places.

Post reply on HN