Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Charset="WTF-8"

wtf-8.xn--stpie-k0a81a.com

Post reply on HN
Print view View original

231–240 of 463 posts

Re: Charset="WTF-8"

#231

Post by wruza »

Muromec wrote: post #132
wruza wrote: post #127

I'll say it again: this is the consequence of Unicode trying to be a mix of html and docx, instead of a charset. It went too far for an average Joe DevGuy to understand how to deal with it, so he just selects a subset he can handle and bans everything else. HN does that too - special symbols simply get removed. Unicode screwed itself up completely. We wanted a common charset for things like latin, extlatin, cjk, cyri…

>so he just selects a subset he can handle and bans everything else. Yes? And the problem is?

The problem is the scale at which it happens and lack of methods-to-go in most runtimes/libs. No one and nothing is ready for unicode complexity out of box, and there's little interest in unscrewing it by oneself, cause it looks like an absurd minefield and likely is one, from the persepective of an average developer. So they get defensive by default, which results in $subj.

Re: Charset="WTF-8"

#232

Post by kgeist »

My rule of thumb is to treat strings as opaque blobs most of the time. The only validation I'd always enforce is some sane length limit, to prevent users from shoving entire novels inside. If you treat your strings as opaque blobs, and use UTF8, most of internationalization problems go away. Imho often times, input validation is an attempt to solve a problem from the wrong side. Say, when XSS or SQL injections are found on a site, I've seen people's first reaction to be validation of user input by looking for "special symbols", or add a whitelist of allowed characters, instead of simply escaping strings right before rendering HTML (and modern frameworks do it automatically), or using parameterized queries if it's SQL. If a user wants to call themselves "alert('hello')", why not? Why the arbitrary limits? I think there're very few exceptions to this, probably something law-related, or if you have to interact with some legacy service.

Re: Charset="WTF-8"

#233

Post by tadfisher »

numpad0 wrote: post #195
tadfisher wrote: post #158

Earlier quoted context omitted.

They are the same character, though. They do not use the same glyph in different language contexts, but Unicode is a character encoding, not a font standard.

They're not. Readers native in one version can't read the other, and there are more than handful that got duplicated in multiple forms, so they're just not same, just similar. You know, obvious presumption underlying Han Unification is that CJK languages must have a continuous dialect continuums, like villagers living in the middle of East China Sea between Shanghai and Nagasaki and Gwangju would speak half-Chinese-J…

You're making the same mistake: the languages are different, but the script is the same (or trivially derived from the Han script). The Ideographic Research Group was well aware of this, having consisted of native speakers of the languages in question.

Re: Charset="WTF-8"

#234

Post by tadfisher »

Muromec wrote: post #189
tadfisher wrote: post #158

Earlier quoted context omitted.

They are the same character, though. They do not use the same glyph in different language contexts, but Unicode is a character encoding, not a font standard.

Yes, but the same is true for overlapping characters in Cyrillic and Latin. A and А are the same glyph, so are т,к,і and t,k,i and you can even see the difference between some of those.

The duplication there is mostly to remain compatible or trivially transformable with existing encodings. Ironically, the two versions of your example "A" do look different on my device (Android), with a slightly lower x-height for the Cyrillic version.

Re: Charset="WTF-8"

#235

Post by JimDabell »

Macha wrote: post #97
pridkett wrote: post #29

Earlier quoted context omitted.

That only works if you’re concatenating the first and last name fields. Some people have no last name and thus would fail this validation if the system had fields for first and last name.

Honestly I wish we could just abolish first and last name fields and replace them with a single free text name field since there's so many edge cases where first and last is an oversimplification that leads to errors. Unfortunately we have to interact with external systems that themselves insist on first and last name fields, and pushing it to the user to decide which is part of what name is wrong less often than str…

It’s not just external systems. In many (most?) places, when sorting by name, you use the family names first, then the given names. So you can’t correctly sort by name unless you split the fields. Having a single field, in this case, is “an oversimplification that leads to errors”.

Re: Charset="WTF-8"

#236

Post by lmm »

Muromec wrote: post #142
matthewbauer wrote: post #140

Earlier quoted context omitted.

You can just show the user the transliteration & have them confirm it makes sense. Always store the original version since you can't reverse the process. But you can compare the transliterated version to make sure it matches. Debit cards a pretty common example of this. I believe you can only have ASCII in the cardholder name field.

>But you can compare the transliterated version to make sure it matches No you can't. Add: Okay, you need to know why. I'm right here a living breathing person with a government id that has the same name scribed in two scripts side by side. There is an algorithm (blessed by the same government that issued said it) which defines how to transliterate names from one to another, published on the parliament web site and i…

Your example fails to explain any problem with GPs proposal. They would show you a transliteration of your name and ask you to confirm it. You would confirm it or not. It might match one or other of your IDs (in which case you would presumably say yes) or not (in which case you would presumably say no). What's the issue?

Re: Charset="WTF-8"

#237

Post by lmm »

tomtomtom777 wrote: post #207
perching_aix wrote: post #157

Earlier quoted context omitted.

In certain cultures yes. Where I live, you can only select from a central, though frequently updated, list of names when naming your child. So theoretically only (given) names that are on that list can occur. Family names are not part of this, but maybe that exists too elsewhere. I don't know how people whose name has been given to them before this list was established is handled however. An alternative method, which…

This assumes every resident is born and registered in said country which is a silly assumption. Surely, any service only catered only to "naturally born citizen" is discriminatory and illegal?

> Surely, any service only catered only to "naturally born citizen" is discriminatory and illegal?

No, that's also a question that is culturally dependent. In some contexts it's normal and expected.

Re: Charset="WTF-8"

#238

Post by n2d4 »

wruza wrote: post #204
meew0 wrote: post #148

Earlier quoted context omitted.

The “invisible symbols” are necessary to correctly represent human language. For instance, one of the most infamous Unicode control characters — the right-to-left override — is required to correctly encode mixed Latin and Hebrew text [1], which are both scripts that you mentioned. Besides, ASCII has control characters as well. The “colorful icons” are not part of Unicode. Emoji are just characters like any other. The…

Copy this󠀠󠀼󠀼󠀼󠀠󠁉󠁳󠀠󠁴󠁨󠁩󠁳󠀠󠁮󠁥󠁣󠁥󠁳󠁳󠁡󠁲󠁹󠀠󠁴󠁯󠀠󠁣󠁯󠁲󠁲󠁥󠁣󠁴󠁬󠁹󠀠󠁲󠁥󠁰󠁲󠁥󠁳󠁥󠁮󠁴󠀠󠁨󠁵󠁭󠁡󠁮󠀠󠁬󠁡󠁮󠁧󠁵󠁡󠁧󠁥󠀿󠀠󠀾󠀾󠀾 sentence into this site and click Decode. (YMMW) https://embracethered.com/blog/ascii-smuggler.html

> Is this necessary to correctly represent human language?

Yes! As soon as you have any invisible characters (eg. RTL or LTR marks, which are required to represent human language), you will be able to encode any data you want.

Re: Charset="WTF-8"

#239

Post by nedt »

KPGv2 wrote: post #39

It seems ridiculous to apply form validation to a name, given the complexity of charsets involved. I don't even validate email addresses. I remember [this]( https://www.netmeister.org/blog/email.html ) wonderful explainer of why your email validation regex is wrong.

In HTML if you use you basically get a regex validation. While it not fully follows the RFC it's a good middle ground. And it gives you an email that you can use on the internet (RFC obviously has some cases that are outside of this scope). That's why I tend to prefer what's defined in the standard: https://html.spec.whatwg.org/multipage/input.html#email-stat...

Re: Charset="WTF-8"

#240

Post by rrr_oh_man »

Y_Y wrote: post #171
bjackman wrote: post #161

Earlier quoted context omitted.

I still don't see how any system in the real world can safely assume its users only have names from that list. Even if you try to imagine a system for a hospital to register newly born babies... What happens if a pregnant tourist is visiting?

For example in Iceland you don't have to name the baby immediately, and the registration times are different for foreign parents. https://www.skra.is/english/people/registration-of-children/... Of course then you may fall foul of classic falsehood 40: People have names.

For today's lucky 10,000: Falsehoods programmers believe about names (https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-...)
Post reply on HN