Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Charset="WTF-8"

wtf-8.xn--stpie-k0a81a.com

Post reply on HN
Print view View original

271–280 of 463 posts

Re: Charset="WTF-8"

#271

Post by aguaviva »

aguaviva wrote: post #255
Muromec wrote: post #141

Earlier quoted context omitted.

It's a general pattern of what russia does to names of places and people, which is aggressively imposing their own cultural paradigm (which follows the more general general pattern). You can look up your civil code provisions around names and ask a question or two of what historical problem they attempt to solve.

It's not a Russian-specific thing by any stretch. This happens all the time when names and loanwords get dragged across linguistic boundaries. Sometimes it results from an attempt to "simplify" the respective spelling and/or sounds (by mapping them into tokens more familiar in the local environment); sometimes there's a more complex process behind it; and other times it just happens for various obscure historical rea…

Correction - it's nothing osbcure at all, but apparently a matter of the shift that accord broadly with the L sound in Polish a few centuries ago (whereby it became "dark" and velarized), affecting a great many other words and names (like słowo, mały, etc). While in parts east and south the "clear" L sound was preserved.

https://en.wikipedia.org/wiki/Ł

Re: Charset="WTF-8"

#272

Post by rafram »

kgeist wrote: post #232

My rule of thumb is to treat strings as opaque blobs most of the time. The only validation I'd always enforce is some sane length limit, to prevent users from shoving entire novels inside. If you treat your strings as opaque blobs, and use UTF8, most of internationalization problems go away. Imho often times, input validation is an attempt to solve a problem from the wrong side. Say, when XSS or SQL injections are fo…

Sanitizing your strings immediately before display is all well and good until you need to pass them to some piece of third-party software that is very dumb and doesn’t sanitize them. You’ll argue that it’s the vendor’s fault, but the vendor will argue that nobody else allows characters like that in their name inputs!

See the Companies House XSS injection situation, where their rationale for forcing a business to change its name was that others using their database could be vulnerable: https://www.theregister.com/2020/10/30/companies_house_xss_s...

Re: Charset="WTF-8"

#273

Post by Muromec »

kmoser wrote: post #256
Muromec wrote: post #254

Earlier quoted context omitted.

You can treat names as byte blobs for as long as you don't use them for their purpose -- naming people. Suppose you have a unicode blob of my name in your database and there is a problem and you need to call me and say hi. Would your customer representative be able to pronounce my name somewhat correctly? >I think there're very few exceptions to this, probably something law-related, or if you have to interact with so…

> Would your customer representative be able to pronounce my name somewhat correctly? Are you implying the CSR's lack of familiarity with the pronunciation of your name means your name should be stored/rendered incorrectly?

Quite the opposite actually. I want it stored correctly and in a way that both me and CSR can understand and so it can be used to interface with other systems.

I don’t however know which unicode subset to use, because you didn’t tell me in the signup form. I have many options, all of them correct, but I don’t know whether your CSR can read Ukrainian Cyrillic and whether you can tell what vocative case is and not use that when inerfacing with the government CA which expects nominative.

Re: Charset="WTF-8"

#274

Post by Dylan16807 »

lmm wrote: post #252
tadfisher wrote: post #158

Earlier quoted context omitted.

They are the same character, though. They do not use the same glyph in different language contexts, but Unicode is a character encoding, not a font standard.

This is a bullshit argument that never gets applied to any other live language. The characters are different, people who actually use them in daily life recognise them as conveying different things. If a thumbs up with a different skin tone is a different character then a different pattern of lines is definitely a different character.

> If a thumbs up with a different skin tone is a different character

Is it? The skin tone modifier is serving the same purpose as a variant selector for the CJK codepoint would be.

Re: Charset="WTF-8"

#275

Post by rurban »

philipwhiuk wrote: post #227
rurban wrote: post #149

Just use the unicode identifier rules, my libu8ident. https://github.com/rurban/libu8ident Windows folks need to convert to UTF—8 first

That's nice. Which restriction level handles all names?

Every. Identifiers are identifiable names. Most languages with unicode name support only support symbols, ie not identifiable names (binary trash).

Re: Charset="WTF-8"

#276

Post by Muromec »

josephcsible wrote: post #262
Muromec wrote: post #250

Earlier quoted context omitted.

The problem is, people exists that have É in their name and will go to court when you spell it as E, the court will also say that 1 ) you have the technical ability to write it as É and 2) they have a right to have their name spelled correctly. Also it's not nice and bad for business to be like this.

> they have a right to have their name spelled correctly IMO, having the law consider this as an unconditional right is the root of the problem. What happens when people start making up their own characters that aren't in Unicode to put in their names? > Also it's not nice and bad for business to be like this. What about having a validated "legal name" for everything official and an unvalidated "display name" that's…

> What happens when people start making up their own characters that aren't in Unicode to put in their names?

They first have to fight the Unicode committee and maybe they actually have a point and the character is made up in a way that is acceptable in a society. Then they will fight their local authorities who run everything on 30 years old system. Only after they become your problem, at which point you fix your cursed regexp.

>an unvalidated "display name" that's only ever parroted back to the person who entered it?

You will do that wrong too. When you send me an email, I would expect my name to be in different form compared to what you display in the active user widget.

The point is, you need to know the exact context in which the name is used and also communicate it to me so I can tell you the right thing to display.

Re: Charset="WTF-8"

#277

Post by Muromec »

hobs wrote: post #269
Muromec wrote: post #254

Earlier quoted context omitted.

You can treat names as byte blobs for as long as you don't use them for their purpose -- naming people. Suppose you have a unicode blob of my name in your database and there is a problem and you need to call me and say hi. Would your customer representative be able to pronounce my name somewhat correctly? >I think there're very few exceptions to this, probably something law-related, or if you have to interact with so…

Absolutely not - do not build anything based on "would your CSR be able to pronounce" something - that's an awful bar - most CSRs cant pronounce my name - would I be excluded from your database? Seriously, what are you going for here?

That’s the most basic consideration for names, unless you only show it to the user themselves — other people have to be able to read it at least somehow.

Which one is why the bag of unicode bytes approach is as wrong as telling Stęphań he has an invalid name.

Re: Charset="WTF-8"

#278

Post by ordu »

MrJohz wrote: post #82
throw310822 wrote: post #35

Earlier quoted context omitted.

I know multiple developers who would just say "well it's their fault, they have to change name then".

I worked with an office of Germans who insisted that ASCII was sufficient. The German language uses letters that cannot be represented in ASCII. In fairness, they mostly wanted stuff to be in English, and when necessary, to transliterate German characters into their English counterparts (in German there is a standardised way of doing this), so I can understand why they didn't see it was necessary. I just never unders…

> I just never understood why I, as the non-German, was forever the one trying to convince them that Germans would probably prefer to use their software in German...

I cannot know, but they could be ideological. For example, they had found it wonderful to use plain ASCII, no need for special keyboard layouts or something like that, and they decided that German would be much better without its non-ASCII characters. They could believe something like this, and they wouldn't say it aloud in the discussion with you because it is irrelevant for the discussion: you weren't trying to change German.

Re: Charset="WTF-8"

#279

Post by hobs »

Muromec wrote: post #277
hobs wrote: post #269

Earlier quoted context omitted.

Absolutely not - do not build anything based on "would your CSR be able to pronounce" something - that's an awful bar - most CSRs cant pronounce my name - would I be excluded from your database? Seriously, what are you going for here?

That’s the most basic consideration for names, unless you only show it to the user themselves — other people have to be able to read it at least somehow. Which one is why the bag of unicode bytes approach is as wrong as telling Stęphań he has an invalid name.

Absolutely not. There's no way to understand what a source user's reading capability is. There's no way to understand how a person will pronounce their name by simply reading it, this only works for common names.

Re: Charset="WTF-8"

#280

Post by benatkin »

Muromec wrote: post #254
kgeist wrote: post #232

My rule of thumb is to treat strings as opaque blobs most of the time. The only validation I'd always enforce is some sane length limit, to prevent users from shoving entire novels inside. If you treat your strings as opaque blobs, and use UTF8, most of internationalization problems go away. Imho often times, input validation is an attempt to solve a problem from the wrong side. Say, when XSS or SQL injections are fo…

You can treat names as byte blobs for as long as you don't use them for their purpose -- naming people. Suppose you have a unicode blob of my name in your database and there is a problem and you need to call me and say hi. Would your customer representative be able to pronounce my name somewhat correctly? >I think there're very few exceptions to this, probably something law-related, or if you have to interact with so…

> Would your customer representative be able to pronounce my name somewhat correctly?

Worse case, just drop to hexadecimal.

Post reply on HN