Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Charset="WTF-8"

wtf-8.xn--stpie-k0a81a.com

Post reply on HN
Print view View original

361–370 of 463 posts

Re: Charset="WTF-8"

#361

Post by [deleted] »

jrochkind1 wrote: post #178
wruza wrote: post #127

I'll say it again: this is the consequence of Unicode trying to be a mix of html and docx, instead of a charset. It went too far for an average Joe DevGuy to understand how to deal with it, so he just selects a subset he can handle and bans everything else. HN does that too - special symbols simply get removed. Unicode screwed itself up completely. We wanted a common charset for things like latin, extlatin, cjk, cyri…

Unicode has metadata on each character that would allow software to easily strip out or normalize emoji's and "decorative" characters. It might have edge case problems -- but the charcters in the OP's name would not be included. Also, stripping out emoji's may not actually be required or the right solution. If security is the concern, Unicode also has recommended processes and algorithms for dealing with that. https:…

[deleted]

Re: Charset="WTF-8"

#362

Post by Muromec »

arghwhat wrote: post #322
Muromec wrote: post #254

Earlier quoted context omitted.

You can treat names as byte blobs for as long as you don't use them for their purpose -- naming people. Suppose you have a unicode blob of my name in your database and there is a problem and you need to call me and say hi. Would your customer representative be able to pronounce my name somewhat correctly? >I think there're very few exceptions to this, probably something law-related, or if you have to interact with so…

> Suppose you have a unicode blob of my name in your database and there is a problem and you need to call me and say hi. Would your customer representative be able to pronounce my name somewhat correctly? You cannot pronounce the name regardless of whether it is written in ASCII. Pronouncing a name requires at the very least knowledge of the language it originated in, and attempts at reading it with an English pronun…

>You must store and operate on the person's name as is. Requiring a name modified, or modifying it automatically, is unacceptable

But this is simply not true in practice and at times it's just plain wrong in theory too. The in practice part is trivially discoverable in the real world.

As to in theory -- I do in fact want a properly functioning service to use my name in a vocative case (which requires modifying it automatically or having a dictionary of names) in their communications that are sent in my native language. Not doing that is plainly grammatically wrong and borderline impolite. In fact I use services that do it just right. I also don't want to know to specify the correct version myself, as it's trivially derivable through established rules of the languages.

Re: Charset="WTF-8"

#363

Post by ptx »

gavinsyancey wrote: post #137

WTF-8 is actually a real encoding, used for encoding invalid UTF-16 unpaired surrogates for UTF-8 systems: https://simonsapin.github.io/wtf-8/

Yeah, that had me confused for a bit. And you would never use "charset=wtf-8" (as in the title for this page) because the spec says:

"Any WTF-8 data must be converted to a Unicode encoding at the system’s boundary before being emitted. UTF-8 is recommended. WTF-8 must not be used to represent text in a file format or for transmission over the Internet."

Re: Charset="WTF-8"

#364

Post by throw_a_grenade »

arghwhat wrote: post #322
Muromec wrote: post #254

Earlier quoted context omitted.

You can treat names as byte blobs for as long as you don't use them for their purpose -- naming people. Suppose you have a unicode blob of my name in your database and there is a problem and you need to call me and say hi. Would your customer representative be able to pronounce my name somewhat correctly? >I think there're very few exceptions to this, probably something law-related, or if you have to interact with so…

> Suppose you have a unicode blob of my name in your database and there is a problem and you need to call me and say hi. Would your customer representative be able to pronounce my name somewhat correctly? You cannot pronounce the name regardless of whether it is written in ASCII. Pronouncing a name requires at the very least knowledge of the language it originated in, and attempts at reading it with an English pronun…

Sorry to nitpick, but you underestimated: "many cases" is really "all cases", no exception, because under GDPR you have right to correct your data (this is about legal name, so obviously covered). So if user requests under GDPR art. 16 that his/her name is to be represented in a way that matches ID card or whatever legal document, then you either do it, or you pay a fine and then you do it.

That a particular technical solution is incapable of storing it in the preferred way is not an excuse. EBCDIC is incompatible with GDPR: https://news.ycombinator.com/item?id=28986735

Re: Charset="WTF-8"

#366

Post by jeroenhd »

voidUpdate wrote: post #360
xigoi wrote: post #80

Earlier quoted context omitted.

This seems to only apply to banks.

Apparently it didn't apply to my bank when I tried to change my name with them, according to them anyone could have made my change of name form, and having ID and correctly putting in the PIN for my card wasn't enough

The Belgian DPA had at least one bank pay a fine for their incompetence: https://gdprhub.eu/index.php?title=APD/GBA_(Belgium)_-_141/2...

Depending on how strongly you feel about this, there's a good chance you'll win if you take your bank to court.

Re: Charset="WTF-8"

#367

Post by shaky-carrousel »

arkh wrote: post #309
rafram wrote: post #272

Earlier quoted context omitted.

Sanitizing your strings immediately before display is all well and good until you need to pass them to some piece of third-party software that is very dumb and doesn’t sanitize them. You’ll argue that it’s the vendor’s fault, but the vendor will argue that nobody else allows characters like that in their name inputs! See the Companies House XSS injection situation, where their rationale for forcing a business to chan…

You sanitize at the frontier of what your code controls. Sending data to a database: parametrized queries to sanitize as it is leaving your control. Sending to display to the user: sanitized for a browser Sending to an API: sanitize for whatever rules the API has Sending to a legacy system: sanitize for it Writing a file to the system: sanitize the path The common point is you don't sanitize before you have to send i…

Be liberal in what you accept, and conservative in what you send.

Re: Charset="WTF-8"

#368

Post by hobs »

Muromec wrote: post #357
soco wrote: post #311

Earlier quoted context omitted.

And here we go again, engineers expecting the world should behave fitting their framework du jour. Unfortunately, the real world doesn't care about our engineering bubble and goes on with life - where you can be called !xóõ Kxau or ꦱꦭꦪꦤ or X Æ A-12.

I can be called what I want and in fact I have perfectly reasonable name that doesn't fit neither ASCII nor FN+LN convention. The thing is, your website accepting whatever utf8 blob my name can be serialized to today, without actually understanding it, makes my life worse, not better.

No, it allows an exact representation of your name, it doesn't do anything to your life.

If you dont like your name, either change it or go complain to your parents. They might tell you that you cultural reference point is more important than some person being able to read your name off of a computer screen.

If you want to store a phonetic name for the destination speaker that's not a bad idea, but a name is a name is a name. It is your unique identifier, do not munge it.

Re: Charset="WTF-8"

#369

Post by Muromec »

hobs wrote: post #368
Muromec wrote: post #357

Earlier quoted context omitted.

I can be called what I want and in fact I have perfectly reasonable name that doesn't fit neither ASCII nor FN+LN convention. The thing is, your website accepting whatever utf8 blob my name can be serialized to today, without actually understanding it, makes my life worse, not better.

No, it allows an exact representation of your name, it doesn't do anything to your life. If you dont like your name, either change it or go complain to your parents. They might tell you that you cultural reference point is more important than some person being able to read your name off of a computer screen. If you want to store a phonetic name for the destination speaker that's not a bad idea, but a name is a name i…

But it does affect my life in a way you refuse to understand. That's the problem -- there isn't a true canonical representation of a name (any name really) that fits all practical purposes. Storing a bag of bytes to display back to user is the easiest of practical purposes and suggesting the practice that solve that is worse than rejecting Stępień, it's refusal to understand complexities, that leads to eventually doing the wrong thing and failing your user without even telling them.

>It is your unique identifier, do not munge it.

It's not a good identifier either. Nobody uses names as identifiers at any scale that matters for computers. You can't assume they don't have collisions, you can't tell whether two bags of bytes identify the same person or two different, they aren't even immutable and sometimes are accidentally mutable.

Re: Charset="WTF-8"

#370

Post by rsynnott »

jandrese wrote: post #17
armada651 wrote: post #4

Earlier quoted context omitted.

Yes, it is essential when you want to avoid doing business with customers who have invalid names.

What if your customer is the artist formerly known as Prince or even X Æ A-12 Musk?

Prince is still mostly screwed, even without spurious validation; Unicode doesn't allow personal symbols. Some discussion here: https://www.unicode.org/mail-arch/unicode-ml/Archives-Old/UM...
Post reply on HN