Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Charset="WTF-8"

wtf-8.xn--stpie-k0a81a.com

Post reply on HN
Print view View original

181–190 of 463 posts

Re: Charset="WTF-8"

#181

Post by yreg »

mkotowski wrote: post #62
jplrssn wrote: post #44

Earlier quoted context omitted.

I imagine the “Poland” part of the address would narrow it down somewhat.

I got curious if I can get data to answer that, and it seems so. Based on xlsx from [0], we got the following ??d? localities in Poland: 1 x Bądy, 1 x Brda, 5 x Buda, 120 x Budy, 4 x Dudy, 1 x Dydy, 1 x Gady, 1 x Judy, 1 x Kady, 1 x Kadź, 1 x Łada, 1 x Lady, 4 x Lądy, 2 x Łady, 1 x Lęda, 1 x Lody, 4 x Łódź, 1 x Nida, 1 x Reda, 1 x Redy, 1 x Redz, 74 x Ruda, 8 x Rudy, 12 x Sady, 2 x Zady, 2 x Żydy Certainly quite a lo…

Experienced postal workers most probably know well that ??d? represents a municipality with three non-ascii characters.

Re: Charset="WTF-8"

#182

Post by numpad0 »

wruza wrote: post #127

I'll say it again: this is the consequence of Unicode trying to be a mix of html and docx, instead of a charset. It went too far for an average Joe DevGuy to understand how to deal with it, so he just selects a subset he can handle and bans everything else. HN does that too - special symbols simply get removed. Unicode screwed itself up completely. We wanted a common charset for things like latin, extlatin, cjk, cyri…

IMO, the sin of Unicode is that they didn't just pick local language authorities and gave them standardized concepts like lines and characters, and start-of-language and end-of-language markers.

Lots of Unicode issues come from handling languages that the code is not expecting, and codes currently has no means to select or report quirk supports.

I suppose they didn't like getting national borders involved in technical standardization bit that's just unavoidable. It is getting involved anyway, and these problems are popping up anyway.

Re: Charset="WTF-8"

#183

Post by rvba »

jowea wrote: post #105
pzduniak wrote: post #16

Earlier quoted context omitted.

I live in Łódź. Love receiving packages addressed to ??d? :)

And the packages get there? Don't you put "Łódź (Lodz)" in the city field? Or the postal code takes care of the issue?

You live in a boat? But how do they know on what sea?

Re: Charset="WTF-8"

#184

Post by Muromec »

perching_aix wrote: post #157
croes wrote: post #3

Earlier quoted context omitted.

Is name validation even possible?

In certain cultures yes. Where I live, you can only select from a central, though frequently updated, list of names when naming your child. So theoretically only (given) names that are on that list can occur. Family names are not part of this, but maybe that exists too elsewhere. I don't know how people whose name has been given to them before this list was established is handled however. An alternative method, which…

>So theoretically only (given) names that are on that list can occur.

Unless of course immigration is allowed and doesn't involve changing a name.

Re: Charset="WTF-8"

#185

Post by jeroenhd »

ajsnigrutin wrote: post #114
ValentinA23 wrote: post #52

Earlier quoted context omitted.

Don't validate names, use transliteration to make them safe for postal services (or whatever). In SQL this is COLLATE, in the command line you can use uconv: >echo "'Lódź'" | uconv -f "UTF-8" -t "UTF-8" -x "Latin-ASCII" >'Lodz'

Yeah, that'll work great.. https://en.wikipedia.org/wiki/%C4%8Celje echo "Čelje" | uconv -f "UTF-8" -t "UTF-8" -x "Latin-ASCII" > "Celje" https://en.wikipedia.org/wiki/Celje (i mean... we do have postal numbers just for problems like this, but both Štefan and Stefan are not-so-uncommon male names over here, so are Jozef and Jožef, etc.)

If you're dealing with a bad API that only takes ASCII, "Celje" is usually better than "ÄŒelje" or "蒌elje".

If you have control over the encoding on the input side and on the output side, you should just use UTF-8 or something comparable. If you don't, you have to try to get something useful on the output side.

Re: Charset="WTF-8"

#186

Post by samatman »

zarzavat wrote: post #15
croes wrote: post #3

Earlier quoted context omitted.

Is name validation even possible?

Presumably there aren't any people with control characters in their name, for example.

It's safe to reject Cc, Cn, and Cs. You should probably reject Co as well, even though elves can't input their names if you do that.

Don't reject Cf. That's asking for trouble.

Re: Charset="WTF-8"

#187

Post by Analemma_ »

Y_Y wrote: post #160
meew0 wrote: post #148

Earlier quoted context omitted.

The “invisible symbols” are necessary to correctly represent human language. For instance, one of the most infamous Unicode control characters — the right-to-left override — is required to correctly encode mixed Latin and Hebrew text [1], which are both scripts that you mentioned. Besides, ASCII has control characters as well. The “colorful icons” are not part of Unicode. Emoji are just characters like any other. The…

I'm happy to discriminate against those damn ancient Sumerians and anyone still using goddamn Linear B.

Sure, but removing those wouldn't make Unicode any simpler, they're just character sets. The GP is complaining about things like combining characters and diacritic modifiers, which make Unicode "ugly" but are necessary if you want to represent real languages used by billions of people.

Re: Charset="WTF-8"

#188

Post by josephcsible »

ks2048 wrote: post #167
josephcsible wrote: post #133

What would be wrong with "enter your name as it appears in the machine-readable zone of your passport" (or "would appear" for people who have never gotten one)? Isn't that the one standard format for names that actually is universal?

There's the problem that "appears" is a visible phenomenon and unicode strings can contain non-visible characters and multiple ways to represent the same visible information. Normalization is supposed to help here, but some sites may fail to do this or do incorrectly, etc.

But the MRZ of a passport doesn't contain any of those problem characters.

Re: Charset="WTF-8"

#189

Post by Muromec »

tadfisher wrote: post #158
asddubs wrote: post #151

Earlier quoted context omitted.

>We wanted a common charset for things like latin, extlatin, cjk, cyrillic, hebrew, etc. And we got it, for a while. we didn't even get that because slightly different looking characters from japanese and chinese (and other languages) got merged to be the same character in unicode due to having the same origin, meaning you have to use a font based on the language context for it to display correctly.

They are the same character, though. They do not use the same glyph in different language contexts, but Unicode is a character encoding, not a font standard.

Yes, but the same is true for overlapping characters in Cyrillic and Latin. A and А are the same glyph, so are т,к,і and t,k,i and you can even see the difference between some of those.

Re: Charset="WTF-8"

#190

Post by ivanjermakov »

rvba wrote: post #183
jowea wrote: post #105

Earlier quoted context omitted.

And the packages get there? Don't you put "Łódź (Lodz)" in the city field? Or the postal code takes care of the issue?

You live in a boat? But how do they know on what sea?

Ironically, there are no big rivers in Łódź (anymore)
Post reply on HN