Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Charset="WTF-8"

wtf-8.xn--stpie-k0a81a.com

Post reply on HN
Print view View original

171–180 of 463 posts

Re: Charset="WTF-8"

#171

Post by Y_Y »

bjackman wrote: post #161
perching_aix wrote: post #157

Earlier quoted context omitted.

In certain cultures yes. Where I live, you can only select from a central, though frequently updated, list of names when naming your child. So theoretically only (given) names that are on that list can occur. Family names are not part of this, but maybe that exists too elsewhere. I don't know how people whose name has been given to them before this list was established is handled however. An alternative method, which…

I still don't see how any system in the real world can safely assume its users only have names from that list. Even if you try to imagine a system for a hospital to register newly born babies... What happens if a pregnant tourist is visiting?

For example in Iceland you don't have to name the baby immediately, and the registration times are different for foreign parents.https://www.skra.is/english/people/registration-of-children/...

Of course then you may fall foul of classic falsehood 40: People have names.

Re: Charset="WTF-8"

#172

Post by bawolff »

meew0 wrote: post #148
wruza wrote: post #127

I'll say it again: this is the consequence of Unicode trying to be a mix of html and docx, instead of a charset. It went too far for an average Joe DevGuy to understand how to deal with it, so he just selects a subset he can handle and bans everything else. HN does that too - special symbols simply get removed. Unicode screwed itself up completely. We wanted a common charset for things like latin, extlatin, cjk, cyri…

The “invisible symbols” are necessary to correctly represent human language. For instance, one of the most infamous Unicode control characters — the right-to-left override — is required to correctly encode mixed Latin and Hebrew text [1], which are both scripts that you mentioned. Besides, ASCII has control characters as well. The “colorful icons” are not part of Unicode. Emoji are just characters like any other. The…

> one of the most infamous Unicode control characters — the right-to-left override

You are linking to an RLM not an RLO. Those are different characters. RLO is generally not needed and more special purpose. RLM causes much less problems than RLO.

Really though, i feel like the newer "first strong isolate" character is much better designed and easier to understand then most of the other rtl characters.

Re: Charset="WTF-8"

#173

Post by poincaredisk »

mkotowski wrote: post #62
jplrssn wrote: post #44

Earlier quoted context omitted.

I imagine the “Poland” part of the address would narrow it down somewhat.

I got curious if I can get data to answer that, and it seems so. Based on xlsx from [0], we got the following ??d? localities in Poland: 1 x Bądy, 1 x Brda, 5 x Buda, 120 x Budy, 4 x Dudy, 1 x Dydy, 1 x Gady, 1 x Judy, 1 x Kady, 1 x Kadź, 1 x Łada, 1 x Lady, 4 x Lądy, 2 x Łady, 1 x Lęda, 1 x Lody, 4 x Łódź, 1 x Nida, 1 x Reda, 1 x Redy, 1 x Redz, 74 x Ruda, 8 x Rudy, 12 x Sady, 2 x Zady, 2 x Żydy Certainly quite a lo…

Interestingly, Lady, Łady and Lądy will end up the same after the usual transliteration.

Re: Charset="WTF-8"

#174

Post by arka2147483647 »

mason_mpls wrote: post #162
wruza wrote: post #127

I'll say it again: this is the consequence of Unicode trying to be a mix of html and docx, instead of a charset. It went too far for an average Joe DevGuy to understand how to deal with it, so he just selects a subset he can handle and bans everything else. HN does that too - special symbols simply get removed. Unicode screwed itself up completely. We wanted a common charset for things like latin, extlatin, cjk, cyri…

This frustration seems unnecessary, unicode isnt more complicated than time and we have far more than enough processing power to handle its most absurd manifestations. We just need good libraries, which is a lot less work than inventing yet another system.

The limiting factor is not compute power, but the time and understanding of a random dev somewhere.

Time also is not well understood by most programmers. Most just seem to convert it to epoch and pretend that it is continuous.

Re: Charset="WTF-8"

#175

Post by rossdavidh »

notanote wrote: post #126
imrejonk wrote: post #43

A system not supporting non-latin characters in personal names is pitiful, but a system telling the user that they have an invalid name is outright insulting.

That’s the best one of the lot. "Dein Name ist ungültig", "Your name is invalid", written with the informal word for "your".

They're trying to say that you and the server are very close friends, you see? No, no, I get this is not correct, just a joke...

Re: Charset="WTF-8"

#176

Post by [deleted] »

riwsky wrote: post #164
kristopolous wrote: post #156

Earlier quoted context omitted.

There's no argument here. We could say it's only for script and alphabets, ok. It includes many undeciphered writing systems from antiquity with only a small handful of extent samples. Should we keep that, very likely to never be used character set, but exclude the extremely popular emojis? Exclude both? Why? Aren't computers capable enough? I used to be on the anti emoji bandwagon but really, it's all indefensible.…

Like how phonetic alphabets save space compared to ideograms by just “write the word how it sounds”, the little SVG-icode would just “write the letter how it’s drawn”

[deleted]

Re: Charset="WTF-8"

#177

Post by blahedo »

epcoa wrote: post #69
seba_dos1 wrote: post #66

Earlier quoted context omitted.

It's for when things go wrong. Same as with wire transfers. Nobody checks it unless there's a dispute.

The thing is though that payment networks do in fact do instant verification and it is interesting what gets verified and when. At gas stations it is very common to ask for a zip code (again US), and this is verified immediately to allow the transaction to proceed. I’ve found that when a street address is asked for there is some verification and often a match on the house number is sufficient. Zip codes are verified…

Funny thing about house numbers: they have their own validation problems. For a while I lived in a building whose house number was of the form 123½ and that was an ongoing source of problems. If it just truncated the ½ that was basically fine (the house at 123 didn't have apartment numbers and the postal workers would deliver it correctly) but validating in online forms (twenty-ish years ago) was a challenge. If they ran any validation at all they'd reject the ½, but it was a crapshoot whether which of "123-1/2" or "123 1/2" would work, or sometimes neither one. The USPS's official recommendation at the time was to enter it as "123 1 2 N Streetname" which usually validated but looked so odd it was my last choice (and some validators rejected the "three numbers" format too).

I don't think I ever tried "123.5", actually.

Re: Charset="WTF-8"

#178

Post by jrochkind1 »

wruza wrote: post #127

I'll say it again: this is the consequence of Unicode trying to be a mix of html and docx, instead of a charset. It went too far for an average Joe DevGuy to understand how to deal with it, so he just selects a subset he can handle and bans everything else. HN does that too - special symbols simply get removed. Unicode screwed itself up completely. We wanted a common charset for things like latin, extlatin, cjk, cyri…

Unicode has metadata on each character that would allow software to easily strip out or normalize emoji's and "decorative" characters.

It might have edge case problems -- but the charcters in the OP's name would not be included.

Also, stripping out emoji's may not actually be required or the right solution. If security is the concern, Unicode also has recommended processes and algorithms for dealing with that.

https://www.unicode.org/reports/tr39/

We need better support for the functions developers actually need on unicode in more platforms and languages.

Global human language is complicated as a domain. Legacy issues in actually existing data adds to the complexity. Unicode does a pretty good job at it. It's actually pretty amazing how well it does. Including a lot more than just the character set, and encoding, but algorithms for various kinds of normalizing, sorting, indexing, under various localizations, etc.

It needs better support in the environments more developers are working in, with raised-to-the-top standard solutions for identified common use cases and problems, that can be implemented simply by calling a performance-optimized library function.

(And, if we really want to argue about emoji's, they seem to be extremely popular, and literally have effected global culture, because people want to use them? Blaming emoji's seems like blaming the user! Unicode's support for them actually supports interoperability and vendor-neutral standards for a thing that is wildly popular? but I actually don't think any of the problems or complexity we are talking about, including the OP's complaint, can or should be laid at the feet of emojis)

Re: Charset="WTF-8"

#179

Post by simonh »

kristopolous wrote: post #156
wruza wrote: post #127

I'll say it again: this is the consequence of Unicode trying to be a mix of html and docx, instead of a charset. It went too far for an average Joe DevGuy to understand how to deal with it, so he just selects a subset he can handle and bans everything else. HN does that too - special symbols simply get removed. Unicode screwed itself up completely. We wanted a common charset for things like latin, extlatin, cjk, cyri…

There's no argument here. We could say it's only for script and alphabets, ok. It includes many undeciphered writing systems from antiquity with only a small handful of extent samples. Should we keep that, very likely to never be used character set, but exclude the extremely popular emojis? Exclude both? Why? Aren't computers capable enough? I used to be on the anti emoji bandwagon but really, it's all indefensible.…

No, no, no, no, no… So then we’d get ‘the same’ character with potentially infinite different encodings. Lovely.

Unicode is a coding system, not a glyph system or font.

Re: Charset="WTF-8"

#180

Post by saurik »

RadiozRadioz wrote: post #28

I've got a good feel now for which forms will accept my name and which won't, though mostly I default to an ASCII version for safety. Similarly, I've found a way to mangle my address to fit a US house/state/city/zip format. I don't feel unwelcome, I emphathize with the developers. I'd certainly hate to figure out address entry for all countries. At least the US format is consistent across websites and I can have a hi…

At the end of the day, a postal address is printed to an envelope or package as a single block of text and then read back and parsed somehow by the people delivering the package (usually by a machine most of the way, but even these days more by humans as the package gets closer to the destination). This means that, in a very real sense, the "correct" way to enter an address is into a single giant multi-line text box with the implication that the user must provide whatever is required to be printed onto the mailing label such that a carrier will successfully be able to find them.

Really, then, the reasons why we bother trying to break out an address into multiple parts is not directly caused by the need for an address at all: it is because we 1) might not trust the user to provide for us everything required to make the address valid (assuming the country or even state, or giving us only a street address with no city or postal code... both mistakes that are likely extremely common without a multi-field form), or 2) need to know some subset of the address ourselves and do not trust ourselves to parse back the fuzzy address the same way as the postal service might, either for taxes or to help establish shipping rates.

FWIW, I'd venture to say that #2 is sufficiently common -- as if you need a street address for shipping you are going to need to be careful about sales taxes and VAT, increasingly often even if you aren't located in the state or even country to which the shipment will be made -- that it almost becomes nonsensical to support accepting an address for a location where you aren't already sure of the format convention ahead of time (as that just leads you to only later realizing you failed to collect a tax, will be charged a fortune to ship there, or even that it simply isn't possible to deliver anything to that country)... and like, if you don't intend to ship anything, you actually do not need the full address anyway (credit cards, as an obvious example, don't need or use the full address).

Post reply on HN