Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Charset="WTF-8"

wtf-8.xn--stpie-k0a81a.com

Post reply on HN
Print view View original

431–440 of 463 posts

Re: Charset="WTF-8"

#431

Post by account42 »

sandreas wrote: post #106
MrJohz wrote: post #82

Earlier quoted context omitted.

I worked with an office of Germans who insisted that ASCII was sufficient. The German language uses letters that cannot be represented in ASCII. In fairness, they mostly wanted stuff to be in English, and when necessary, to transliterate German characters into their English counterparts (in German there is a standardised way of doing this), so I can understand why they didn't see it was necessary. I just never unders…

You should have asked how they would encode the german currency sign (€ for euro) in ASCII or its german counterpart latin1/iso-8859-1... It's not possible. However I bet they would argument to use iso-8859-15 (latin9 / latin0) with the international currency sign (¤) instead or insist that char 128 of latin1 is almost always meant as €, so just ignore the standard in these cases and use a new font. This would only f…

UTF-8 is simple, it's Unicode that is complex.

Re: Charset="WTF-8"

#432

Post by account42 »

hooby wrote: post #331
MrJohz wrote: post #82

Earlier quoted context omitted.

I worked with an office of Germans who insisted that ASCII was sufficient. The German language uses letters that cannot be represented in ASCII. In fairness, they mostly wanted stuff to be in English, and when necessary, to transliterate German characters into their English counterparts (in German there is a standardised way of doing this), so I can understand why they didn't see it was necessary. I just never unders…

There are some valid reasons to use software in English as a German speaker. Main among those is probably translations. If you can speak English, you might be better of using the software in English, as having to deal with the English language can often be less of hassle, than having to deal with inconsistent, weird, or outright wrong translations. Even high quality translations might run into issues, where the same…

Compiler errors or low level error messages in general are a good example. Translating them reduces the ability of someone who doesn't share your language to help you.

Re: Charset="WTF-8"

#433

Post by account42 »

MrJohz wrote: post #82
throw310822 wrote: post #35

Earlier quoted context omitted.

I know multiple developers who would just say "well it's their fault, they have to change name then".

I worked with an office of Germans who insisted that ASCII was sufficient. The German language uses letters that cannot be represented in ASCII. In fairness, they mostly wanted stuff to be in English, and when necessary, to transliterate German characters into their English counterparts (in German there is a standardised way of doing this), so I can understand why they didn't see it was necessary. I just never unders…

Perhaps you shouldn't be speaking for Germans then? Personally, I'd rather not have localization forces on me. Looking at you, Google.

Re: Charset="WTF-8"

#434

Post by [deleted] »

chx wrote: post #308
taneliv wrote: post #298

Earlier quoted context omitted.

Not the OP, but immigration often involves changing your name in the way digital systems store and display it. For example, from محمد to Muhammad or from 陳 to Chen. The pronunciation ideally should stay the same, but obviously there's often slight differences. But if the differences are annoying or confusing, someone might choose an entirely different name as well.

Yes but GP said > Where I live, you can only select from a central, though frequently updated, list of names when naming your child I was born in such a country too and still have frequent connections there and I can confirm the laws only apply to citizens of said country so indeed immigration creates exceptions to this rule even if they transliterate their name.

[deleted]

Re: Charset="WTF-8"

#435

Post by account42 »

throw_a_grenade wrote: post #383
Muromec wrote: post #110

Earlier quoted context omitted.

The non-joke answer for Europe is extened Latin, dashes, spaces and apostrophe sign, separated into two (or three) distinct ordered fields. Just because it's written in a different script originally, doesn't mean it will printed only with that on your id in the country of residence or travel document issued at home. My name isn't written in Latin characters and it's fine. I know you can't even try to pronounce them,…

Non-joke answer for Europe is at least Latin, Greek or Cyrillic (български is already one of the official EU languages!). No reason to treat them differently, just don't allow for mixing them so you won't get homoglyphs. EURid (.eu-NIC) gets it mostly right I believe.

The non-theoretical answer for Europe is just Latin because the names need to eventually be read by people who don't know Greek or Cyrillic.

Re: Charset="WTF-8"

#436

Post by account42 »

ajsnigrutin wrote: post #114
ValentinA23 wrote: post #52

Earlier quoted context omitted.

Don't validate names, use transliteration to make them safe for postal services (or whatever). In SQL this is COLLATE, in the command line you can use uconv: >echo "'Lódź'" | uconv -f "UTF-8" -t "UTF-8" -x "Latin-ASCII" >'Lodz'

Yeah, that'll work great.. https://en.wikipedia.org/wiki/%C4%8Celje echo "Čelje" | uconv -f "UTF-8" -t "UTF-8" -x "Latin-ASCII" > "Celje" https://en.wikipedia.org/wiki/Celje (i mean... we do have postal numbers just for problems like this, but both Štefan and Stefan are not-so-uncommon male names over here, so are Jozef and Jožef, etc.)

So? Names are not unique to begin with.

Re: Charset="WTF-8"

#437

Post by Y_Y »

account42 wrote: post #430
Y_Y wrote: post #166

Earlier quoted context omitted.

Weirdly the old Deutsch Mark doesn't seem to have its own code point in the block start U+20A0, whereas the Spanish equivalent (Peseta, ₧, not just Pt) does.

It's not a Unicode issue, there just isn't a dedicated symbol for it, everyone just used the letters DM. Unicode (at least back then) was mostly a superset of existing character sets and then distinct glyphs.

That would be a fine answer, but for the fact that other currencies like the rupee (₨) that are "just letters" do have their own codepoint. Being made up of two symbols doesn't necessarily make something not a symbols, in semiotics or in Unicode.

In fact this is one of the root problems, there are plenty of Unicode symbols you can make out of others, either juxtaposing or overstriking or using a combining character, but this isn't consistently done.

Re: Charset="WTF-8"

#438

Post by account42 »

chrismorgan wrote: post #257
pwdisswordfishz wrote: post #59

Earlier quoted context omitted.

Or unpaired surrogates. Or unassigned code points. Or fullwidth characters. Or "mathematical bold" characters. Though the latter two should be probably solved with NFKC normalization instead.

> Or unpaired surrogates. That’s just an invalid Unicode string, then. Unicode strings are sequences of Unicode scalar values , not code points. > unassigned code points Ah, the tyranny of Unicode version support. I was going to suggest that it could be reasonable to check all code points are assigned at data ingress time, but then you urgently need to make sure that your ingress system always supports the latest ver…

> > Or unpaired surrogates.

> That’s just an invalid Unicode string, then. Unicode strings are sequences of Unicode scalar values, not code points.

Because surrogates were retrofitted onto UCS-2 to make it into UTF-8, they are both code units and (reserved) code points.

Re: Charset="WTF-8"

#439

Post by account42 »

crazygringo wrote: post #55
Seb-C wrote: post #26

Earlier quoted context omitted.

> I'm curious if anyone can think of any other non-alphabetic characters used in legal names around the world, in other scripts? Latin characters are NOT allowed in official names for Japanese citizens. It must be written in Japanese characters only. For foreigners living in Japan it's quite frequent to end up in a situation where their official name in Latin does not pass the validation rules of many forms online. I…

Very interesting about Japan! To be clear, I wasn't thinking about within a specific country though. More like, what is the set of all characters that are allowed in legal names across the world? You know, to eliminate things like emoji, mathematical symbols, and so forth.

I imagine at least Sealand has relatively lax (or at least informal) restrictions.

Re: Charset="WTF-8"

#440

Post by account42 »

malfist wrote: post #12
gmuslera wrote: post #8

Earlier quoted context omitted.

You may not want Bobby Tables in your system.

If you're prohibiting valid letters to protect your database because you didn't parametrize your queries, you're solving the problem from the wrong end

This is all well and good until the company looses real money becaus some other system you are interfacing with got compromised because of your attitude and fingers start being pointed. Defense in depth is a thing.
Post reply on HN