Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Charset="WTF-8"

wtf-8.xn--stpie-k0a81a.com

Post reply on HN
Print view View original

291–300 of 463 posts

Re: Charset="WTF-8"

#291

Post by kgeist »

Muromec wrote: post #254
kgeist wrote: post #232

My rule of thumb is to treat strings as opaque blobs most of the time. The only validation I'd always enforce is some sane length limit, to prevent users from shoving entire novels inside. If you treat your strings as opaque blobs, and use UTF8, most of internationalization problems go away. Imho often times, input validation is an attempt to solve a problem from the wrong side. Say, when XSS or SQL injections are fo…

You can treat names as byte blobs for as long as you don't use them for their purpose -- naming people. Suppose you have a unicode blob of my name in your database and there is a problem and you need to call me and say hi. Would your customer representative be able to pronounce my name somewhat correctly? >I think there're very few exceptions to this, probably something law-related, or if you have to interact with so…

>Would your customer representative be able to pronounce my name somewhat correctly?

Typical input validation doesn't really solve the problem. For instance, I could enter my name as 'Vrdtpsk,' which is a perfectly valid ASCII string that passes all validation rules, but no one would be able to pronounce it correctly. I believe the representative (if on a call) should simply ask the customer how they would like to be addressed. Unless we want to implement a whitelist of allowed names for customers to choose from...

Re: Charset="WTF-8"

#292

Post by tugu77 »

ddulaney wrote: post #241
kgeist wrote: post #232

My rule of thumb is to treat strings as opaque blobs most of the time. The only validation I'd always enforce is some sane length limit, to prevent users from shoving entire novels inside. If you treat your strings as opaque blobs, and use UTF8, most of internationalization problems go away. Imho often times, input validation is an attempt to solve a problem from the wrong side. Say, when XSS or SQL injections are fo…

There's at least one major exception to this: Unicode normalization. It's possible for the same logical character to have two different sets of code points (for example, a-with-umlaut as a single character, vs a followed by umlaut combining diacritic). Related, distinguishing between the "a" character in Latin, Greek, Cyrillic, and the handful of other times it shows up throughout Unicode. This comes up in at least 3…

Type system ftw? As long as it's a blob (unnormalized), it should have a blob type which can do very little besides storing and retrieving, perhaps printing. Only the normalized version should be even comparable.

Re: Charset="WTF-8"

#293

Post by dgfitz »

Muromec wrote: post #273
kmoser wrote: post #256

Earlier quoted context omitted.

> Would your customer representative be able to pronounce my name somewhat correctly? Are you implying the CSR's lack of familiarity with the pronunciation of your name means your name should be stored/rendered incorrectly?

Quite the opposite actually. I want it stored correctly and in a way that both me and CSR can understand and so it can be used to interface with other systems. I don’t however know which unicode subset to use, because you didn’t tell me in the signup form. I have many options, all of them correct, but I don’t know whether your CSR can read Ukrainian Cyrillic and whether you can tell what vocative case is and not use…

In this specific case, it seems like your concerns are a hypothetical, no?

Re: Charset="WTF-8"

#294

Post by tugu77 »

Y_Y wrote: post #166
asddubs wrote: post #154

Earlier quoted context omitted.

or just double all the numbers and use DM

Weirdly the old Deutsch Mark doesn't seem to have its own code point in the block start U+20A0, whereas the Spanish equivalent (Peseta, ₧, not just Pt) does.

TIL

https://www.compart.com/en/unicode/block/U+20A0

Even Bitcoin is there. And "German Penny Sign"?

Re: Charset="WTF-8"

#295

Post by manarth »

kgeist wrote: post #291
Muromec wrote: post #254

Earlier quoted context omitted.

You can treat names as byte blobs for as long as you don't use them for their purpose -- naming people. Suppose you have a unicode blob of my name in your database and there is a problem and you need to call me and say hi. Would your customer representative be able to pronounce my name somewhat correctly? >I think there're very few exceptions to this, probably something law-related, or if you have to interact with so…

>Would your customer representative be able to pronounce my name somewhat correctly? Typical input validation doesn't really solve the problem. For instance, I could enter my name as 'Vrdtpsk,' which is a perfectly valid ASCII string that passes all validation rules, but no one would be able to pronounce it correctly. I believe the representative (if on a call) should simply ask the customer how they would like to be…

Derek would like a word.

https://www.youtube.com/watch?v=hNoS2BU6bbQ

Re: Charset="WTF-8"

#296

Post by swiftcoder »

dgfitz wrote: post #293
Muromec wrote: post #273

Earlier quoted context omitted.

Quite the opposite actually. I want it stored correctly and in a way that both me and CSR can understand and so it can be used to interface with other systems. I don’t however know which unicode subset to use, because you didn’t tell me in the signup form. I have many options, all of them correct, but I don’t know whether your CSR can read Ukrainian Cyrillic and whether you can tell what vocative case is and not use…

In this specific case, it seems like your concerns are a hypothetical, no?

Not really, no. A lot of us only really have to deal with English-adjacent input (i.e. European languages that share the majority of character forms with English, or cultures that explicitly Anglicise their names when dealing with English folks).

As soon as you have to deal with users with a radically different alphabet/input-method, the wheels tend to come off. Can your CSR reps pronounce names written in Chinese logographs? In Arabic script? In the Hebrew alphabet?

Re: Charset="WTF-8"

#297

Post by epcoa »

poizan42 wrote: post #124
jjmarr wrote: post #89

Earlier quoted context omitted.

At American gas stations, if you have a Canadian credit card, you type in 00000 because Canadians don't have ZIP codes.

Are we sure they don't actually validate against a more generic postal code field? Then again some countries have letters in their postcodes (the UK comes to mind), so that might be a problem anyways.

Canada has letters in postal codes. That’s the issue the GP is referring to, since US gas stations invariably just have a simple 5 numeric digit input for “zip” code.

Re: Charset="WTF-8"

#298

Post by taneliv »

Muromec wrote: post #184
perching_aix wrote: post #157

Earlier quoted context omitted.

In certain cultures yes. Where I live, you can only select from a central, though frequently updated, list of names when naming your child. So theoretically only (given) names that are on that list can occur. Family names are not part of this, but maybe that exists too elsewhere. I don't know how people whose name has been given to them before this list was established is handled however. An alternative method, which…

>So theoretically only (given) names that are on that list can occur. Unless of course immigration is allowed and doesn't involve changing a name.

Not the OP, but immigration often involves changing your name in the way digital systems store and display it. For example, from محمد to Muhammad or from 陳 to Chen. The pronunciation ideally should stay the same, but obviously there's often slight differences. But if the differences are annoying or confusing, someone might choose an entirely different name as well.

Re: Charset="WTF-8"

#299

Post by wruza »

n2d4 wrote: post #238
wruza wrote: post #204

Earlier quoted context omitted.

Copy this󠀠󠀼󠀼󠀼󠀠󠁉󠁳󠀠󠁴󠁨󠁩󠁳󠀠󠁮󠁥󠁣󠁥󠁳󠁳󠁡󠁲󠁹󠀠󠁴󠁯󠀠󠁣󠁯󠁲󠁲󠁥󠁣󠁴󠁬󠁹󠀠󠁲󠁥󠁰󠁲󠁥󠁳󠁥󠁮󠁴󠀠󠁨󠁵󠁭󠁡󠁮󠀠󠁬󠁡󠁮󠁧󠁵󠁡󠁧󠁥󠀿󠀠󠀾󠀾󠀾 sentence into this site and click Decode. (YMMW) https://embracethered.com/blog/ascii-smuggler.html

> Is this necessary to correctly represent human language? Yes! As soon as you have any invisible characters (eg. RTL or LTR marks, which are required to represent human language), you will be able to encode any data you want.

How many direction marks can we see in this hidden text?

Re: Charset="WTF-8"

#300

Post by rob74 »

rafram wrote: post #272
kgeist wrote: post #232

My rule of thumb is to treat strings as opaque blobs most of the time. The only validation I'd always enforce is some sane length limit, to prevent users from shoving entire novels inside. If you treat your strings as opaque blobs, and use UTF8, most of internationalization problems go away. Imho often times, input validation is an attempt to solve a problem from the wrong side. Say, when XSS or SQL injections are fo…

Sanitizing your strings immediately before display is all well and good until you need to pass them to some piece of third-party software that is very dumb and doesn’t sanitize them. You’ll argue that it’s the vendor’s fault, but the vendor will argue that nobody else allows characters like that in their name inputs! See the Companies House XSS injection situation, where their rationale for forcing a business to chan…

> but the vendor will argue that nobody else allows characters like that in their name inputs

...and maybe they will even link to this page to support that statement! But, seeing that most of the pages are German, I bet they do accept the usual German "special" letters (ÄÖÜß) in names?

Post reply on HN