Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Charset="WTF-8"

wtf-8.xn--stpie-k0a81a.com

Post reply on HN
Print view View original

301–310 of 463 posts

Re: Charset="WTF-8"

#301

Post by Intermernet »

Muromec wrote: post #254
kgeist wrote: post #232

My rule of thumb is to treat strings as opaque blobs most of the time. The only validation I'd always enforce is some sane length limit, to prevent users from shoving entire novels inside. If you treat your strings as opaque blobs, and use UTF8, most of internationalization problems go away. Imho often times, input validation is an attempt to solve a problem from the wrong side. Say, when XSS or SQL injections are fo…

You can treat names as byte blobs for as long as you don't use them for their purpose -- naming people. Suppose you have a unicode blob of my name in your database and there is a problem and you need to call me and say hi. Would your customer representative be able to pronounce my name somewhat correctly? >I think there're very few exceptions to this, probably something law-related, or if you have to interact with so…

Many Japanese companies require an alternative name entered in half width kana to alleviate this exact problem. Unfortunately, most Japanese websites have a million other UX problems that overshadow this clever solution to the problem.

Re: Charset="WTF-8"

#303

Post by RobotToaster »

cobbzilla wrote: post #47
zarzavat wrote: post #15

Earlier quoted context omitted.

Presumably there aren't any people with control characters in their name, for example.

Watch as someone names themselves the bell character, “^G” (ASCII code 7) [1] When they meet people, they tell them their name is unpronounceable, it’s the sound of a PC speaker from the late 20th century, but you can call them by their preferred nickname “beep”. In paper and online forms they are probably forced to go by the name “BEL”. [1] https://en.wikipedia.org/wiki/Bell_character

I can finally change my name to something that represents my personality: ^G^C

https://en.wikipedia.org/wiki/End-of-Text_character

Re: Charset="WTF-8"

#304

Post by cowsandmilk »

swiftcoder wrote: post #296
dgfitz wrote: post #293

Earlier quoted context omitted.

In this specific case, it seems like your concerns are a hypothetical, no?

Not really, no. A lot of us only really have to deal with English-adjacent input (i.e. European languages that share the majority of character forms with English, or cultures that explicitly Anglicise their names when dealing with English folks). As soon as you have to deal with users with a radically different alphabet/input-method, the wheels tend to come off. Can your CSR reps pronounce names written in Chinese lo…

You can analyze the name and direct a case to a CSR who can handle it. May be unrealistic for a 1-2 person company, but every 20+ person company I’ve worked at has intentionally hired CSRs with different language abilities.

Re: Charset="WTF-8"

#305

Post by numpad0 »

tadfisher wrote: post #234
Muromec wrote: post #189

Earlier quoted context omitted.

Yes, but the same is true for overlapping characters in Cyrillic and Latin. A and А are the same glyph, so are т,к,і and t,k,i and you can even see the difference between some of those.

The duplication there is mostly to remain compatible or trivially transformable with existing encodings. Ironically, the two versions of your example "A" do look different on my device (Android), with a slightly lower x-height for the Cyrillic version.

The irony is you calling it irony. CJK "the same or trivially derived" characters are nowhere close to that yet given same code points. CJK unified ideographs is just broken.

Re: Charset="WTF-8"

#306

Post by Skeime »

rrr_oh_man wrote: post #240
Y_Y wrote: post #171

Earlier quoted context omitted.

For example in Iceland you don't have to name the baby immediately, and the registration times are different for foreign parents. https://www.skra.is/english/people/registration-of-children/... Of course then you may fall foul of classic falsehood 40: People have names.

For today's lucky 10,000: Falsehoods programmers believe about names ( https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-... )

For today's lucky 10,000: Ten Thousand (https://xkcd.com/1053/)

Re: Charset="WTF-8"

#307

Post by wruza »

Analemma_ wrote: post #187
Y_Y wrote: post #160

Earlier quoted context omitted.

I'm happy to discriminate against those damn ancient Sumerians and anyone still using goddamn Linear B.

Sure, but removing those wouldn't make Unicode any simpler, they're just character sets. The GP is complaining about things like combining characters and diacritic modifiers, which make Unicode "ugly" but are necessary if you want to represent real languages used by billions of people.

I’m actually complaining about more “advanced” features like hiding text (see my comment above) or zalgoing it.

And of course endless variations of skin color and gender of three people in a pictogram of a family or something, which is purely a product of a specific subculture that doesn’t have anything in common with text/charset.

If unicode cared about characters, which happens to be an evolving but finite set, it would simply include them all, together with exactly two direction specifiers. Instead it created a language/format/tag system within itself to build characters most of which make zero sense to anyone in the world, except for grapheme linguists, if that job title even exists.

It will eventually overengineer itself into a set bigger than the set of all real characters, if not yet.

Practicality and implications of such system is clearly demonstrated by the $subj.

Re: Charset="WTF-8"

#308

Post by chx »

taneliv wrote: post #298
Muromec wrote: post #184

Earlier quoted context omitted.

>So theoretically only (given) names that are on that list can occur. Unless of course immigration is allowed and doesn't involve changing a name.

Not the OP, but immigration often involves changing your name in the way digital systems store and display it. For example, from محمد to Muhammad or from 陳 to Chen. The pronunciation ideally should stay the same, but obviously there's often slight differences. But if the differences are annoying or confusing, someone might choose an entirely different name as well.

Yes but GP said

> Where I live, you can only select from a central, though frequently updated, list of names when naming your child

I was born in such a country too and still have frequent connections there and I can confirm the laws only apply to citizens of said country so indeed immigration creates exceptions to this rule even if they transliterate their name.

Re: Charset="WTF-8"

#309

Post by arkh »

rafram wrote: post #272
kgeist wrote: post #232

My rule of thumb is to treat strings as opaque blobs most of the time. The only validation I'd always enforce is some sane length limit, to prevent users from shoving entire novels inside. If you treat your strings as opaque blobs, and use UTF8, most of internationalization problems go away. Imho often times, input validation is an attempt to solve a problem from the wrong side. Say, when XSS or SQL injections are fo…

Sanitizing your strings immediately before display is all well and good until you need to pass them to some piece of third-party software that is very dumb and doesn’t sanitize them. You’ll argue that it’s the vendor’s fault, but the vendor will argue that nobody else allows characters like that in their name inputs! See the Companies House XSS injection situation, where their rationale for forcing a business to chan…

You sanitize at the frontier of what your code controls.

Sending data to a database: parametrized queries to sanitize as it is leaving your control.

Sending to display to the user: sanitized for a browser

Sending to an API: sanitize for whatever rules the API has

Sending to a legacy system: sanitize for it

Writing a file to the system: sanitize the path

The common point is you don't sanitize before you have to send it somewhere. And the advantage of this method is that you limit the chances of getting bit by reflected injections. You interrogate some API you don't control, you may just get malicious content, but you sanitize when sending it so all is good. Because you're sanitizing on output and not on input.

Re: Charset="WTF-8"

#310

Post by afiori »

rafram wrote: post #272
kgeist wrote: post #232

My rule of thumb is to treat strings as opaque blobs most of the time. The only validation I'd always enforce is some sane length limit, to prevent users from shoving entire novels inside. If you treat your strings as opaque blobs, and use UTF8, most of internationalization problems go away. Imho often times, input validation is an attempt to solve a problem from the wrong side. Say, when XSS or SQL injections are fo…

Sanitizing your strings immediately before display is all well and good until you need to pass them to some piece of third-party software that is very dumb and doesn’t sanitize them. You’ll argue that it’s the vendor’s fault, but the vendor will argue that nobody else allows characters like that in their name inputs! See the Companies House XSS injection situation, where their rationale for forcing a business to chan…

Forbidding users to use your service to propagate "litte bobby tables" pseudo-pranks is likely a good choice.

The choice is different if like most apps you are almost only a data sink, but if you are also a data source for others it pays to be cautious.

Post reply on HN