Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Charset="WTF-8"

wtf-8.xn--stpie-k0a81a.com

Post reply on HN
Print view View original

281–290 of 463 posts

Re: Charset="WTF-8"

#281

Post by lelandbatey »

77pt77 wrote: post #245
kgeist wrote: post #232

My rule of thumb is to treat strings as opaque blobs most of the time. The only validation I'd always enforce is some sane length limit, to prevent users from shoving entire novels inside. If you treat your strings as opaque blobs, and use UTF8, most of internationalization problems go away. Imho often times, input validation is an attempt to solve a problem from the wrong side. Say, when XSS or SQL injections are fo…

> If you treat your strings as opaque blobs, and use UTF8, most of internationalization problems go away This is laughably naive. So many things can go wrong. Strings are not arrays of bytes. There is a price to pay if someone doesn't understand that or chooses to ignore it.

And yet when stored on any computer system, that string will be encoded using some number of bytes. Which you can set a limit on even though you cannot cut, delimit, or make any other inference about that string from the bytes without doing some kind of interpretation. But the bytes limit is enough for the situation the OP is talking about.

Re: Charset="WTF-8"

#282

Post by lmm »

Dylan16807 wrote: post #274
lmm wrote: post #252

Earlier quoted context omitted.

This is a bullshit argument that never gets applied to any other live language. The characters are different, people who actually use them in daily life recognise them as conveying different things. If a thumbs up with a different skin tone is a different character then a different pattern of lines is definitely a different character.

> If a thumbs up with a different skin tone is a different character Is it? The skin tone modifier is serving the same purpose as a variant selector for the CJK codepoint would be.

The underlying implementation mechanism is not the issue. If unicode had actual support for Japanese characters so that when one e.g. converted text from Shift-JIS (in the default, supported way) one could be confident that one's characters would not change into different characters, I wouldn't be complaining, whether the implementation mechanism involved variant selectors or otherwise.

Re: Charset="WTF-8"

#283

Post by Izkata »

cobbzilla wrote: post #47
zarzavat wrote: post #15

Earlier quoted context omitted.

Presumably there aren't any people with control characters in their name, for example.

Watch as someone names themselves the bell character, “^G” (ASCII code 7) [1] When they meet people, they tell them their name is unpronounceable, it’s the sound of a PC speaker from the late 20th century, but you can call them by their preferred nickname “beep”. In paper and online forms they are probably forced to go by the name “BEL”. [1] https://en.wikipedia.org/wiki/Bell_character

It's not exactly a bell, but there are clicks: https://en.wikipedia.org/wiki/Click_consonant

https://www.reddit.com/r/Damnthatsinteresting/comments/1614k...

Re: Charset="WTF-8"

#284

Post by Dylan16807 »

lmm wrote: post #282
Dylan16807 wrote: post #274

Earlier quoted context omitted.

> If a thumbs up with a different skin tone is a different character Is it? The skin tone modifier is serving the same purpose as a variant selector for the CJK codepoint would be.

The underlying implementation mechanism is not the issue. If unicode had actual support for Japanese characters so that when one e.g. converted text from Shift-JIS (in the default, supported way) one could be confident that one's characters would not change into different characters, I wouldn't be complaining, whether the implementation mechanism involved variant selectors or otherwise.

Okay, that's fair. The support for the selectors is very half-assed and there's no other good mechanism.

Re: Charset="WTF-8"

#285

Post by dathinab »

fun fact there is a semi standard encoding called wtf-8 which is utf-8 extended in a way so that it can represent non well formed utf-16 (bad surrogate code points)

it's used in situations like when a utf-8 based system has to interact with Windows file paths

Re: Charset="WTF-8"

#286

Post by shakna »

77pt77 wrote: post #245
kgeist wrote: post #232

My rule of thumb is to treat strings as opaque blobs most of the time. The only validation I'd always enforce is some sane length limit, to prevent users from shoving entire novels inside. If you treat your strings as opaque blobs, and use UTF8, most of internationalization problems go away. Imho often times, input validation is an attempt to solve a problem from the wrong side. Say, when XSS or SQL injections are fo…

> If you treat your strings as opaque blobs, and use UTF8, most of internationalization problems go away This is laughably naive. So many things can go wrong. Strings are not arrays of bytes. There is a price to pay if someone doesn't understand that or chooses to ignore it.

> Strings are not arrays of bytes.

That very much depends on the language that you are using. In some, they are.

Re: Charset="WTF-8"

#287

Post by onionisafruit »

bjackman wrote: post #161
perching_aix wrote: post #157

Earlier quoted context omitted.

In certain cultures yes. Where I live, you can only select from a central, though frequently updated, list of names when naming your child. So theoretically only (given) names that are on that list can occur. Family names are not part of this, but maybe that exists too elsewhere. I don't know how people whose name has been given to them before this list was established is handled however. An alternative method, which…

I still don't see how any system in the real world can safely assume its users only have names from that list. Even if you try to imagine a system for a hospital to register newly born babies... What happens if a pregnant tourist is visiting?

The name a system knows you as doesn’t need to correspond to your legal name or what you are called by others.

Re: Charset="WTF-8"

#288

Post by numpad0 »

tadfisher wrote: post #233
numpad0 wrote: post #195

Earlier quoted context omitted.

They're not. Readers native in one version can't read the other, and there are more than handful that got duplicated in multiple forms, so they're just not same, just similar. You know, obvious presumption underlying Han Unification is that CJK languages must have a continuous dialect continuums, like villagers living in the middle of East China Sea between Shanghai and Nagasaki and Gwangju would speak half-Chinese-J…

You're making the same mistake: the languages are different, but the script is the same (or trivially derived from the Han script). The Ideographic Research Group was well aware of this, having consisted of native speakers of the languages in question.

That's not "mistake", that's the reality. They don't exchange, and they're not the same. "Same or trivially derived" is just a completely false statement that solely exist to justify Han Unification, or maybe something that made sense in the 80s, it doesn't make literal sense.

Re: Charset="WTF-8"

#289

Post by asddubs »

tadfisher wrote: post #158
asddubs wrote: post #151

Earlier quoted context omitted.

>We wanted a common charset for things like latin, extlatin, cjk, cyrillic, hebrew, etc. And we got it, for a while. we didn't even get that because slightly different looking characters from japanese and chinese (and other languages) got merged to be the same character in unicode due to having the same origin, meaning you have to use a font based on the language context for it to display correctly.

They are the same character, though. They do not use the same glyph in different language contexts, but Unicode is a character encoding, not a font standard.

It doesn't matter to me what bullshit semantics theoretical excuse there is, for practical purposes it means that UTF-8 is insufficient for displaying any human language, especially if you want chinese and japanese in the same document/context without switching fonts (like, say, a website)

Re: Charset="WTF-8"

#290

Post by numpad0 »

simonh wrote: post #179
kristopolous wrote: post #156

Earlier quoted context omitted.

There's no argument here. We could say it's only for script and alphabets, ok. It includes many undeciphered writing systems from antiquity with only a small handful of extent samples. Should we keep that, very likely to never be used character set, but exclude the extremely popular emojis? Exclude both? Why? Aren't computers capable enough? I used to be on the anti emoji bandwagon but really, it's all indefensible.…

No, no, no, no, no… So then we’d get ‘the same’ character with potentially infinite different encodings. Lovely. Unicode is a coding system, not a glyph system or font.

macOS already does different encoding for filenames in Japanese than what Windows/Linux do, and I'm sure someone mentioned same situation in Korean here.

Unicode is already a non-deterministic mess.

Post reply on HN