Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Charset="WTF-8"

wtf-8.xn--stpie-k0a81a.com

Post reply on HN
Print view View original

201–210 of 463 posts

Re: Charset="WTF-8"

#201

Post by kevin_thibedeau »

numpad0 wrote: post #182
wruza wrote: post #127

I'll say it again: this is the consequence of Unicode trying to be a mix of html and docx, instead of a charset. It went too far for an average Joe DevGuy to understand how to deal with it, so he just selects a subset he can handle and bans everything else. HN does that too - special symbols simply get removed. Unicode screwed itself up completely. We wanted a common charset for things like latin, extlatin, cjk, cyri…

IMO, the sin of Unicode is that they didn't just pick local language authorities and gave them standardized concepts like lines and characters, and start-of-language and end-of-language markers. Lots of Unicode issues come from handling languages that the code is not expecting, and codes currently has no means to select or report quirk supports. I suppose they didn't like getting national borders involved in technica…

> start-of-language and end-of-language markers

Unicode used to have language tagging but they've been (mostly) deprecated:

https://en.wikipedia.org/wiki/Tags_(Unicode_block)

https://www.unicode.org/reports/tr7/tr7-1.html

Re: Charset="WTF-8"

#202

Post by cryptonector »

anttihaapala wrote: post #101
ahazred8ta wrote: post #76

Earlier quoted context omitted.

N=2 is common in Việt Nam. (vowel sound + tonal pitch)

Yet Vietnamese can be written in Unicode without any combining characters whatsoever - in NFC normalization each character is one code point - just like the U+1EC7 LATIN SMALL LETTER E WITH CIRCUMFLEX AND DOT BELOW in your example.

u/egypurnash's point was about limiting glyph complexity. You could canonically decompose then look for more than N (say, N=3) combining codepoints in a row and reject if any are found. Canonical forms have nothing to do with actual glyph complexity, but conceptually[0] normalizing first might be a good first step.

[0] I say conceptually because you might implement a form-insensitive Zalgo detector that looks at each non-combining codepoint, looks it up in the Unicode database to find how many combining codepoints one would need if canonically decomposing and call that `n`, then count from there all the following combining codepoints, and if that exceeds `N` then reject. This approach is fairly optimal because most of the time most characters in most strings don't decompose to more than one codepoint, and even if they do you save the cost of allocating a buffer to normalize into and the associated memory stores.

Re: Charset="WTF-8"

#203

Post by cryptonector »

seba_dos1 wrote: post #65
cabirum wrote: post #18

How do I allow "stępień" while detecting Zalgo-isms?

There's nothing special about "Stępień", it has no combining characters, just the usual diacritics that have their own codepoints in Basic Multilingual Plane (U+0119 and U+0144). I bet there are some names out there that would make it harder, but this isn't one.

If you decompose then it uses combining codepoints. Still nothing special.

Re: Charset="WTF-8"

#204

Post by wruza »

meew0 wrote: post #148
wruza wrote: post #127

I'll say it again: this is the consequence of Unicode trying to be a mix of html and docx, instead of a charset. It went too far for an average Joe DevGuy to understand how to deal with it, so he just selects a subset he can handle and bans everything else. HN does that too - special symbols simply get removed. Unicode screwed itself up completely. We wanted a common charset for things like latin, extlatin, cjk, cyri…

The “invisible symbols” are necessary to correctly represent human language. For instance, one of the most infamous Unicode control characters — the right-to-left override — is required to correctly encode mixed Latin and Hebrew text [1], which are both scripts that you mentioned. Besides, ASCII has control characters as well. The “colorful icons” are not part of Unicode. Emoji are just characters like any other. The…

Copy this󠀠󠀼󠀼󠀼󠀠󠁉󠁳󠀠󠁴󠁨󠁩󠁳󠀠󠁮󠁥󠁣󠁥󠁳󠁳󠁡󠁲󠁹󠀠󠁴󠁯󠀠󠁣󠁯󠁲󠁲󠁥󠁣󠁴󠁬󠁹󠀠󠁲󠁥󠁰󠁲󠁥󠁳󠁥󠁮󠁴󠀠󠁨󠁵󠁭󠁡󠁮󠀠󠁬󠁡󠁮󠁧󠁵󠁡󠁧󠁥󠀿󠀠󠀾󠀾󠀾 sentence into this site and click Decode. (YMMW)

https://embracethered.com/blog/ascii-smuggler.html

Re: Charset="WTF-8"

#205

Post by Muromec »

ajsnigrutin wrote: post #152
Muromec wrote: post #129

Earlier quoted context omitted.

Most places where telling Štefan from Stefan is a problem use postal numbers for people too, or/and ask for your DOB.

I don't have a problem from differentiatin Štefan from Stefan, 's' and 'š' sound pretty different to everyone around here. But if someone runs that script above and transliterates "š" to "s" it can cause confusion. And no, we don't use "postal numbers for humans".

>And no, we don't use "postal numbers for humans".

An email, a phone number, a tax or social security number, demographic identifier, billing/contract number or combination of them.

All of those will help you tell Stefan from Štefan in the most practical situations.

>But if someone runs that script above and transliterates "š" to "s" it can cause confusion.

It's not nice, it will certainly make Štefan unhappy, but it's not like you will debit the money from the wrong account or deliver to a different address or contact the wrong customer because of that.

Re: Charset="WTF-8"

#206

Post by teaearlgraycold »

powersnail wrote: post #122

As someone who really think name field should just be one field with any printable unicode characters, I do wonder what the hell would I need to do if I take customer names in this form, and then my system has to interact with some other service that requires first/last name split, and/or [a-zA-Z] validation, like a bank or postal service. Automatic transliteration seems to be very dangerous (wrong name on bank accou…

Legal name vs. display name

Re: Charset="WTF-8"

#207

Post by tomtomtom777 »

perching_aix wrote: post #157
croes wrote: post #3

Earlier quoted context omitted.

Is name validation even possible?

In certain cultures yes. Where I live, you can only select from a central, though frequently updated, list of names when naming your child. So theoretically only (given) names that are on that list can occur. Family names are not part of this, but maybe that exists too elsewhere. I don't know how people whose name has been given to them before this list was established is handled however. An alternative method, which…

This assumes every resident is born and registered in said country which is a silly assumption. Surely, any service only catered only to "naturally born citizen" is discriminatory and illegal?

Re: Charset="WTF-8"

#208

Post by darkhorn »

mkotowski wrote: post #63
poizan42 wrote: post #10

I have an 'æ' in my middle name (formally secondary first name because history reasons). Usually I just don't use it, but it's always funny when a payment form instructs me to write my full name exactly as written on my credit card, and then goes on to tell me my name is invalid.

Still much better when it fails at the first step. I once got myself in a bit of a struggle with Windows 10 by using "ł" as part of Windows username. Amusingly/irritatingly large number of applications, even some of Microsoft's own ones, could not cope with that.

For a similar reason many Java applications do not work in Turkish Windowses. The Turkish İi Iı problem.

Re: Charset="WTF-8"

#209

Post by nicbou »

powersnail wrote: post #122

As someone who really think name field should just be one field with any printable unicode characters, I do wonder what the hell would I need to do if I take customer names in this form, and then my system has to interact with some other service that requires first/last name split, and/or [a-zA-Z] validation, like a bank or postal service. Automatic transliteration seems to be very dangerous (wrong name on bank accou…

We had problems with a Ukrainian refugee we helped because certified translations of her documents did not match. Her name was transliterated the German way in one place and the English way in another.

Those are translations coming from professionals who swore an oath. Don’t try to do it with code.

Re: Charset="WTF-8"

#210

Post by Aeolun »

Analemma_ wrote: post #187
Y_Y wrote: post #160

Earlier quoted context omitted.

I'm happy to discriminate against those damn ancient Sumerians and anyone still using goddamn Linear B.

Sure, but removing those wouldn't make Unicode any simpler, they're just character sets. The GP is complaining about things like combining characters and diacritic modifiers, which make Unicode "ugly" but are necessary if you want to represent real languages used by billions of people.

I mean, we can just make the languages simpler? We can also remove all the hundred different ways to pronounce English sounds. All elementary students will thank you for it xD
Post reply on HN