Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Charset="WTF-8"

wtf-8.xn--stpie-k0a81a.com

Post reply on HN
Print view View original

391–400 of 463 posts

Re: Charset="WTF-8"

#391

Post by tadfisher »

numpad0 wrote: post #288
tadfisher wrote: post #233

Earlier quoted context omitted.

You're making the same mistake: the languages are different, but the script is the same (or trivially derived from the Han script). The Ideographic Research Group was well aware of this, having consisted of native speakers of the languages in question.

That's not "mistake", that's the reality. They don't exchange, and they're not the same. "Same or trivially derived" is just a completely false statement that solely exist to justify Han Unification, or maybe something that made sense in the 80s, it doesn't make literal sense.

> "Same or trivially derived" is just a completely false statement

You'd have to ignore a lot of reality to believe this. It's even in the names of the writing systems: Kanji, Hanja, Chữ Hán. Of course they don't exchange, because they don't carry the same meaning, just as the word "chat" means completely different things in French and English. But it is literally the same script, albeit with numerous stylistic differences and simplified forms.

Re: Charset="WTF-8"

#392

Post by int_19h »

throw0101a wrote: post #354
MrJohz wrote: post #82

Earlier quoted context omitted.

I worked with an office of Germans who insisted that ASCII was sufficient. The German language uses letters that cannot be represented in ASCII. In fairness, they mostly wanted stuff to be in English, and when necessary, to transliterate German characters into their English counterparts (in German there is a standardised way of doing this), so I can understand why they didn't see it was necessary. I just never unders…

> I just never understood why I, as the non-German, was forever the one trying to convince them that Germans would probably prefer to use their software in German... I've heard that German is often one of the first localizations of (desktop) software because there were often super-long words in the translations of various concepts, so if you wanted to test typeface rendering and menu breakage it was good language to…

Or you use pseudo-localization, which does simple programmatic substitution to make all English strings longer by e.g. doubling letters or inserting random non-alphabetic characters, adding diacritics etc while still retaining readability to English speakers.

Windows actually ships with a locale like that.

Re: Charset="WTF-8"

#393

Post by Symbiote »

pzduniak wrote: post #16
poizan42 wrote: post #10

I have an 'æ' in my middle name (formally secondary first name because history reasons). Usually I just don't use it, but it's always funny when a payment form instructs me to write my full name exactly as written on my credit card, and then goes on to tell me my name is invalid.

I live in Łódź. Love receiving packages addressed to ??d? :)

It goes both ways.

I received a large delivery from Poland recently, and it was addressed to "K?benhavn", with similar mistakes in the street name.

Re: Charset="WTF-8"

#394

Post by int_19h »

aguaviva wrote: post #271
aguaviva wrote: post #255

Earlier quoted context omitted.

It's not a Russian-specific thing by any stretch. This happens all the time when names and loanwords get dragged across linguistic boundaries. Sometimes it results from an attempt to "simplify" the respective spelling and/or sounds (by mapping them into tokens more familiar in the local environment); sometimes there's a more complex process behind it; and other times it just happens for various obscure historical rea…

Correction - it's nothing osbcure at all, but apparently a matter of the shift that accord broadly with the L sound in Polish a few centuries ago (whereby it became "dark" and velarized), affecting a great many other words and names (like słowo, mały, etc). While in parts east and south the "clear" L sound was preserved. https://en.wikipedia.org/wiki/Ł

Velarized L is a common phoneme in Slavic languages, inherited from their common ancestor. What makes Polish somewhat unusual is that the pronunciation of velarized L eventually shifted to /w/ pretty much everywhere (a similar process happened in Ukrainian and Belarusian, but only in some contexts).

Re: Charset="WTF-8"

#395

Post by int_19h »

Muromec wrote: post #141
pavel_lishin wrote: post #135

Earlier quoted context omitted.

I just looked up the Russian wikipedia entry for it, and it's spelled "Лодзь", but it sounds like it's pronounced "Вуджь", and this fact irritates the hell out of me. Why would it be transliterated with an Л? And an О? And a з? None of this makes sense.

It's a general pattern of what russia does to names of places and people, which is aggressively imposing their own cultural paradigm (which follows the more general general pattern). You can look up your civil code provisions around names and ask a question or two of what historical problem they attempt to solve.

Adapting foreign names to phonotactics and/or spelling practices of one's native language is a common practice throughout the world. The city's name is spelled Lodz in Spanish, for example.

Re: Charset="WTF-8"

#396

Post by int_19h »

nicbou wrote: post #209
powersnail wrote: post #122

As someone who really think name field should just be one field with any printable unicode characters, I do wonder what the hell would I need to do if I take customer names in this form, and then my system has to interact with some other service that requires first/last name split, and/or [a-zA-Z] validation, like a bank or postal service. Automatic transliteration seems to be very dangerous (wrong name on bank accou…

We had problems with a Ukrainian refugee we helped because certified translations of her documents did not match. Her name was transliterated the German way in one place and the English way in another. Those are translations coming from professionals who swore an oath. Don’t try to do it with code.

This actually sounds like the translator didn't do their job properly. Most countries have detailed official guidelines on how to transliterate names into Latin script, because they need that to issue internationally recognized passports. For Ukrainian, the relevant standard is https://zakon.rada.gov.ua/laws/show/55-2010-%D0%BF

The real problem here is that transliteration rules depend on the source language, and you often don't have the context to distinguish that (e.g. the name can be spelled identically in Russian, Bulgarian, and Ukrainian, yet transliterations will be different). And even where you do have sufficient context, this still requires the system to be able to handle any source language that can be potentially thrown at it, which is just about anything. So it never gets implemented properly in practice.

Re: Charset="WTF-8"

#397

Post by kazinator »

kgeist wrote: post #232

My rule of thumb is to treat strings as opaque blobs most of the time. The only validation I'd always enforce is some sane length limit, to prevent users from shoving entire novels inside. If you treat your strings as opaque blobs, and use UTF8, most of internationalization problems go away. Imho often times, input validation is an attempt to solve a problem from the wrong side. Say, when XSS or SQL injections are fo…

> treat strings as opaque blobs most of the time

While being transparent is great and better than stupidly mangling or rejecting data, the problem is that if we just pass through anything, there are situations and contexts in which some kinds of software could be used as part of a chain of deception involving Unicode/font tricks.

Passing through is mostly good. Even in software that has to care about this, not every layer through which text passes should have the responsibility.

Re: Charset="WTF-8"

#398

Post by int_19h »

hedora wrote: post #267
mmoskal wrote: post #242

Earlier quoted context omitted.

Abstract WTF-8 (Wobbly Transformation Format − 8-bit) is a superset of UTF-8 that encodes surrogate code points if they are not in a pair.

Can you still assume the bytes 0x00 and 0xFF are not present in the string (like in UTF-8?)

Yes. The only difference between UTF-8 and WTF-8 is that the latter does not reject otherwise valid UTF-8 byte sequences that correspond to codepoints in range U+D800 to U+DFFF (which means that, in practice, a lot of things that say they are UTF-8 are actually WTF-8).

Re: Charset="WTF-8"

#399

Post by int_19h »

wruza wrote: post #307
Analemma_ wrote: post #187

Earlier quoted context omitted.

Sure, but removing those wouldn't make Unicode any simpler, they're just character sets. The GP is complaining about things like combining characters and diacritic modifiers, which make Unicode "ugly" but are necessary if you want to represent real languages used by billions of people.

I’m actually complaining about more “advanced” features like hiding text (see my comment above) or zalgoing it. And of course endless variations of skin color and gender of three people in a pictogram of a family or something, which is purely a product of a specific subculture that doesn’t have anything in common with text/charset. If unicode cared about characters, which happens to be an evolving but finite set, it…

"Zalgoing" text is just piling up combining marks, but there are plenty of real-world languages that require more than one combining mark per character to be properly spelled. Vietnamese is a rather extreme example.

Re: Charset="WTF-8"

#400

Post by int_19h »

Aeolun wrote: post #321
scripturial wrote: post #218

Earlier quoted context omitted.

You can make a language simpler but old books still exist. I guess if we burn all old books and disallow a means to print these old books again, people would be happy?

Reprint them with new spelling? We have 500 year old books that are unreadable. 99.99% of all books published will not be relevant to anyone that isn’t consuming them right at that moment anyway. Lovers can read the lord of the rings in the ‘original’ spelling.

The point is that you still want the universal encoding to be able to represent such texts.
Post reply on HN