Live data from Hacker News

I couldn't debug the code because of my name

mikolaj-kaminski.com

131–140 of 300 posts

Re: I couldn't debug the code because of my name

#131
post #110

Earlier quoted context omitted.

Can Ł have an alternative representation? For example the German ß => ss. Also I think ö can be written as oe. In English we simply shake the big bag of letters, pick a few at random and then throw them at the page until a few stick.

> Can Ł have an alternative representation? Nope. Neither can ź, ć, ś, ą or ę. You can, and people do write them as z, c, s, a and e when writing in a restriced character set, but that is not 'correct' and is not a bijection, ie. „półka” and „polka” mean two different things. There's also the case of technically-same-sounding-especially-recently ż/rz and ó/u (whose replacement would let you get rid of two 'non standa…

I do find this sort of stuff fascinating and also faintly frustrating but of course my mother tongue is (in)famous for being a bit loose at first sight.

According to one of my employees (Polish) Ł sounds roughly like w as in win or water but not as in what. A quick read of this: https://en.wikipedia.org/wiki/%C5%81 doesn't help too much.

Does enforcing Ł instead of say w cause your written language to fail in some way? I don't want to cause offense, I want to understand the causes of difference.

Re: I couldn't debug the code because of my name

#132
post #110

Earlier quoted context omitted.

> Can Ł have an alternative representation? Nope. Neither can ź, ć, ś, ą or ę. You can, and people do write them as z, c, s, a and e when writing in a restriced character set, but that is not 'correct' and is not a bijection, ie. „półka” and „polka” mean two different things. There's also the case of technically-same-sounding-especially-recently ż/rz and ó/u (whose replacement would let you get rid of two 'non standa…

I do find this sort of stuff fascinating and also faintly frustrating but of course my mother tongue is (in)famous for being a bit loose at first sight. According to one of my employees (Polish) Ł sounds roughly like w as in win or water but not as in what. A quick read of this: https://en.wikipedia.org/wiki/%C5%81 doesn't help too much. Does enforcing Ł instead of say w cause your written language to fail in some wa…

'W' in Polish is already used, but for a different sound - it's pronounced like the English 'v'. 'V' in turn is not present the Polish alphabet (in the sense of it not being present in words of Polish origin).

If you wanna change that, you might as well change the entire writing system of the language, eg. to be more in line with some other, more common writing system (ie. other latin alphabets or the cyrillic alphabet which would probably make the most sense phonetically). But no-one's gonna go for that any time soon.

Re: I couldn't debug the code because of my name

#134
post #26
post #9

Using non-ascii characters in file paths, toolchain config files, and other non-display contexts is just asking for trouble, even if it is your name...

This wouldn't have happened if using rust!

Some of the other attempts are a little subtle, this one is a pretty blatant attempt to rile up the folks that are already angry about rust for whatever reason. Please stop.

Re: I couldn't debug the code because of my name

#135

I can very much relate to this but also have very little sympathy here. I have a special character in my name, an apostrophe, and it causes trouble regularly online and with tooling. A number of years ago I decided just to never use it when it came to anything to do with technical work be it email, logins or usernames. Unicode characters are a pain to deal with and I have suffered from it first hand trying to handle…

I'm really surprised someone technically minded thought it's a good idea to put a non ASCII character in their username. I'd never do that.

I'm really surprised someone technically minded thought it's a good idea to not allow non ASCII alphanumerics in a username.

Unicode has been a thing since 1988. Names have included non a-z characters since forever.

Re: I couldn't debug the code because of my name

#136
post #92

For a list of strings that often cause problems to, e.g., add to a test suite, see https://github.com/minimaxir/big-list-of-naughty-strings

It's also important to width-test fields. Never forget to make sure that WWWWWWWWWWWW doesn't cause weird application wrapping.

Related (we do this at my work): https://en.wikipedia.org/wiki/Pseudolocalization

Re: I couldn't debug the code because of my name

#137
post #2

It's somewhat common to see videogames issue a patch shortly after release where they fix crashes due to non-ASCII Windows usernames or non-English locales. I'm not sure what the root cause of the confusion is, other than text strings being hard in general.

It's text encoding confusion: https://en.wikipedia.org/wiki/Mojibake

Re: I couldn't debug the code because of my name

#138
post #101

Very similar problem to one described started my exodus from Google services. I also have non-latin characters in my name however I knew it was always an issue so I never used it in paths etc. At some point, long time ago, I was tasked to do some maintance with Google Cloud service (can't remember the name of the service now) which was doable only through Python CLI utility and it failed with very similar Python erro…

This is exactly why I hate the way Python3 handles Unicode. EVERY language should _try_ to handle Unicode such that if a data sequence were valid before it remains valid after. NONE should ever FORCE validation, since sometimes, like in the article's case, the correct answer is GIGO. Just pass it through and hope it continues to work. Sometimes the error is trying to enforce that validation.

Python 3 usually handles this correctly, and I'm a little bit confused what's going on in the article, exactly.

For UNIX path names (and other OS data like environment variables), Python uses the "surrogateescape" error handling method, which does exactly what you ask. Any byte sequence can be converted to a string. If it decodes as valid UTF-8, it will do that. If it hits a byte that does not decode as valid UTF-8 (necessarily a byte >= 128), it will map it to code points U+DC80 through U+DCFF. These are in a reserved ranges of code points ("surrogates", which make it possible to represent code points > 0xFFFF in UTF-16), and they can't show up in actual Unicode text (i.e., there is no UTF-8 encoding of them, strictly speaking, and if you applied the UTF-8 encoding algorithm to a code point in the U+D800 to U+DFFF range, you would get bytes that aren't valid UTF-8).

On the way out, this is reversed. So you get the results you expect if your filenames are in UTF-8, but since UNIX has no requirement that filenames are indeed UTF-8 (the only constraint is they can't contain NUL or ASCII-forward-slash), the bytes are preserved in a funky-looking format in Python and you get the exact same output on the other end.

See https://www.python.org/dev/peps/pep-0383/ for more on what's going on. The tl;dr for users of Python is that if you want to interact with, say, subprocess output as mostly-normal strings (instead of bytes) but you want to be robust to non-UTF-8 bytes, you should do something like

    subprocess.check_output(["some", "command"], errors="surrogateescape")
You don't need to do this for APIs that directly interact with pathnames, because they do it already. You just need to do it for things like subprocess output and file contents that Python doesn't know you want to handle in this way.

...

On Windows, however, path names must be valid Unicode and are stored in UTF-16. So the idea of a "ł" that doesn't decode properly shouldn't even happen! Mikołaj's home directory ought to be a very boring (and valid) 004d 0069 006b 006f 0142 0061 006a on disk.

Windows doesn't enforce that file paths are valid UTF-16 though (specifically, the surrogate code points are only supposed to show up in a certain way, but nothing enforces that and you can have random surrogates on disk), and hence Rust, which internally represents all strings in UTF-8, has a solution ("WTF-8") that's basically the inverse of surrogateescape - it uses extrapolated-UTF-8-encoding-of-surrogates to handle unpaired surrogates. http://simonsapin.github.io/wtf-8/ But it seems very odd to me that the directory C:\Users\Mikołaj would actually contain any of those, and if it doesn't, I would expect it to very easily turn into a Python Unicode string.

Maybe this is from a Python version before https://www.python.org/dev/peps/pep-0529/ , which is claimed to "fail to round-trip characters outside of the user's active code page"? Maybe this is from a Python version after that change and it's wrong?

Re: I couldn't debug the code because of my name

#139
The solution to this is extremely simple: don't validate usernames, period.

The rationale is from an article someone linked here ("Falsehoods Programmer's Believe About Names"):

> Anything someone tells you is their name is—by definition—an appropriate identifier for them.

If you try to validate by checking for profanity, knowing full well that people can have names that contain profane substrings, I have a tongue-in-check message for you—you are a fucking asshole.

Re: I couldn't debug the code because of my name

#140
post #101

Very similar problem to one described started my exodus from Google services. I also have non-latin characters in my name however I knew it was always an issue so I never used it in paths etc. At some point, long time ago, I was tasked to do some maintance with Google Cloud service (can't remember the name of the service now) which was doable only through Python CLI utility and it failed with very similar Python erro…

This is exactly why I hate the way Python3 handles Unicode. EVERY language should _try_ to handle Unicode such that if a data sequence were valid before it remains valid after. NONE should ever FORCE validation, since sometimes, like in the article's case, the correct answer is GIGO. Just pass it through and hope it continues to work. Sometimes the error is trying to enforce that validation.

How is this Python's fault? It's not like the `docker-compose` file would have worked any better if it silently replaced one of the volumes with an inaccessible file. Instead, you'd just get a failure from the Windows filesystem API when you tried to access or create a file at "C:\\Users\\Miko�aj\\AppData\\Local\\JetBrains\\Rider2021.2\\log\\DebuggerWorker\\\", right?
Post reply on HN