Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I couldn't debug the code because of my name

mikolaj-kaminski.com

Post reply on HN
Print view View original

271–280 of 300 posts

Re: I couldn't debug the code because of my name

#271

Post by [deleted] »

geofft wrote: post #138
mjevans wrote: post #116

Earlier quoted context omitted.

This is exactly why I hate the way Python3 handles Unicode. EVERY language should _try_ to handle Unicode such that if a data sequence were valid before it remains valid after. NONE should ever FORCE validation, since sometimes, like in the article's case, the correct answer is GIGO. Just pass it through and hope it continues to work. Sometimes the error is trying to enforce that validation.

Python 3 usually handles this correctly, and I'm a little bit confused what's going on in the article, exactly. For UNIX path names (and other OS data like environment variables), Python uses the "surrogateescape" error handling method, which does exactly what you ask. Any byte sequence can be converted to a string. If it decodes as valid UTF-8, it will do that. If it hits a byte that does not decode as valid UTF-8 (…

[deleted]

Re: I couldn't debug the code because of my name

#272

Post by account42 »

deathanatos wrote: post #196
nightpool wrote: post #143

Earlier quoted context omitted.

The incorrect docker-compose file was generated by Java (Jetbrains) but consumed by Python (docker-compose). The GP comment was complaining about Python's strict Unicode consumption, not Java's invalid Unicode generation.

The Docker compose file is YAML. My reading of YAML's standard is that it must be in one of the Unicode encodings, and the smell I get from the article is that it is probably in windows-1250 (the CP Windows would use for Polish; Mikołaj is a Polish name, 0xb3, the octet in the error, is the Windows-1250 encoding of "ł"); thus, it isn't valid YAML. I'm not sure what sane behavior Python could have here besides errorri…

> This sequence was never valid

Neither were four-byte UTF-8 characters at some point.

> and never will be.

We shall see.

Re: I couldn't debug the code because of my name

#273

Post by account42 »

stubish wrote: post #199
geofft wrote: post #178

Earlier quoted context omitted.

Oh, I see. But if it was UTF-8 it would have worked... I guess the problem is that JetBrains is generating the file in (e.g.) Windows-1252, and Python needs to be told that? Does it work if you set the environment variable PYTHONENCODING to cp1252? (I suppose I should either contact the author, or try it myself...)

JetBrains is generating an invalid YAML file, which are UTF-8. If they were using a decent YAML library, it would have crashed at that point. And firmly pointed the finger at the real bug, reading raw bytes from the environment or a .properties file parser and assuming it is valid UTF-8. And this is why you always validate your data when you slurp it in, or else you pass crap down several layers where it crashes or m…

I'd expect a decent YAML library to have functions taking UTF-8 and not wasting time verifying that the data passed is actually UTF-8 in release builds.

Re: I couldn't debug the code because of my name

#274

Post by kspacewalk2 »

nextaccountic wrote: post #227
kspacewalk2 wrote: post #220

Earlier quoted context omitted.

I find the very idea of putting an exclamation mark on one's username and not expecting eventual problems to be quite curious.

I find the idea of expecting names from other cultures to follow the customs of one's culture to be quite curious. https://shinesolutions.com/2018/01/08/falsehoods-programmers... (but, in any case, this discussion here on HN https://news.ycombinator.com/item?id=18567548 provides some nuance)

It should not cause issues. For regular users, it should be an expectation that, in well-written software, it won't cause issues. But in our world of power+ users... It will cause issues. It just will. Maybe in some number of decades that won't be the case, but it's irrelevant right now.

Re: I couldn't debug the code because of my name

#275

Post by kspacewalk2 »

bluecatswim wrote: post #245
kspacewalk2 wrote: post #220

Earlier quoted context omitted.

I find the very idea of putting an exclamation mark on one's username and not expecting eventual problems to be quite curious.

Really? If someone asked you if exclamation marks in usernames would cause problems before you were aware of this, would you have said yes? It's a very common special character, it's not like it's a control character or some obscure unicode thing. Besides, it's a common thing to add to usernames outside of services where you use your real name.

Absolutely, I would have said yes to the question "do exclamation marks have the potential to cause problems in some piece of software or other, the likelihood going way up with the number and the obscurity of software you use on this machine".

Re: I couldn't debug the code because of my name

#276

Post by account42 »

horsawlarway wrote: post #68
SAI_Peregrinus wrote: post #66

Earlier quoted context omitted.

This is on the Windows version. Windows 2000 is when the OS changed to UTF-16 by default. Before that Windows NT was UCS-2, IIRC only the DOS-based Windows versions were Windows-1252 internally, starting from Windows 1.0. So while ł wasn't supported in Windows 1, characters like ñ were. Windows has literally NEVER been an ASCII-based OS.

Sure, but having used a lot of the windows system apis (admittedly - a lot of years ago) it was a complete hodgepodge of which api would take a char vs a wchar, and then they tried to hide the whole thing behind tchar, which just made it even harder to keep track of. Basically - I agree: This shouldn't be a problem, and 7 months is a long time to wait for a basic fix. But there are a lot of footguns hanging around in…

> Which takes a long pointer to tchar string (LPTSTR) - so this behavior is dependent on the unicode settings of the project at compile time, even today.

The documentation is simply wrong, GetUserProfileDirectoryA which you linked always takes a LPSTR (always "ANSI") while GetUserProfileDirectoryW always takes a LPWSTR (always WTF-16). This is reflected in the function prototype at the top. Only the define GetUserProfileDirectory switches between these two. The define is a compatibility hack and arguably was a mistake, but you can always the W-suffixed function no matter what the project settings are.

Re: I couldn't debug the code because of my name

#277

Post by account42 »

david_allison wrote: post #115
SAI_Peregrinus wrote: post #66

Earlier quoted context omitted.

This is on the Windows version. Windows 2000 is when the OS changed to UTF-16 by default. Before that Windows NT was UCS-2, IIRC only the DOS-based Windows versions were Windows-1252 internally, starting from Windows 1.0. So while ł wasn't supported in Windows 1, characters like ñ were. Windows has literally NEVER been an ASCII-based OS.

> Windows 2000 is when the OS changed to UTF-16 by default. Paths are UTF-16 + unpaired surrogates, so a Windows path isn't legally representable in UTF-8.

But WTF-16 paths can be converted to WTF-8 just fine. You can even use the same algorithm, just only pair surrogates if they are matching an otherwise interpret them as UCS-2 values and encode those normally to "UTF-8".

Re: I couldn't debug the code because of my name

#278

Post by account42 »

GoblinSlayer wrote: post #262
deathanatos wrote: post #205

Earlier quoted context omitted.

Given the frequency with which Windows-12* mojibake occurs, people are either a number of holdouts still using Windows 98 SE, or there are a good number of paths in Windows that still use the non-Unicode encodings.

Windows supports Windows 98 API and it's more natural to use for some languages like C++. No change is planned there. Windows 98 API is also closer to Unix API, which can incentivize the programmer to use the same approach on windows and unix.

All windows needed to do is support setting that API to UTF-8. It's not like it doesn't already support multi-byte encodings. It's not like they even needed to even assign an ID for UTF-8 or implement the conversions - those existed already. All they needed to do is allow programs to set their codepage to UTF-8. This finally became possible two years ago. Better late than never I guess.

Re: I couldn't debug the code because of my name

#279

Post by account42 »

jerf wrote: post #18
supernes wrote: post #2

It's somewhat common to see videogames issue a patch shortly after release where they fix crashes due to non-ASCII Windows usernames or non-English locales. I'm not sure what the root cause of the confusion is, other than text strings being hard in general.

It's easy to think the answer is "just UTF-8 everything" but unfortunately the long and twisty history of filesystems means that's not the correct answer, and the "correct answer" is really hard to write down quickly. If you never display the filename, the answer is to treat existing filenames as bags of bytes, but that breaks down as soon as you need to display them, or if you need to manipulate them by appending un…

> If you never display the filename, the answer is to treat existing filenames as bags of bytes, but that breaks down as soon as you need to display them, or if you need to manipulate them by appending unicode to them, in which case you have to decide on an encoding.

No you don't. On Windows you treat paths as a u16'\' an/or u16'/'-separated sequences of uint16_t. On Unix it's a '/'-separated sequence of bytes. If you want to display, you need to decode, but for display only - so errors should use replacement characters as a graceful failure. For appending you encode your string and then append the bytes. Never do you decode externally provided paths for the purpose of manipulation.

> There's some obscure solutions to this problem, like https://simonsapin.github.io/wtf-8/ (which includes discussion of the 16 bit encodings you need for Windows)

It's relatively new, but has wide enough adoption cosidering - e.g. it's what Rust uses for Windows paths. It's also straightforward - just encode the unmatched surrogate pairs as if they were the corresponding reserved unicode characters using the normal UTF-8 algorithm.

Re: I couldn't debug the code because of my name

#280

Post by account42 »

jan_Inkepa wrote: post #114
supernes wrote: post #2

It's somewhat common to see videogames issue a patch shortly after release where they fix crashes due to non-ASCII Windows usernames or non-English locales. I'm not sure what the root cause of the confusion is, other than text strings being hard in general.

I've been bitten on a few small releases by forgetting that C# localises number->string conversion by default (which makes sense. But if you forget, and you're writing floats to csv files and the decimal points become decimal commas....).

I disagree that having localization for number formatting based on a system setting by default makes sense. Formatted numbers are needed for both human and machine consumption and only one of those can deal with unexpected formatting.
Post reply on HN