Live data from Hacker News

When str.lower() is a security vulnerability in Python

sethmlarson.dev

81–85 of 85 posts

Re: When str.lower() is a security vulnerability in Python

#81

If this is a vulnerability every bug in every API is a vulnerability. This is very spacebar-heating. It's not a vulnerability, it's a bug. A system that used this bug in a way that relied on it to perform a security task would have a vulnerability. We need to stop seeing library functions that are not themselves security systems as having vulnerabilities.

How else are we supposed to have claims that we are finding all the vulnerabilities in the world?

Re: When str.lower() is a security vulnerability in Python

#82
post #28

> The fix was to create new exceptions so that str.lower() would behave as if it was using Unicode 3.2.0 for only particular function. So, we go through each Unicode codepoint and record when the behavior of str.lower() is different when comparing the Unicode version shipped with Python and Unicode 3.2.0 This sounds like a really hacky solution compared to implementing a separate frozen Unicode 3.2.0 lower.

To be clear (because the snippet is non-explanatory). For encode("idna") what they did is use lower() except where it would produce a result different to 3.2.0 and then instead use the result from 3.2.0 instead. Essentially they've frozen the IDNA encoding to be based on 3.2.0 by overriding any changes.

Yeah, that's what I figured, but my worry upon seeing this is "what happens if lower() changes again and people forget to update the list of exceptions?".

Unless they have unit testing on the entire Unicode code space to ensure what they're doing is always identical to 3.2.0.

Re: When str.lower() is a security vulnerability in Python

#83

Earlier quoted context omitted.

What’s a way to flag to an engineering team that they should do a thorough review of their usage of a particular API because it has footguns in it? This is a rhetorical question because there isn’t a generally accepted way of doing so. Automatically patch everything is a silly way to do vulnerability management but software is cheap to change, so it’s often easier at scale to just force engineering teams to patch eve…

You could have a CO which emits overridable warnings or requires additional / specific reviewers. Of course that can then lead to warnings fatigue so it’s not necessarily a big improvement, or an improvement at all, in the long run, depends a lot on the org philosophy and habits.

Warning fatigue is a very good callout. I find it’s really easy to get to a state where you are inundated by notifications that aren’t actionable.

Re: When str.lower() is a security vulnerability in Python

#84
post #4

> This is why calling str.lower() represents a difference in the implementation and the specification, and therefore a vulnerability: I wish there was some explanation how this is a vulnerability and not just a bug generating erroneous data. Vulnerability for me sounds like there’s a reasonable way to create an exploit from the bug, and I don’t see one here as someone who’s not very familiar with the topic.

It creates a parser differential; two different components of the system can treat the same string as different hostnames. Things that have trusted hostnames, or privileged/admin hostnames that are screened out, or SSRF filters all depend on accurately comparing presented hostnames. This is pretty situational, though, isn't it? You still have to be dealing with IDN names.

It's a big deal in anything dealing with permitted/excluded names because once you take the insane C/Posix locale rules into account you end up with nondeterministic comparison results: compare a with b, get a match, another thread changes the locale setting, compare the same a with b, get a non-match. You've also got nontransitive case-changes, e.g. toupper('ı') = 'I', tolower('I') = 'i', 'ı' != 'i'. This means that an attacker can get around any name-based checks by choosing a character set that compares however they want it to, which is why a lot of OSS security projects implement their own guaranteed-deterministic comparison functions, e.g. WireGuard's built-in replacement for the ctype.h macros/functions.

Re: When str.lower() is a security vulnerability in Python

#85
post #4

Earlier quoted context omitted.

It creates a parser differential; two different components of the system can treat the same string as different hostnames. Things that have trusted hostnames, or privileged/admin hostnames that are screened out, or SSRF filters all depend on accurately comparing presented hostnames. This is pretty situational, though, isn't it? You still have to be dealing with IDN names.

IDN is a security flaw masquerading as a standard.

Nice! Although I'd perhaps put it as "IDN is a come-hither sign for attackers masquerading as ...".
Post reply on HN