Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

codeconfessions.substack.com

Post reply on HN
Print view View original

41–50 of 101 posts

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#41

Post by bmitc »

magnio wrote: post #37
bmitc wrote: post #14

Earlier quoted context omitted.

You can whitewash this away by discussing order of evaluation, but it still is a great example of how Python continually mixes up concepts of expressions and statements and immutable and mutable data. In real life, if I ask someone to go get me four empty buckets, I expect them to come back with four distinct empty buckets. The intuitive interpretation of [[]]*4 is asking just that. Instead Python creates a bucket an…

> In more sane languages, the expression [] means give me an empty list value, nothing more and nothing less. So then whatever the syntax for [[]]*4 is in such sane languages, it gives you four empty list values. I don't buy this. Either the syntax to give you four empty list values is some kind of comprehension, which Python has and does exactly what you like, or it must have lazy evaluation somehow. Otherwise, how…

My question is: what's a real-world example where this [...]*n syntax buys you something that can't be done simply otherwise and that is worth the complexity tradeoff of people often using it subtly wrong?

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#42

Post by ponco »

EdwardDiego wrote: post #4

Using * on a sequence wouldn't get past code review with me, it's an obscure feature with surprising side-effects and it doesn't explain your thinking. Only context I've ever found it useful in, is if you're unit testing something that accepts a string of max length M, then bad_str = "A" * (M + 1) Is quite nice and also, easy to interpret. But if you're calling multiply on a list[list[Any]] lol nah mate, what the hel…

I feel like far too much attention is given to `[[]] * n`. In the grand scheme of things, no serious python programmer is using multiply on a sequence outside of the string construction convenience.

It's also remarkebly easy to diagnose once you see the unexpected behaviour, so anyone asking for help is going to be instantly told not to do it and use a comprehension instead.

I'm sure there are plenty of other pitfalls in other languages, though I concede this one is especially unintuitive to a new person.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#43

Post by planede »

Joker_vD wrote: post #8
kerkeslager wrote: post #6

I like the deep dive into the source, but big picture, I think if you understand the general order of evaluation which applies to most languages, this isn't all that unintuitive. The arguments going into the outer [ ] get evaluated before the outer [ ], meaning the inner [] gets evaluated first. Once that happens the inner list is created, and there's no code left that even visually could create more inner lists. The…

Well, those things are called "default argument values ", not "default argument expressions ", you know. It's a value that's reused for every function call, not an expression that re-evaluated at every function call.

I think calling it a default argument object would be much more clear. "Value" can still refer to an abstract, immutable concept, even the python docs use it like that at some places.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#44

Post by arijun »

bmitc wrote: post #36
arijun wrote: post #33

Earlier quoted context omitted.

Why are you comparing functions/closures with values? You can do the same in python: [list() for _ in range(4)]

> Why are you comparing functions/closures with values? I'm not. The fun _ -> [] is needed because the argument is the element's index. That's just what List.init happens to expect, a function that takes an index and initializes the element, but the index is not needed here.

Are you saying f# special-cases something that looks like a closure, but isn’t in this one context? Because that, to me, seems like a much more grievous issue than the one at hand.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#45

Post by wruza »

How is that counter-intuitive? Or at least what’s your general intuition then? If it was [x] with x being a complex object, would you expect it to be deep copied? Shallow copied? Copied via IRepeatable? What’s your intuition on a process of repeating a value that isn’t reference-copy? What should a[3] = a[2] = a[1] = a[0] or an equivalent chunk do, assuming trivial __setitem__()? Why that should be different from a * 4?

I’m not even a python guy, if you think I’m biased. These questions are natural for a programmer and a programmer has to ask themselves about what they expect, why, is it consistent, is it safe to assume.

Making [[]]*4 to create three new lists implies that you add a non-obvious ephemeral re-evaluation point in your grammar. Should [[f()]]*4 call f() three more times?

These may be added for convenience. But you have to make sure that syntactic context makes it dead clear. E.g. in a for-i iterator part, loop conditions, function argument init, etc. Locations which are known to be executed more than once. (Python specifically doesn’t even do that for arguments. Every def f(x=[]) shares the same list as a default for x.)

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#46

Post by banish-m4 »

In other realms, SSA and immutable data makes things faster (cache friendly) and more predictable without impure side-effects cascading in unanticipated ways. In general, references are half pointers and quarter gotos in terms of papercut yourself factor.

I think Erlang got this one right.

Perhaps folks replace their containers and fundamental types like strs, lists, and dicts with immutable ones using an appropriate library.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#47

Post by bmitc »

arijun wrote: post #44
bmitc wrote: post #36

Earlier quoted context omitted.

> Why are you comparing functions/closures with values? I'm not. The fun _ -> [] is needed because the argument is the element's index. That's just what List.init happens to expect, a function that takes an index and initializes the element, but the index is not needed here.

Are you saying f# special-cases something that looks like a closure, but isn’t in this one context? Because that, to me, seems like a much more grievous issue than the one at hand.

No. F# lambdas are closures. I was just saying that I'm not using the concept of closures to explain here. It's just a side effect of using the List.init function as a demonstration.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#48

Post by twixfel »

wruza wrote: post #45

How is that counter-intuitive? Or at least what’s your general intuition then? If it was [x] with x being a complex object, would you expect it to be deep copied? Shallow copied? Copied via IRepeatable? What’s your intuition on a process of repeating a value that isn’t reference-copy? What should a[3] = a[2] = a[1] = a[0] or an equivalent chunk do, assuming trivial __setitem__()? Why that should be different from a *…

Indeed at first glance it looks odd, but the alternative is that multiplying a sequence always does a deep copy. Actually what is "strange" is the behaviour in combination with the syntax of a nested list. The behaviour itself is fine. Replace it with

class X: pass

y = [X()] * 5

and suddenly I think the behaviour looks fairly reasonable.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#49

Post by [deleted] »

scbrg wrote: post #12
blt wrote: post #9

Earlier quoted context omitted.

This is pretty subtle. I wish more languages explicitly distinguished references and values like C++ and Rust.

Everything falls in place when you accept that in python all names (including list indexes) are just pointers to an object, and objects aren't copied unless you explicitly copy them. Everything else follows from that. If you had done [1] * 4 you'd ended up with a list of four references to the same object (1) as well. It's just that 1 isn't mutable, so nobody cares.

[deleted]

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#50

Post by arijun »

bmitc wrote: post #36
arijun wrote: post #33

Earlier quoted context omitted.

Why are you comparing functions/closures with values? You can do the same in python: [list() for _ in range(4)]

> Why are you comparing functions/closures with values? I'm not. The fun _ -> [] is needed because the argument is the element's index. That's just what List.init happens to expect, a function that takes an index and initializes the element, but the index is not needed here.

(I can’t respond to your other comment)

Ah I see, you’re saying that the construction of a list in F# mandates a lambda, thereby reducing one possible point of confusion. I could agree with that. But then it comes at a cost of comfort: [0]*4 has less boilerplate than the F# version. So it comes down to trade-offs again.

Post reply on HN