Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

codeconfessions.substack.com

Post reply on HN
Print view View original

11–20 of 101 posts

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#11

Post by Joker_vD »

throwaway0665 wrote: post #10
Joker_vD wrote: post #8

Earlier quoted context omitted.

Well, those things are called "default argument values ", not "default argument expressions ", you know. It's a value that's reused for every function call, not an expression that re-evaluated at every function call.

Sure there is a reason it is that way. But it doesn't mean it's not incredibly unintuitive and error-prone.

Yes, the reason is that it's a tradition to have it this way. Python IMHO has decent enough assignment semantics that using "default argument expressions" model would've been fine, unlike e.g. in C++.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#12

Post by scbrg »

blt wrote: post #9
10000truths wrote: post #5

TL;DR is that multiplying a list performs a shallow copy of the list's elements. Great article. The only thing it's missing is a snippet that shows what you should do instead: [[] for _ in range(N)] Since generator-like expressions evaluates the inner expression (i.e. the []) on each iteration, the elements of the outer list are guaranteed to be unique references.

This is pretty subtle. I wish more languages explicitly distinguished references and values like C++ and Rust.

Everything falls in place when you accept that in python all names (including list indexes) are just pointers to an object, and objects aren't copied unless you explicitly copy them. Everything else follows from that.

If you had done [1] * 4 you'd ended up with a list of four references to the same object (1) as well. It's just that 1 isn't mutable, so nobody cares.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#13

Post by njharman »

kerkeslager wrote: post #6

I like the deep dive into the source, but big picture, I think if you understand the general order of evaluation which applies to most languages, this isn't all that unintuitive. The arguments going into the outer [ ] get evaluated before the outer [ ], meaning the inner [] gets evaluated first. Once that happens the inner list is created, and there's no code left that even visually could create more inner lists. The…

I've never understood peoples confusion about function defaults.

They are part of function definition, they aren't in the function body. You should expect them to be "executed" when the function is defined.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#14

Post by bmitc »

kerkeslager wrote: post #6

I like the deep dive into the source, but big picture, I think if you understand the general order of evaluation which applies to most languages, this isn't all that unintuitive. The arguments going into the outer [ ] get evaluated before the outer [ ], meaning the inner [] gets evaluated first. Once that happens the inner list is created, and there's no code left that even visually could create more inner lists. The…

You can whitewash this away by discussing order of evaluation, but it still is a great example of how Python continually mixes up concepts of expressions and statements and immutable and mutable data. In real life, if I ask someone to go get me four empty buckets, I expect them to come back with four distinct empty buckets. The intuitive interpretation of [[]]*4 is asking just that. Instead Python creates a bucket and then gives you a piece of paper that lists its address four times. Why does one want that, by default?

In more sane languages, the expression [] means give me an empty list value, nothing more and nothing less. So then whatever the syntax for [[]]*4 is in such sane languages, it gives you four empty list values. If you want something more imperative, you can get it, but it most cases, you don't want or even need such things.

Python makes no distinction, at a level able to be visualized by the user, between immutable and mutable data. There are a lot of bizarre situations in Python that will bite you like this. And in most cases, it's strange that such behaviors were chosen as the deault.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#15

Post by Scarblac »

njharman wrote: post #13
kerkeslager wrote: post #6

I like the deep dive into the source, but big picture, I think if you understand the general order of evaluation which applies to most languages, this isn't all that unintuitive. The arguments going into the outer [ ] get evaluated before the outer [ ], meaning the inner [] gets evaluated first. Once that happens the inner list is created, and there's no code left that even visually could create more inner lists. The…

I've never understood peoples confusion about function defaults. They are part of function definition, they aren't in the function body. You should expect them to be "executed" when the function is defined.

Or they could have been part of the function call, and run when the function is called without that argument.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#16

Post by maweki »

EdwardDiego wrote: post #4

Using * on a sequence wouldn't get past code review with me, it's an obscure feature with surprising side-effects and it doesn't explain your thinking. Only context I've ever found it useful in, is if you're unit testing something that accepts a string of max length M, then bad_str = "A" * (M + 1) Is quite nice and also, easy to interpret. But if you're calling multiply on a list[list[Any]] lol nah mate, what the hel…

> Using * on a sequence wouldn't get past code review

On a mutable sequence. Or anything mutable. [] being a sequence is not the issue here.

If you work with immutable stuff, * is really not an issue.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#17

Post by Joker_vD »

bmitc wrote: post #14
kerkeslager wrote: post #6

I like the deep dive into the source, but big picture, I think if you understand the general order of evaluation which applies to most languages, this isn't all that unintuitive. The arguments going into the outer [ ] get evaluated before the outer [ ], meaning the inner [] gets evaluated first. Once that happens the inner list is created, and there's no code left that even visually could create more inner lists. The…

You can whitewash this away by discussing order of evaluation, but it still is a great example of how Python continually mixes up concepts of expressions and statements and immutable and mutable data. In real life, if I ask someone to go get me four empty buckets, I expect them to come back with four distinct empty buckets. The intuitive interpretation of [[]]*4 is asking just that. Instead Python creates a bucket an…

I really wonder what "more sane" languages you mean. Because a single [] does give you an empty list value in Python: one list, and not four different empty lists. Why would it give your four? That'd need to somehow bring in the surrounding context and in "sane" languages, (closed) expressions generally don't change the meaning because the context around them is changing.

Actually, if you really want to have re-evaluting (sub)expressions, then Python does give you an option: comprehensions.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#18

Post by bmitc »

mamonster wrote: post #7

I think anybody that ever did LeetCode/HackerRank in Python had that bug at least once when making an empty 2-D DP table or a return matrix. My bigger question is how does a professor of machine learning does not know about one of the oldest Python footguns to exist: https://stackoverflow.com/questions/12791501/why-does-this-c... Very nice article otherwise, I never actually bothered looking into why it behaved this…

> My bigger question is how does a professor of machine learning does not know about one of the oldest Python footguns to exist

Because there's a lot of them. And as soon as you touch a sane language that fits your intuitive model of how things should work, going back to Python means that you'll start doing intuitive things that Python will happily and silently transform into unintuitive operations.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#19

Post by drones »

10000truths wrote: post #5

TL;DR is that multiplying a list performs a shallow copy of the list's elements. Great article. The only thing it's missing is a snippet that shows what you should do instead: [[] for _ in range(N)] Since generator-like expressions evaluates the inner expression (i.e. the []) on each iteration, the elements of the outer list are guaranteed to be unique references.

I only use the [x] * n form of array initialization for primitive types (int,float,bool,string). So

  #good
  a = [True] * 8 # bool is primitive
  b = [[True] * 8 for i in range(8)] # range creates new lists each time

  #bad
  a = [{}] * 8 # dict is not a primitive
  b = [[True] * 8] * 8 # list is not a primitive

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#20

Post by bmitc »

blt wrote: post #9
10000truths wrote: post #5

TL;DR is that multiplying a list performs a shallow copy of the list's elements. Great article. The only thing it's missing is a snippet that shows what you should do instead: [[] for _ in range(N)] Since generator-like expressions evaluates the inner expression (i.e. the []) on each iteration, the elements of the outer list are guaranteed to be unique references.

This is pretty subtle. I wish more languages explicitly distinguished references and values like C++ and Rust.

F# does this nicely with the mutable keyword and the <- assignment operator.
Post reply on HN