Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code
codeconfessions.substack.com
Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code
1–10 of 101 posts
Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code
#2The author has a number of other good articles going into the depths of CPython here: https://codeconfessions.substack.com/t/cpython-internals.
One of them goes into how many lines of C it takes to execute "a + b" in Python (great explanation, but you have to go to the comments for the answer!). I remember one of my professors opening up gdb and printing every C line to execute "print("hello, world")", and the author's writings give a better understanding of what some of those lines are.
I'd be very interested in a follow-up post going in-depth on the ins and outs of list comprehension.
Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code
#3It is a good explanation of what is happening. However, I think that many of not most Python linters will warn you about this. It is perfectly valid, and something you might actually want, so it should not throw an error.
Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code
#4Using * on a sequence wouldn't get past code review with me, it's an obscure feature with surprising side-effects and it doesn't explain your thinking.
Only context I've ever found it useful in, is if you're unit testing something that accepts a string of max length M, then
bad_str = "A" * (M + 1)
But if you're calling multiply on a list[list[Any]] lol nah mate, what the hell are you playing at, make a for loop and add comments.
Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code
#5TL;DR is that multiplying a list performs a shallow copy of the list's elements. Great article. The only thing it's missing is a snippet that shows what you should do instead:
[[] for _ in range(N)]
Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code
#6I like the deep dive into the source, but big picture, I think if you understand the general order of evaluation which applies to most languages, this isn't all that unintuitive. The arguments going into the outer [ ] get evaluated before the outer [ ], meaning the inner [] gets evaluated first. Once that happens the inner list is created, and there's no code left that even visually could create more inner lists.
The more unintuitive example of this to me is in function defaults, because those feel more like part of the function signature--i.e. I'd expect them to be assigned at function call time. But apparently, function defaults get called once when the function is defined.
>>> def grow(items=[0]):
... items.append(items[-1] + 1)
... return items
...
>>> grow()
[0, 1]
>>> grow()
[0, 1, 2]
>>> grow()
[0, 1, 2, 3]Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code
#7I think anybody that ever did LeetCode/HackerRank in Python had that bug at least once when making an empty 2-D DP table or a return matrix.
My bigger question is how does a professor of machine learning does not know about one of the oldest Python footguns to exist:
https://stackoverflow.com/questions/12791501/why-does-this-c...
Very nice article otherwise, I never actually bothered looking into why it behaved this way(at this point most people initialize using list comprehension, so this bug is IMO very rare in production), very well explained.
Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code
#8I like the deep dive into the source, but big picture, I think if you understand the general order of evaluation which applies to most languages, this isn't all that unintuitive. The arguments going into the outer [ ] get evaluated before the outer [ ], meaning the inner [] gets evaluated first. Once that happens the inner list is created, and there's no code left that even visually could create more inner lists. The…
Well, those things are called "default argument values", not "default argument expressions", you know. It's a value that's reused for every function call, not an expression that re-evaluated at every function call.
Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code
#9TL;DR is that multiplying a list performs a shallow copy of the list's elements. Great article. The only thing it's missing is a snippet that shows what you should do instead: [[] for _ in range(N)] Since generator-like expressions evaluates the inner expression (i.e. the []) on each iteration, the elements of the outer list are guaranteed to be unique references.
This is pretty subtle. I wish more languages explicitly distinguished references and values like C++ and Rust.
Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code
#10I like the deep dive into the source, but big picture, I think if you understand the general order of evaluation which applies to most languages, this isn't all that unintuitive. The arguments going into the outer [ ] get evaluated before the outer [ ], meaning the inner [] gets evaluated first. Once that happens the inner list is created, and there's no code left that even visually could create more inner lists. The…
Well, those things are called "default argument values ", not "default argument expressions ", you know. It's a value that's reused for every function call, not an expression that re-evaluated at every function call.
Sure there is a reason it is that way. But it doesn't mean it's not incredibly unintuitive and error-prone.