Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

codeconfessions.substack.com

Post reply on HN
Print view View original

61–70 of 101 posts

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#61

Post by mypalmike »

bmitc wrote: post #40
shiandow wrote: post #35

Earlier quoted context omitted.

In that case what would you expect the following to return? a = [] b = [a]*4 a.append(1) print(b) I mean it is possible to change Python and make [...]*n a shorthand for "repeat this statement n times" but that would be odd on its own. Lists pass by reference, tuples pass by value, if you want to understand Python you're just going to have to get used to that distinction. There are other options, but most are worse.

> , if you want to understand Python you're just going to have to get used to that distinction My argument is that that distinction isn't needed, by default, in a high-level language, and that plenty of people get it wrong, understandably. It is simply complexity that is not needed. These simple examples can be toyed with, but sooner or later one is going to want to solve real problems, and unintuitive, default behav…

It's really not that complex, but it does have to be learned.

Everything in Python is an object with an id. Assignment and parameter passing is by object id.

There are some mutable objects ("objects", lists, dicts) and some immutable objects (numbers, strings, tuples).

The []*n syntax is treated as assignment, hence the id is duplicated. This isn't an issue with immutable objects since there's no way to change the underlying object - you can only change the id of each object in the list. Obviously there are some gotchas like with mutable objects here where the succinct syntax can be confusing. But I think the simplicity of everything being an id has deep intuitive value and is one reason Python has been so successful, even if many users never really think much about ids.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#62

Post by mmaniac »

The root problem here is how references and mutability combine to cause spooky action at a distance, and the non-obviousness of how they combine.

The higher-level a programming language is, the more likely it is to hide the use of references, and also the more likely it is to hide when a new object is created, which would break shared mutable state.

In C or Go any reference is taken explicitly and objects are also created explicitly, so it's very easy to see at a glance what can be mutated from outside the immediate scope or when a defensive copy is made.

Java and C# use lots of implicit references but still require keyword new for nearly all object instantiations. It's easy to look at the expression Collections.nCopies(4, new ArrayList()) and understand that the same list will be referenced 4 times.

At the highest extreme you have languages like Haskell where immutability makes details about when objects are copied or referenced irrelevant. Surprising behaviour is outright impossible now.

Python appears to be in a tarpit where object instantiation and referencing is obscured just enough that it can take you by surprise when it silently results in shared mutable state.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#63

Post by zarzavat »

bmitc wrote: post #28
thomasahle wrote: post #27

Earlier quoted context omitted.

> I expect them to come back with four distinct empty buckets. The intuitive interpretation of [[]]*4 is asking just that. I think the small example is misguiding the intuition here. If I write `my_list = f(expensive operation); long_list = my_list * 4`, I don't expect python to recreate the whole thing four times. I justly assume `f` will be evaluated once, and then copied.

That's a different statement and expected, but in a language like Python, it isn't always clear whether my_list is a value or a reference. And because of that, it leads to unexpected behavior when an intuitively expected value is actually a reference.

Lists in Python are always references.

There’s a separate type called tuple specifically designed to represent lists-as-values.

If you do [()]*4 then you get exactly what you want.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#64

Post by abhi9u »

pbrowne011 wrote: post #2

The author has a number of other good articles going into the depths of CPython here: https://codeconfessions.substack.com/t/cpython-internals . One of them goes into how many lines of C it takes to execute "a + b" in Python (great explanation, but you have to go to the comments for the answer!). I remember one of my professors opening up gdb and printing every C line to execute "print("hello, world")", and the autho…

Thank you (author here).

I have not looked into list comprehensions. I think that and how generators work internally might be interesting to write about. I will try to cover them.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#65

Post by abhi9u »

kerkeslager wrote: post #6

I like the deep dive into the source, but big picture, I think if you understand the general order of evaluation which applies to most languages, this isn't all that unintuitive. The arguments going into the outer [ ] get evaluated before the outer [ ], meaning the inner [] gets evaluated first. Once that happens the inner list is created, and there's no code left that even visually could create more inner lists. The…

Yes, the default function argument values is another surprising behavior.

In this case, the default arguments get compiled as part of the function code and are part of the function's environment. As a result every time that function is called it is always having the same environment, i.e. the default argument values are not reinterpreted on every invocation. So the mutations to the environment are visible across function calls.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#66

Post by thomasahle »

bmitc wrote: post #28
thomasahle wrote: post #27

Earlier quoted context omitted.

> I expect them to come back with four distinct empty buckets. The intuitive interpretation of [[]]*4 is asking just that. I think the small example is misguiding the intuition here. If I write `my_list = f(expensive operation); long_list = my_list * 4`, I don't expect python to recreate the whole thing four times. I justly assume `f` will be evaluated once, and then copied.

That's a different statement and expected, but in a language like Python, it isn't always clear whether my_list is a value or a reference. And because of that, it leads to unexpected behavior when an intuitively expected value is actually a reference.

> in a language like Python, it isn't always clear whether my_list is a value or a reference.

I can't think of anything that's a value? At least since Java, the idea of "everything is an object" seems pretty dominant in most languages.

There are a few immutable types, like ints, strings and tuples, but you still refer to them by reference.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#67

Post by thomasahle »

EdwardDiego wrote: post #4

Using * on a sequence wouldn't get past code review with me, it's an obscure feature with surprising side-effects and it doesn't explain your thinking. Only context I've ever found it useful in, is if you're unit testing something that accepts a string of max length M, then bad_str = "A" * (M + 1) Is quite nice and also, easy to interpret. But if you're calling multiply on a list[list[Any]] lol nah mate, what the hel…

There's always the `zip(*[iter(x)]*2)` to split an iterable into even/odds.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#68

Post by kerkeslager »

bmitc wrote: post #14
kerkeslager wrote: post #6

I like the deep dive into the source, but big picture, I think if you understand the general order of evaluation which applies to most languages, this isn't all that unintuitive. The arguments going into the outer [ ] get evaluated before the outer [ ], meaning the inner [] gets evaluated first. Once that happens the inner list is created, and there's no code left that even visually could create more inner lists. The…

You can whitewash this away by discussing order of evaluation, but it still is a great example of how Python continually mixes up concepts of expressions and statements and immutable and mutable data. In real life, if I ask someone to go get me four empty buckets, I expect them to come back with four distinct empty buckets. The intuitive interpretation of [[]]*4 is asking just that. Instead Python creates a bucket an…

> In real life, if I ask someone to go get me four empty buckets, I expect them to come back with four distinct empty buckets.

That's not what you're asking for, though. You're asking for one bucket, and then you're putting it in another bucket, and then you're duplicating the contents of the second bucket four times.

This isn't whitewashing, this is just what those expressions mean.

It's not like it's hard to get four buckets in Python, BTW:

    [[] for _ in range(4)]

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#69

Post by kerkeslager »

bmitc wrote: post #41
magnio wrote: post #37

Earlier quoted context omitted.

> In more sane languages, the expression [] means give me an empty list value, nothing more and nothing less. So then whatever the syntax for [[]]*4 is in such sane languages, it gives you four empty list values. I don't buy this. Either the syntax to give you four empty list values is some kind of comprehension, which Python has and does exactly what you like, or it must have lazy evaluation somehow. Otherwise, how…

My question is: what's a real-world example where this [...]*n syntax buys you something that can't be done simply otherwise and that is worth the complexity tradeoff of people often using it subtly wrong?

This feels like a legitimate critique to me. I'm developing a programming language myself and I'm not going to include a syntax like this--multiplying an array by a number feels like a type error to me.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#70

Post by kerkeslager »

abhi9u wrote: post #65
kerkeslager wrote: post #6

I like the deep dive into the source, but big picture, I think if you understand the general order of evaluation which applies to most languages, this isn't all that unintuitive. The arguments going into the outer [ ] get evaluated before the outer [ ], meaning the inner [] gets evaluated first. Once that happens the inner list is created, and there's no code left that even visually could create more inner lists. The…

Yes, the default function argument values is another surprising behavior. In this case, the default arguments get compiled as part of the function code and are part of the function's environment. As a result every time that function is called it is always having the same environment, i.e. the default argument values are not reinterpreted on every invocation. So the mutations to the environment are visible across func…

> In this case, the default arguments get compiled as part of the function code and are part of the function's environment.

Yeah, I understand that; I'm saying that was a choice they made, and I think having the defaults be evaluated at each call time would have been a better choice.

Post reply on HN