Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

codeconfessions.substack.com

Post reply on HN
Print view View original

51–60 of 101 posts

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#51

Post by Skeime »

bmitc wrote: post #40
shiandow wrote: post #35

Earlier quoted context omitted.

In that case what would you expect the following to return? a = [] b = [a]*4 a.append(1) print(b) I mean it is possible to change Python and make [...]*n a shorthand for "repeat this statement n times" but that would be odd on its own. Lists pass by reference, tuples pass by value, if you want to understand Python you're just going to have to get used to that distinction. There are other options, but most are worse.

> , if you want to understand Python you're just going to have to get used to that distinction My argument is that that distinction isn't needed, by default, in a high-level language, and that plenty of people get it wrong, understandably. It is simply complexity that is not needed. These simple examples can be toyed with, but sooner or later one is going to want to solve real problems, and unintuitive, default behav…

The distinction is needed, I think, unless you want to go the purely-functional route. I love that route, but just as many people will complain about purely functional languages being unintuitive.

If you want mutable values in your language, you need to expect values to mutate. So what should list multiplication do?

- There is the current implementation, where a change to the first entry of [f(x)] * 4 will be reflected in all entries of the list (because they're all references to the same value). - It could only work on list literals. So [f(x)] * 4 would call f four times and collect the values in a list. Then, a = [f(x)]; a * 4 would not work, which would be surprising. - It could work differently on list literals and lists. So [f(x)] * 4 would call f four times collecting the results, but a = [f(x)]; a * 4 would just return a list of four references to the one result of f(x). - It could copy the value in the list. How would you do that if the list contained values like file handlers or network connections that can't be copied?

Within the context of Python (an imperative language with mutable values), the current implementation is the most sensible one. If you work in such a language, you must always exercise some care when working with mutable values. Learn your tool!

Python could have been designed with different decisions. But who knows if we would even be talking about it today in that case?

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#52

Post by Skeime »

blt wrote: post #9
10000truths wrote: post #5

TL;DR is that multiplying a list performs a shallow copy of the list's elements. Great article. The only thing it's missing is a snippet that shows what you should do instead: [[] for _ in range(N)] Since generator-like expressions evaluates the inner expression (i.e. the []) on each iteration, the elements of the outer list are guaranteed to be unique references.

This is pretty subtle. I wish more languages explicitly distinguished references and values like C++ and Rust.

In Python, everything is a reference, so you don't need special syntax to mark them.

There just happen to be some types where you cannot change the referenced value in any way. These behave like value types in other languages.

(If you believe that they should be marked explicitly, do you also believe the same thing about Rust? I.e. should the Rust compiler throw an error if you never actually change a reference type, as you "should" be using a value type in that case?)

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#53

Post by maxcoder4 »

bmitc wrote: post #41
magnio wrote: post #37

Earlier quoted context omitted.

> In more sane languages, the expression [] means give me an empty list value, nothing more and nothing less. So then whatever the syntax for [[]]*4 is in such sane languages, it gives you four empty list values. I don't buy this. Either the syntax to give you four empty list values is some kind of comprehension, which Python has and does exactly what you like, or it must have lazy evaluation somehow. Otherwise, how…

My question is: what's a real-world example where this [...]*n syntax buys you something that can't be done simply otherwise and that is worth the complexity tradeoff of people often using it subtly wrong?

It's syntax sugar. I use it all the time, but of course it could always be respected by, i don't know, a for loop. It also makes writing duck typed code easier, as X3 will work for strings, bytes and lists.

Overall it's a very useful feature and I don't think it's that surprising. It's pretty obvious [x]4 will reuse the same reference. Just like manually adding "x" four times will.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#54

Post by mapreduce »

bmitc wrote: post #41
magnio wrote: post #37

Earlier quoted context omitted.

> In more sane languages, the expression [] means give me an empty list value, nothing more and nothing less. So then whatever the syntax for [[]]*4 is in such sane languages, it gives you four empty list values. I don't buy this. Either the syntax to give you four empty list values is some kind of comprehension, which Python has and does exactly what you like, or it must have lazy evaluation somehow. Otherwise, how…

My question is: what's a real-world example where this [...]*n syntax buys you something that can't be done simply otherwise and that is worth the complexity tradeoff of people often using it subtly wrong?

> that is worth the complexity tradeoff of people often using it subtly wrong?

I don't think that makes the complexity tradeoff worth it. This is the kind of tradeoff that makes the language implementation, reference and semantics more and more complex. Unfortunately Python has a lot of such complexities already. We don't need more of it. Such complexity hurts the creation of completing implementations.

IMHO programming language specifications and implementations should be simple and consistent so that competing implementations can be developed easily.

> people often using it subtly wrong?

I think that... you know... before writing serious software, these people need to roll up their sleeves and just learn the damn language. Just learn what references are and how they work in Python. It's not that hard. It is most definitely easier than adding a sphagetti monster to the compiler make it bend to the will of programmers who can't be bothered to read the tutorial. I mean this gotcha about references is taught in the 4th chapter of the tutorial. Can't developers even bother to RTFM these days before developing serious software with a programming language?

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#55

Post by maxcoder4 »

bmitc wrote: post #23
Joker_vD wrote: post #17

Earlier quoted context omitted.

I really wonder what "more sane" languages you mean. Because a single [] does give you an empty list value in Python: one list, and not four different empty lists. Why would it give your four? That'd need to somehow bring in the surrounding context and in "sane" languages, (closed) expressions generally don't change the meaning because the context around them is changing. Actually, if you really want to have re-evalu…

F#: List.init 4 (fun _ -> []) [for _ in 1..4 -> []] Or any sane language that uses immutable values by default like Elixir, Erlang, Clojure, Racket, Scheme, OCaml, etc. > Actually, if you really want to have re-evaluting (sub)expressions, then Python does give you an option: comprehensions. That's the complaint, that [[]]*4 is not shorthand for list comprehensions and that it's something entirely different. This is c…

>That's the complaint, that [[]]*4 is not shorthand for list comprehensions and that it's something entirely different.

Well, of course. It's a shorthand for duplicating the list elements four times. In this case, the list contains a single reference to an empty list, do the result contains four references to an empty list.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#56

Post by maxcoder4 »

bmitc wrote: post #28
thomasahle wrote: post #27

Earlier quoted context omitted.

> I expect them to come back with four distinct empty buckets. The intuitive interpretation of [[]]*4 is asking just that. I think the small example is misguiding the intuition here. If I write `my_list = f(expensive operation); long_list = my_list * 4`, I don't expect python to recreate the whole thing four times. I justly assume `f` will be evaluated once, and then copied.

That's a different statement and expected, but in a language like Python, it isn't always clear whether my_list is a value or a reference. And because of that, it leads to unexpected behavior when an intuitively expected value is actually a reference.

I think you're confused about the difference between [x]*4 and [x for _ in range (4)]. It's not, at any point, a difference between something being a value or reference.

The first code is equivalent to:

list_multiply([x], 4)

(the function is actually called list.__mul__, but that's an unimportant detail). The latter is a syntax sugar for:

out = [] for _ in range(4): out.append(x)

(actually a bit more complex, but close enough).

When you understand it, the behavior is obvious in both cases.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#57

Post by Joker_vD »

maxcoder4 wrote: post #55
bmitc wrote: post #23

Earlier quoted context omitted.

F#: List.init 4 (fun _ -> []) [for _ in 1..4 -> []] Or any sane language that uses immutable values by default like Elixir, Erlang, Clojure, Racket, Scheme, OCaml, etc. > Actually, if you really want to have re-evaluting (sub)expressions, then Python does give you an option: comprehensions. That's the complaint, that [[]]*4 is not shorthand for list comprehensions and that it's something entirely different. This is c…

>That's the complaint, that [[]]*4 is not shorthand for list comprehensions and that it's something entirely different. Well, of course. It's a shorthand for duplicating the list elements four times. In this case, the list contains a single reference to an empty list, do the result contains four references to an empty list.

And it's literally impossible for [[]]*4 to be a shorthand for a list comprehension, because nothing in Python can be a shorthand for a list comprehension. Instead, it is but a method call.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#58

Post by mypalmike »

Someone wrote: post #38
arijun wrote: post #29

Earlier quoted context omitted.

Languages with immutable data structures are relatively rare. With its high level focus python could have been one of them, but went for different trade offs. > In more sane languages, the expression [] means give me an empty list value, nothing more and nothing less What is a “list value”? It’s either a pointer to some other space where the lists values are stored, or it’s the values stored themselves. If you say it…

> If you say it’s the second (I can’t think of any language that does), then `a = […]; b = a` is an expensive copy by value assignment In Swift collections are value types, so b = a conceptually is a copy. It’s not expensive, though, because Swift does copy-on-write. You pay for that by a combination of having more expensive collection updates (they, at least conceptually, have to check whether to make that copy) and…

Just teaching myself Swift this week. Thanks, this is helpful!

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#59

Post by Joker_vD »

chii wrote: post #39
Joker_vD wrote: post #8

Earlier quoted context omitted.

Well, those things are called "default argument values ", not "default argument expressions ", you know. It's a value that's reused for every function call, not an expression that re-evaluated at every function call.

if you compare > a = [0] and > b = [0] these two expressions produces different 'value' objects for `[0]`, and modifying `a.append(a[-1] + 1)` does not change b. so it's unintuitive that using `[0]` as a default value in a function argument can cause that argument to retain the same value across multiple function calls, when programmers would assume the argument to a function is not the same across multiple invocatio…

Very technically speaking, in "def f(a=[0]): ..." or in "a = [[0]] * 4" there is only one textual occurrence of "[0]", and the function definition is literally evaluated to define a function (you can try def "f(a=sys.stdout.write('Hi!')): return" in REPL to see when the default value gets constructed).

On the other hand, if we abstract from the technical minutiae, yes, it would have been entirely possible for Python to treat default arguments in the function definitions as expressions, by using machinery similar to what gets used for comprehensions: after all, as you point out, function's arguments (and local variables) do get re-assigned on every function call.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#60

Post by layer8 »

kerkeslager wrote: post #6

I like the deep dive into the source, but big picture, I think if you understand the general order of evaluation which applies to most languages, this isn't all that unintuitive. The arguments going into the outer [ ] get evaluated before the outer [ ], meaning the inner [] gets evaluated first. Once that happens the inner list is created, and there's no code left that even visually could create more inner lists. The…

The * operator could be defined to use a deepcopy operation, but isn’t, presumably for efficiency reasons. This has little to do with order of evaluation.

Alternatively, the issue is that Python does not distinguish between value and reference types.

Post reply on HN