Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code • Hacker News Board

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#31

Post by dragonwriter » Tue, Apr 09, 2024, 7:41 AM UTC

bmitc wrote: post #23

Joker_vD wrote: post #17

Earlier quoted context omitted.

I really wonder what "more sane" languages you mean. Because a single [] does give you an empty list value in Python: one list, and not four different empty lists. Why would it give your four? That'd need to somehow bring in the surrounding context and in "sane" languages, (closed) expressions generally don't change the meaning because the context around them is changing. Actually, if you really want to have re-evalu…

F#: List.init 4 (fun _ -> []) [for _ in 1..4 -> []] Or any sane language that uses immutable values by default like Elixir, Erlang, Clojure, Racket, Scheme, OCaml, etc. > Actually, if you really want to have re-evaluting (sub)expressions, then Python does give you an option: comprehensions. That's the complaint, that [[]]*4 is not shorthand for list comprehensions and that it's something entirely different. This is c…

> That's the complaint, that [[]]4 is not shorthand for list comprehensions

There are lots of reasons you would want items in a list repeated rather than having, essentially, the result of separate executions of the same constructor -- [] is just shorthand for list() -- so its good that [x()]4 is not shorthand for [x() for _ in range(4)]. Especially since that would mean it would have to be special-case sytax for list construction, and not a list operator that you could use on existing lists. The mildly annyoing edge case you need to learn about isn’t worth wiping out the whole broadly-useful language feature, or creating a great special-case inconsistency which would be more cognitive load to deal with, to eliminate.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#32

Post by dragonwriter » Tue, Apr 09, 2024, 7:43 AM UTC

arijun wrote: post #25

bmitc wrote: post #14

Earlier quoted context omitted.

You can whitewash this away by discussing order of evaluation, but it still is a great example of how Python continually mixes up concepts of expressions and statements and immutable and mutable data. In real life, if I ask someone to go get me four empty buckets, I expect them to come back with four distinct empty buckets. The intuitive interpretation of [[]]*4 is asking just that. Instead Python creates a bucket an…

Languages with immutable data structures are relatively rare. With its high level focus python could have been one of them, but went for different trade offs. > In more sane languages, the expression [] means give me an empty list value, nothing more and nothing less What is an “empty list value”?

Python has immutable data structures, lists just aren't one of them. (Tuples are. So are, unlike in some languages, strings.)

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#33

Post by arijun » Tue, Apr 09, 2024, 7:43 AM UTC

bmitc wrote: post #23

Joker_vD wrote: post #17

Earlier quoted context omitted.

I really wonder what "more sane" languages you mean. Because a single [] does give you an empty list value in Python: one list, and not four different empty lists. Why would it give your four? That'd need to somehow bring in the surrounding context and in "sane" languages, (closed) expressions generally don't change the meaning because the context around them is changing. Actually, if you really want to have re-evalu…

F#: List.init 4 (fun _ -> []) [for _ in 1..4 -> []] Or any sane language that uses immutable values by default like Elixir, Erlang, Clojure, Racket, Scheme, OCaml, etc. > Actually, if you really want to have re-evaluting (sub)expressions, then Python does give you an option: comprehensions. That's the complaint, that [[]]*4 is not shorthand for list comprehensions and that it's something entirely different. This is c…

Why are you comparing functions/closures with values? You can do the same in python:

  [list() for _ in range(4)]

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#34

Post by karelpeeters » Tue, Apr 09, 2024, 7:58 AM UTC

arijun wrote: post #29

bmitc wrote: post #14

Earlier quoted context omitted.

You can whitewash this away by discussing order of evaluation, but it still is a great example of how Python continually mixes up concepts of expressions and statements and immutable and mutable data. In real life, if I ask someone to go get me four empty buckets, I expect them to come back with four distinct empty buckets. The intuitive interpretation of [[]]*4 is asking just that. Instead Python creates a bucket an…

Languages with immutable data structures are relatively rare. With its high level focus python could have been one of them, but went for different trade offs. > In more sane languages, the expression [] means give me an empty list value, nothing more and nothing less What is a “list value”? It’s either a pointer to some other space where the lists values are stored, or it’s the values stored themselves. If you say it…

Matlab is one language that chooses the second. Every assignment a=b where b is a matrix creates a distinct copy of the matrix.

The interpreter is slightly more clever and instead implements this as a copy-on-writr mechanism, but that's just an implementation detail .

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#35

Post by shiandow » Tue, Apr 09, 2024, 7:58 AM UTC

bmitc wrote: post #14

kerkeslager wrote: post #6

I like the deep dive into the source, but big picture, I think if you understand the general order of evaluation which applies to most languages, this isn't all that unintuitive. The arguments going into the outer [ ] get evaluated before the outer [ ], meaning the inner [] gets evaluated first. Once that happens the inner list is created, and there's no code left that even visually could create more inner lists. The…

You can whitewash this away by discussing order of evaluation, but it still is a great example of how Python continually mixes up concepts of expressions and statements and immutable and mutable data. In real life, if I ask someone to go get me four empty buckets, I expect them to come back with four distinct empty buckets. The intuitive interpretation of [[]]*4 is asking just that. Instead Python creates a bucket an…

In that case what would you expect the following to return?

    a = []
    b = [a]*4
    a.append(1)
    print(b)

I mean it is possible to change Python and make [...]*n a shorthand for "repeat this statement n times" but that would be odd on its own.

Lists pass by reference, tuples pass by value, if you want to understand Python you're just going to have to get used to that distinction. There are other options, but most are worse.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#36

Post by bmitc » Tue, Apr 09, 2024, 7:58 AM UTC

arijun wrote: post #33

bmitc wrote: post #23

Earlier quoted context omitted.

F#: List.init 4 (fun _ -> []) [for _ in 1..4 -> []] Or any sane language that uses immutable values by default like Elixir, Erlang, Clojure, Racket, Scheme, OCaml, etc. > Actually, if you really want to have re-evaluting (sub)expressions, then Python does give you an option: comprehensions. That's the complaint, that [[]]*4 is not shorthand for list comprehensions and that it's something entirely different. This is c…

Why are you comparing functions/closures with values? You can do the same in python: [list() for _ in range(4)]

> Why are you comparing functions/closures with values?

I'm not. The fun _ -> [] is needed because the argument is the element's index. That's just what List.init happens to expect, a function that takes an index and initializes the element, but the index is not needed here.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#37

Post by magnio » Tue, Apr 09, 2024, 8:00 AM UTC

bmitc wrote: post #14

kerkeslager wrote: post #6

I like the deep dive into the source, but big picture, I think if you understand the general order of evaluation which applies to most languages, this isn't all that unintuitive. The arguments going into the outer [ ] get evaluated before the outer [ ], meaning the inner [] gets evaluated first. Once that happens the inner list is created, and there's no code left that even visually could create more inner lists. The…

You can whitewash this away by discussing order of evaluation, but it still is a great example of how Python continually mixes up concepts of expressions and statements and immutable and mutable data. In real life, if I ask someone to go get me four empty buckets, I expect them to come back with four distinct empty buckets. The intuitive interpretation of [[]]*4 is asking just that. Instead Python creates a bucket an…

> In more sane languages, the expression [] means give me an empty list value, nothing more and nothing less. So then whatever the syntax for [[]]*4 is in such sane languages, it gives you four empty list values.

I don't buy this. Either the syntax to give you four empty list values is some kind of comprehension, which Python has and does exactly what you like, or it must have lazy evaluation somehow. Otherwise, how can the language know you don't want to copy the existing reference but create a new empty list? If [[]]*4 were to create new list on the fly, then `[print('hello')]*4` would have to print hello 4 times, i.e., the print statement is lazily executed, which does not fit into Python at all.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#38

Post by Someone » Tue, Apr 09, 2024, 8:01 AM UTC

arijun wrote: post #29

bmitc wrote: post #14

Earlier quoted context omitted.

You can whitewash this away by discussing order of evaluation, but it still is a great example of how Python continually mixes up concepts of expressions and statements and immutable and mutable data. In real life, if I ask someone to go get me four empty buckets, I expect them to come back with four distinct empty buckets. The intuitive interpretation of [[]]*4 is asking just that. Instead Python creates a bucket an…

Languages with immutable data structures are relatively rare. With its high level focus python could have been one of them, but went for different trade offs. > In more sane languages, the expression [] means give me an empty list value, nothing more and nothing less What is a “list value”? It’s either a pointer to some other space where the lists values are stored, or it’s the values stored themselves. If you say it…

> If you say it’s the second (I can’t think of any language that does), then `a = […]; b = a` is an expensive copy by value assignment

In Swift collections are value types, so b = a conceptually is a copy. It’s not expensive, though, because Swift does copy-on-write.

You pay for that by a combination of having more expensive collection updates (they, at least conceptually, have to check whether to make that copy) and a more complex compiler (it tries to avoid having to actually do those checks at runtime)

  var a: [Int]=[]
  var b = [a,a,a,a]
  b[0].append(1)
  print(b)

will produce

  [[1],[],[],[]]

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#39

Post by chii » Tue, Apr 09, 2024, 8:03 AM UTC

Joker_vD wrote: post #8

kerkeslager wrote: post #6

I like the deep dive into the source, but big picture, I think if you understand the general order of evaluation which applies to most languages, this isn't all that unintuitive. The arguments going into the outer [ ] get evaluated before the outer [ ], meaning the inner [] gets evaluated first. Once that happens the inner list is created, and there's no code left that even visually could create more inner lists. The…

Well, those things are called "default argument values ", not "default argument expressions ", you know. It's a value that's reused for every function call, not an expression that re-evaluated at every function call.

if you compare

> a = [0]

and

> b = [0]

these two expressions produces different 'value' objects for `[0]`, and modifying `a.append(a[-1] + 1)` does not change b.

so it's unintuitive that using `[0]` as a default value in a function argument can cause that argument to retain the same value across multiple function calls, when programmers would assume the argument to a function is not the same across multiple invocations.

The issue really here is the usage of an array object, which is not a value object (immutable), but a reference object. It would not have been a problem if the default argument was a string, or a number.

This is a reason not to use arrays in default arguments.

Re: Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

#40

Post by bmitc » Tue, Apr 09, 2024, 8:04 AM UTC

shiandow wrote: post #35

bmitc wrote: post #14

Earlier quoted context omitted.

You can whitewash this away by discussing order of evaluation, but it still is a great example of how Python continually mixes up concepts of expressions and statements and immutable and mutable data. In real life, if I ask someone to go get me four empty buckets, I expect them to come back with four distinct empty buckets. The intuitive interpretation of [[]]*4 is asking just that. Instead Python creates a bucket an…

In that case what would you expect the following to return? a = [] b = [a]*4 a.append(1) print(b) I mean it is possible to change Python and make [...]*n a shorthand for "repeat this statement n times" but that would be odd on its own. Lists pass by reference, tuples pass by value, if you want to understand Python you're just going to have to get used to that distinction. There are other options, but most are worse.

> , if you want to understand Python you're just going to have to get used to that distinction

My argument is that that distinction isn't needed, by default, in a high-level language, and that plenty of people get it wrong, understandably. It is simply complexity that is not needed.

These simple examples can be toyed with, but sooner or later one is going to want to solve real problems, and unintuitive, default behavior doesn't help with that. Python's mode of operation tends to favor implicit over explicit. And the syntax [...]*n is implicit.