Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python dicts are now ordered

softwaremaniacs.org

Post reply on HN
Print view View original

321–330 of 457 posts

Re: Python dicts are now ordered

#321

Post by mypalmike »

baddox wrote: post #240
baq wrote: post #208

Earlier quoted context omitted.

I’d call undependable but maybe it’s the same thing in practice?

That's generally what people mean when they say "nondeterministic" in the context of computing. Yeah, in philosophy it generally means something like "the future is not completely determined by the past," but in computing it means something closer to "the programmer cannot reasonably determine the behavior and thus should not depend on specific behavior."

In computing it means a given set of inputs lead to a given set of outputs. It has nothing to do with how difficult it is for a programmer to reason about. Deterministic builds, deterministic tests, etc.

Re: Python dicts are now ordered

#322

Post by lukeschlather »

masklinn wrote: post #265
lukeschlather wrote: post #187

Earlier quoted context omitted.

Especially with the (lack of) change to sets I'm interested to benchmark some regular things before and after the change. set(dict.keys()) set(dict.values()) thing1 = dict(...) thing2 = copy.deepcopy(thing1) I feel like there could be some other testcases. I'm wholly in support of the change (regardless of benchmarks) but depending on the results I could see some arguments against.

1. sets don't use the new dict's implementation 2. IIRC the new dicts are no(t significantly) slower than the old dicts, however they use less memory, and iterate faster The iteration order is actually a side-effect of implementation details, the original goals were a more compact representation and a faster iteration: https://mail.python.org/pipermail/python-dev/2012-December/1...

> 1. sets don't use the new dict's implementation

Yes, of course. That's why I'm wondering if turning dict keys into a set is slower now.

Re: Python dicts are now ordered

#323

Post by duckerude »

taeric wrote: post #294
duckerude wrote: post #115

Earlier quoted context omitted.

Ordered dictionaries already existed as a separate type. Much of the gain was in the use of dicts for core language features. In particular, Python 3.6 guarantees that the order in which class attributes are defined is preserved, and that functions which take variadic keyword arguments receive them in the order in which they were passed. It permits other implementations to use types other than dict to accomplish that…

And this is all the more reason using a type would make sense. Curious why things use ordered? Look at all the places that declare they need ordered. :)

You don't know in advance when ordering might be useful, though.

Class __annotations__ was new in 3.6. For its original use there's no need for ordering. They were ordered, because dicts were ordered, but only as an implementation detail.

Dataclasses were added to 3.7, after 3.6's features made a nice syntax possible. If __annotations__ hadn't happened to be ordered already then I would guess it wouldn't have been made ordered just for dataclasses - dataclasses just wouldn't have existed.

Making everything ordered in one go opens up possibilities you haven't even thought of yet.

Re: Python dicts are now ordered

#324

Post by divbzero »

The historical background for those interested…

The ordered dict for Python was originally an implementation detail, part of a more compact dict representation that was first proposed for Python in 2012 [1], implemented for PyPy in 2015 [2], and merged into CPython 3.6 in 2016 [3] [4].

[1]: https://mail.python.org/pipermail/python-dev/2012-December/1...

[2]: https://morepypy.blogspot.com/2015/01/faster-more-memory-eff...

[3]: https://mail.python.org/pipermail/python-dev/2016-September/...

[4]: https://news.ycombinator.com/item?id=12460936

The decision to add it officially to the language spec wasn’t made until Python 3.7 in 2017. [5]

[5]: https://mail.python.org/pipermail/python-dev/2017-December/1... "Guido says ‘Make it so.’"

Re: Python dicts are now ordered

#325

Post by fyp »

reidrac wrote: post #257
rossmohax wrote: post #247

Earlier quoted context omitted.

Python dicts are not ordered maps, they are dicts preserving insertion order.

The title of the post is misleading: it means "ordered by insertion order" (not sorted). Previously it was unpredictable, and that's why collections have OrderedDict; which makes sense instead of trusting an implementation detail of CPython from 3.6. Is not mentioned in the docs: https://docs.python.org/3/library/stdtypes.html#mapping-type... EDIT: but the behaviour is documented in OrderedDict itself! https://docs.p…

I wonder why OrderedDict isn't outright deprecated (or reimplemented as a trivial wrapper over dict).

You can get the first/last key of a 3.8 dict with dict.keys() and reversed(dict.keys()) which you can then delete to reimplement OrderedDict.popitem.

Deleting a key and reinserting it will let you reimplement move_to_end.

The only cool thing that OrderedDict's implementation might be useful for is to move to front (or anywhere else not the back) but it doesn't expose that api.

Re: Python dicts are now ordered

#326

Post by jwkane »

philwelch wrote: post #296
jwkane wrote: post #255

Earlier quoted context omitted.

when you are diffing the serialized output?

From what I recall of the JSON standard itself, there's no guarantee about key ordering being significant. If you're diffing serialized output to compare two JSON objects you need to be serializing it in a consistent format, otherwise even whitespace is going to throw you off.

It's significant to a human that wants to know what has changed.

Re: Python dicts are now ordered

#327

Post by raziel2p »

throw18374 wrote: post #190
throwawaylolx wrote: post #54

Where "now" dates back to 2018? https://docs.python.org/3/whatsnew/3.7.html

Came to say the same.. insertion order of dict has been here for years for 90+% of Python users (CPython implementation) and a part of the language spec also for almost as many years. I guess it’s good to spread awareness to HN readers who apparently were unaware, but the headline is very misleading.

90% seems optimistic. I don't think that many Python projects are running the most recent version in production. From personal experience, upgrading to 3.7 was especially a pain because of packages that used "async" for variable/kwarg names.

Re: Python dicts are now ordered

#328

Post by taeric »

duckerude wrote: post #323
taeric wrote: post #294

Earlier quoted context omitted.

And this is all the more reason using a type would make sense. Curious why things use ordered? Look at all the places that declare they need ordered. :)

You don't know in advance when ordering might be useful, though. Class __annotations__ was new in 3.6. For its original use there's no need for ordering. They were ordered, because dicts were ordered, but only as an implementation detail. Dataclasses were added to 3.7, after 3.6's features made a nice syntax possible. If __annotations__ hadn't happened to be ordered already then I would guess it wouldn't have been ma…

That argument cuts both ways. You don't know when the benefits of random will be there...

As soon as you do something that cares about order, state how it is derived. Sometimes, insertion order is right. Sometimes, not.

Don't get me wrong alists are nice. And order def matters to those. But it is part of their definition. And reinsertion changes the order in obvious ways. Not even clear what it does to just "dict".

Now, I will concede this is overblown. Life will easily go on.

Re: Python dicts are now ordered

#329

Post by cdelsolar »

skrebbel wrote: post #242
svnpenn wrote: post #226

Earlier quoted context omitted.

> PHP really got this one right (and immediately messed it up by mixing it with arrays, but hey, PHP). No, it didnt. What PHP calls "array" is actually an Ordered Map, as you alluded to: https://yaml.org/type/omap it literally says that in the documentation, paragraph one, sentence one: > An array in PHP is actually an ordered map. https://php.net/types.array The issue, if one exists, is that PHP doesnt have a sequen…

Sure, PHP arrays are actually ordered maps, but it's super confusing when used as arrays. I mean, look at code like this: $arr = []; $arr[2] = "two"; $arr[0] = "zero"; $arr[1] = "one"; echo join($arr, ", "); // two, zero, one In any other language, the result of similar code would be "zero, one, two". You'd need to `ksort` this thing to get it to behave like a normal bog-standard boring array. Everybody's writing PHP…

I literally just learned "bog-standard" 10 minutes ago and now I see it here. And no, don't Baader-Meinhof me, I would have noticed this weird term in random text in the past.

Re: Python dicts are now ordered

#330

Post by fock »

hprotagonist wrote: post #305
fock wrote: post #256

Earlier quoted context omitted.

that's because tensorflow is advertisement for Google and while it's technically open-source, it doesn't stand for any kind of community-project, it's all there to show off (and ingrain in its users) the way Google wants things to Go (just look at the byzantine Bazel-build-processes - tensorflow taking hours to build and pytorch about 10minutes...).

my torch builds also take hours. facebook is just as capable of writing hot garbage, sadly.

Yeah, still it's an order of magnitudes faster...
Post reply on HN