Live data from Hacker News

Python dicts are now ordered

softwaremaniacs.org

131–140 of 457 posts

Re: Python dicts are now ordered

#131

I've written a lot of Python, but more Java. This is where I have a gripe with "batteries included." In Java, I'd have to think slightly about this, then use a LinkedHashMap. It's been in Java since *2002. It also has a Set flavor. Python just doesn't have as rich of a collection of included data structures, and the APIs are more limited.

OrderedDict has existed in python since 2.7, released in 2009.

Re: Python dicts are now ordered

#132

Great decision IMO. I remember maintaining a bunch of Python code that we supported on both OSX and Windows. Out of all the platform-specific bugs we had (where it worked on one OS and not the other), one of the most common causes was code that relied on a certain key order. And we knew that relying on key order was bad, we're supposed to use things like OrderedDict, blah blah blah. It was still a really easy mistake…

One particular annoyance of OrderedDict objects was initialising them with literals. You couldn't just do: od = OrderedDict({ "y": "first", "x": "second", }) because, by the time the data got to OrderedDict's constructor, it had already been through a dict. Instead you had to supply a list of lists (or tuple of tuples etc.): od = OrderedDict([ ("y", "first"), ("x", "second"), ]) For hierarchically nested dictionaries…

But you can also just create them with named arguments, like this:

    collections.OrderedDict(y = "first", x = "second")

Re: Python dicts are now ordered

#133

I'd love to know more about where an ordered dict comes in handy. Anyone have use cases? Otherwise, guaranteeing this behavior just seems ‾\_(ツ)_/‾ If this is useful, I wonder if an ordered set is useful.

Imagine a trie implemented as a tree of dictionaries (ignore thinking about whether a tree of arrays may actually be better for now). Let’s say you want to implement an autocomplete algorithm based on a dictionary of words from a corpus. The autocomplete algorithm is naive: if a user types in “abc” you recommend whichever word starting in “abc” has the most occurrences in the corpus.

With ordered dicts you can use the trie to compute your autocomplete very quickly with no additional data structures. There are lots of other trie operations which are more efficient with ordered dicts too. I’m sure someone can come up with less complicated examples but this was the first one I thought of

Re: Python dicts are now ordered

#134
post #84

Great decision IMO. I remember maintaining a bunch of Python code that we supported on both OSX and Windows. Out of all the platform-specific bugs we had (where it worked on one OS and not the other), one of the most common causes was code that relied on a certain key order. And we knew that relying on key order was bad, we're supposed to use things like OrderedDict, blah blah blah. It was still a really easy mistake…

The Go designers went the other way (as they often do): > When iterating over a map with a range loop, the iteration order is not specified and is not guaranteed to be the same from one iteration to the next. Actually it is not only "not guaranteed to be the same", the runtime actively makes sure that the iteration order is actually different so you don't even start to rely on it...

Python kinda sorta did the same from Python 3.3 to 3.5: iteration order was not actively randomised but the hash was "seeded" (for DDOS resistance) so iteration order would only be coherent within a run, which in effect made it unreliable.

Python 3.6 changed the underlying implementation to one which incidentally made iteration order not rely on the hash function, and as that seemed like a useful property and one which risked causing compatibility issues with alternate implementation (developers would come to rely on cpython's iteration order and code would break on jython or whatever) they decided to make it part of the spec for 3.7.

Re: Python dicts are now ordered

#135
post #84

Earlier quoted context omitted.

The Go designers went the other way (as they often do): > When iterating over a map with a range loop, the iteration order is not specified and is not guaranteed to be the same from one iteration to the next. Actually it is not only "not guaranteed to be the same", the runtime actively makes sure that the iteration order is actually different so you don't even start to rely on it...

That seems like it would be a performance hit to actively make it different? That seems like a very weird and strange design decision if true

It's a very small performance hit because it doesn't do the work to be uniformly random, just "not always the same". If you think about how you have to iterate through a hash table or similar data structure anyhow, it's either O(1) or O(log n) paid once per "range" on the data structure, which is dwarfed by the actual act of ranging on the data structure.

Go's philosophy is also definitely willing to pay that price to avoid a large class of known bugs that has hit all kinds of code bases. It is not about being the fastest language. As compiled languages go, it's solidly middle-tier, and not likely to go up much from there. (Among the C-style compiled languages, it's low-tier on performance, at around half the speed of C in general. However there's enough compiled languages like Haskell that are still generally relatively slow so that Go is mid-tier for compiled languages over all.)

Re: Python dicts are now ordered

#136

Earlier quoted context omitted.

One particular annoyance of OrderedDict objects was initialising them with literals. You couldn't just do: od = OrderedDict({ "y": "first", "x": "second", }) because, by the time the data got to OrderedDict's constructor, it had already been through a dict. Instead you had to supply a list of lists (or tuple of tuples etc.): od = OrderedDict([ ("y", "first"), ("x", "second"), ]) For hierarchically nested dictionaries…

But you can also just create them with named arguments, like this: collections.OrderedDict(y = "first", x = "second")

kwarg order wasn't guaranteed either until Python 3.6.

Re: Python dicts are now ordered

#137

I'm glad to see other languages finally catching up to PHP. I'm joking (kind of) but after a lot of years of doing this, I've begun de-prioritizing pure abstractions and favoring the way that humans tend to do things on their own. Technically this is along the lines of the worse-is-better philosophy. The single biggest cost in software development is friction. Performance, size, etc are all less important, because th…

>Performance, size, etc are all less important, because they become less important with each passing year as computers grow more powerful. That isn't as true as it once was. We are running up against fundamental limits and people are even starting to talk about the environmental impact of computing.

It should be noted that the ordering of python dict occurred as a side-effect of a new implementation with better memory profile (and as good or better performances).

Re: Python dicts are now ordered

#138
post #15

Am I the only one that thinks this is a stupid decision? This will silently break code that starts to rely on this behaviour that gets executed on Python3.5 and lower. I would consider changing how a builtin works to be a major breaking change. It would have been fine if this was a change between 2 and 3 but on a minor version? Thats insane.

I think you are giving it bigger weight than it deserves. Writing Python code I would not want to depend on internal order of dict elements without a very good reason. If I am writing some low level code that could benefit from it, maybe I would use this feature, but I would then encapsulate it and insert a version check to complain.

This is like many evaluation guarantees in latest C++-NN standards. Good to know, might be useful for performance tuning of automatic code generators, etc. , but irrelevant most of the time. My 2c.

Re: Python dicts are now ordered

#139

Earlier quoted context omitted.

That is an original line of thinking in the world of software, “Let’s not add a feature to a new version because it wouldn’t work if used in an older version”.

It would silently introduce bugs, that's the whole issue.

Right. In fact, if it explicitly broke something there'd be less of an issue. Instead, it will appear to work/interpret but you'll get unreliable results. That makes the bug more nebulous.

Re: Python dicts are now ordered

#140
post #109
post #70

Earlier quoted context omitted.

Really? I would have figured they'd just do something like: class OrderedDict(dict): pass (Plus a little bit of API shimming.)

No, OrderedDict has it's own C implementation which was created just before it was decided that dict would preserve order across iteration. Further there is a big difference, regular dict preserves order across iteration but OrderedDict treats order up to equality. I.e. this returns True: {1: 1, 2: 2} == {2: 2, 1: 1} Where as this returns False: OrderedDict({1: 1, 2: 2}) == OrderedDict({2: 2, 1: 1}) To make that diff…

Also ordereddicts provide methods to move items to the start or end, and remove items specifically at the start or end, not so for regular dicts.
Post reply on HN