Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A “frozen” dictionary for Python

lwn.net

Post reply on HN
Print view View original

101–110 of 172 posts

Re: A “frozen” dictionary for Python

#101

Post by perrygeo »

shadowgovt wrote: post #80
perrygeo wrote: post #75

Python discovers immutable data, and gets it wrong. Frozendict is a blunt instrument - instead of a mutable free-for-all, it just locks the data for the entire lifecycle. Brace for the wave of code littered with deep copies, proudly proclaiming how functional it all is. If you want real immutable data structures, not a cheap imitation, check out pyrsistent.

If your dicts are frozen, you shouldn't need to deep-copy. The point of immutability is that if you want a new frozendict based on another one, you just rebuild the indirection data structure up top and leave the values it references alone.

You're absolutely right: an "indirection data structure" is necessary. Freezing the data is the least interesting part - it doesn't give you any of the benefits typically associated with immutable data structures in functional languages. That's my point - Python is shipping a half solution that's being mistaken for a proper one.

You think Python developers are going to roll their own HAMT on top of frozendicts? Or are they just gonna make copies? Personally, I'd just use pyrsistent which seems to get it right.

Re: A “frozen” dictionary for Python

#102

Post by mcherm »

zahlman wrote: post #62
jonathaneunice wrote: post #58

Earlier quoted context omitted.

Ordering, like stability in sorting, is an incredibly useful property. If it costs a little, then so be it. This is optimizing for the common case, where memory is generally plentiful and dicts grow more than they shrink. Python has so many memory inefficiencies that occasional tombstones in the dict internal structure is unlikely to be a major effect. If you're really concerned, do `d = dict(d)` after aggressive del…

> Ordering, like stability in sorting, is an incredibly useful property. I can't say I've noticed any good reasons to rely on it. Didn't reach for `OrderedDict` often back in the day either. I've had more use for actual sorting than for preserving the insertion order.

Personally, I find lots of reasons to prefer an orders Dict to an unordered one. Even small effects like "the debugging output will appear in a consistent order making it easier to compare" can be motivation enough in many use cases.

Re: A “frozen” dictionary for Python

#103

Post by morshu9001 »

pansa2 wrote: post #20

I wonder whether Raymond Hettinger has an opinion on this PEP. A long time ago, he wrote: "freezing dicts is a can of worms and not especially useful". https://mail.python.org/pipermail/python-dev/2006-February/0...

I agree, same with frozenset. If you really want to use one of those as a key, convert to a tuple. There might be niche use cases for all this, but it's not something that the language or even the standard lib need to support.

Re: A “frozen” dictionary for Python

#104

Post by morshu9001 »

zahlman wrote: post #62
jonathaneunice wrote: post #58

Earlier quoted context omitted.

Ordering, like stability in sorting, is an incredibly useful property. If it costs a little, then so be it. This is optimizing for the common case, where memory is generally plentiful and dicts grow more than they shrink. Python has so many memory inefficiencies that occasional tombstones in the dict internal structure is unlikely to be a major effect. If you're really concerned, do `d = dict(d)` after aggressive del…

> Ordering, like stability in sorting, is an incredibly useful property. I can't say I've noticed any good reasons to rely on it. Didn't reach for `OrderedDict` often back in the day either. I've had more use for actual sorting than for preserving the insertion order.

Same. Recently I saw interview feedback where someone complained that the candidate used OrderedDict instead of the built-in dict that is now ordered, but they'll let it slide... As if writing code that will silently do different things depending on minor Python version is a good idea.

Re: A “frozen” dictionary for Python

#105

Post by BiteCode_dev »

zahlman wrote: post #62
jonathaneunice wrote: post #58

Earlier quoted context omitted.

Ordering, like stability in sorting, is an incredibly useful property. If it costs a little, then so be it. This is optimizing for the common case, where memory is generally plentiful and dicts grow more than they shrink. Python has so many memory inefficiencies that occasional tombstones in the dict internal structure is unlikely to be a major effect. If you're really concerned, do `d = dict(d)` after aggressive del…

> Ordering, like stability in sorting, is an incredibly useful property. I can't say I've noticed any good reasons to rely on it. Didn't reach for `OrderedDict` often back in the day either. I've had more use for actual sorting than for preserving the insertion order.

Ordering is very useful for testing.

This morning for example, I tested an object serialized through a JSON API. My test data seems to never match the next run.

After a while, I realized one of the objects was using a set of objects, which in the API was turned into a JSON array, but the order of said array would change depending of the initial Python VM state.

3 days ago, I used itertools.group by to group a bunch of things. But itertools.group by only works on iterable that are sorted by the grouping key.

Now granted, none of those recent example are related to dicts, but dict is not a special case. And it's iterated over regularly.

Re: A “frozen” dictionary for Python

#106

Post by morshu9001 »

dkarl wrote: post #64
pansa2 wrote: post #20

I wonder whether Raymond Hettinger has an opinion on this PEP. A long time ago, he wrote: "freezing dicts is a can of worms and not especially useful". https://mail.python.org/pipermail/python-dev/2006-February/0...

It's interesting that he concludes that freezing dicts is "not especially useful" after addressing only a single motivation: the use of a dictionary as a key. He doesn't address the reason that most of us in 2025 immediately think of, which is that it's easier to reason about code if you know that certain values can't change after they're created. What a change in culture over the last 20 years!

You can't really tell though. Maybe the dict is frozen but the values inside aren't. C++ tried to handle this with constness, but that has its own caveats that make some people argue against using it.

Re: A “frozen” dictionary for Python

#107

Post by BiteCode_dev »

perrygeo wrote: post #101
shadowgovt wrote: post #80

Earlier quoted context omitted.

If your dicts are frozen, you shouldn't need to deep-copy. The point of immutability is that if you want a new frozendict based on another one, you just rebuild the indirection data structure up top and leave the values it references alone.

You're absolutely right: an "indirection data structure" is necessary. Freezing the data is the least interesting part - it doesn't give you any of the benefits typically associated with immutable data structures in functional languages. That's my point - Python is shipping a half solution that's being mistaken for a proper one. You think Python developers are going to roll their own HAMT on top of frozendicts? Or ar…

pyrsistent is super slow, though. Just ran a quick benchmark:

  - Creation - 8-12x slower  
  - Lookup - 22-27x slower  
  - Contains check - 30-34x slower  
  - Iteration - 5-14x slower  
  - Merge - 32-158x slower
Except at 10k+ items, batchup dates on 100K+ items or inserting 100 keys.

This is rarely the case in practice, most dictionaries and dict operations are small, if you have a huge dict, you probably should be chunking your load or delegating that to infra.

Not to mention pyrsistent's API is incompatible with dicts, so you can't pass it to external code without conversion.

You'd better have an incredible ROI to justify that.

Re: A “frozen” dictionary for Python

#108

Post by tehnub »

morshu9001 wrote: post #104
zahlman wrote: post #62

Earlier quoted context omitted.

> Ordering, like stability in sorting, is an incredibly useful property. I can't say I've noticed any good reasons to rely on it. Didn't reach for `OrderedDict` often back in the day either. I've had more use for actual sorting than for preserving the insertion order.

Same. Recently I saw interview feedback where someone complained that the candidate used OrderedDict instead of the built-in dict that is now ordered, but they'll let it slide... As if writing code that will silently do different things depending on minor Python version is a good idea.

Well it's been guaranteed since 3.7 which came out in 2018, and 3.6 reached end-of-life in 2021, so it's been a while. I could see the advantage if you're writing code for the public (libraries, applications), but for example I know at my job my code is never going to be run with Python 3.6 or older.

Re: A “frozen” dictionary for Python

#109

Post by boothby »

morshu9001 wrote: post #103
pansa2 wrote: post #20

I wonder whether Raymond Hettinger has an opinion on this PEP. A long time ago, he wrote: "freezing dicts is a can of worms and not especially useful". https://mail.python.org/pipermail/python-dev/2006-February/0...

I agree, same with frozenset. If you really want to use one of those as a key, convert to a tuple. There might be niche use cases for all this, but it's not something that the language or even the standard lib need to support.

Problem being that sets aren't consistently ordered and conversion to a tuple can result in an exponential (specifically, factorial) explosion in the number of possible keys associated with a single set. Nor can you sort all objects. Safe conversion of sets to tuples for use as keys is possible but the only technique I know requires an auxiliary store of objects (mapping objects to the order in which they were first observed), which doesn't parallelize well.

Re: A “frozen” dictionary for Python

#110

Post by rented_mule »

perrygeo wrote: post #101
shadowgovt wrote: post #80

Earlier quoted context omitted.

If your dicts are frozen, you shouldn't need to deep-copy. The point of immutability is that if you want a new frozendict based on another one, you just rebuild the indirection data structure up top and leave the values it references alone.

You're absolutely right: an "indirection data structure" is necessary. Freezing the data is the least interesting part - it doesn't give you any of the benefits typically associated with immutable data structures in functional languages. That's my point - Python is shipping a half solution that's being mistaken for a proper one. You think Python developers are going to roll their own HAMT on top of frozendicts? Or ar…

It's a little early to say that "Python is shipping" anything related to this. The PEP is still in draft status and it may be modified (as it was three hours ago) or even rejected. The article says it is likely to be accepted, but that has not happened yet. That also means there is time to comment on the PEP in the discussion it links to if you'd like.

https://peps.python.org/pep-0814/

Post reply on HN