Live data from Hacker News

A Python dict that can report which keys you did not use

peterbe.com

21–30 of 45 posts

Re: A Python dict that can report which keys you did not use

#22
post #5

Just a heads up, this fails to track usage of get and setdefault . The ability to iterate over dicts makes the whole question rather murky.

Along with those and iteration, it also would need to handle del/pop/popitem/update/copy/or/ror/... some of which might necessitate a decision on whether comparisons/repr also count as access.

Re: A Python dict that can report which keys you did not use

#23
post #5

Just a heads up, this fails to track usage of get and setdefault . The ability to iterate over dicts makes the whole question rather murky.

Indeed. Inheriting from 'collections.UserDict' instead of 'dict' will make TFA's code work as intended for most of those edge cases.

UserDict will route '.get', '.setdefault', and even iteration via '.items()' through the '__getitem__' method.

edited to remove "(maybe all?) edge cases". As soon as I posted, I thought of several less common/obvious edge cases.

Re: A Python dict that can report which keys you did not use

#24
post #16

Earlier quoted context omitted.

setdefault was a go to method before defaultdict was added to the collections module in Python 2.5, which replaced the biggest use case.

It's been some time since I last benchmarked defaultdict but last time I did (circa 3.6 and less?), it was considerably slower than judicious use of setdefault.

One time that defaultdict may come out ahead is if the default value is expensive to construct and rarely needed:

    d.setdefault(k, computevalue())
defaultdict takes a factory function, so it's only called if the key is not already present:

    d = defaultdict(computevalue)
This applies to some extent even if the default value is just an empty dictionary (as it often is in my experience). You can use dict() as the factory function in that case.

But I have never benchmarked!

Re: A Python dict that can report which keys you did not use

#25
Only tangentially related but I am really excited about PEP 764¹ (inline typed dictionaries). If it gets accepted, we can finally replace entire hierarchies of dataclasses with simple nested dictionary types and call it a day.

I am currently teaching (typed) Python to a team of Windows sysadmins and it's been incredibly difficult to explain when to use a dataclass, a NamedTuple, a Pydantic model, or a dictionary.

¹) https://peps.python.org/pep-0764/

Re: A Python dict that can report which keys you did not use

#26
post #16

Earlier quoted context omitted.

It's been some time since I last benchmarked defaultdict but last time I did (circa 3.6 and less?), it was considerably slower than judicious use of setdefault.

One time that defaultdict may come out ahead is if the default value is expensive to construct and rarely needed: d.setdefault(k, computevalue()) defaultdict takes a factory function, so it's only called if the key is not already present: d = defaultdict(computevalue) This applies to some extent even if the default value is just an empty dictionary (as it often is in my experience). You can use dict() as the factory…

> if the default value is expensive to construct and rarely needed:

I'd say "or" rather than "and": defaultdict has higher overhead to initialise the default (especially if you don't need a function call in the setdefault call) but because it uses a fallback of dict lookup it's essentially free if you get a hit. As a result, either a very high redundancy with a cheap default or a low amount of redundancy with a costly default will have the defaultdict edge out.

For the most extreme case of the former,

    d = {}
    for i in range(N):
        d.setdefault(0, [])
versus

    d = defaultdict(list)
    for i in range(N):
        d[0]
has the defaultdict edge out at N=11 on my machine (561ns for setdefault versus 545 for defaultdict). And that's with a literal list being quite a bit cheaper than a list() call.

Re: A Python dict that can report which keys you did not use

#27

Only tangentially related but I am really excited about PEP 764¹ (inline typed dictionaries). If it gets accepted, we can finally replace entire hierarchies of dataclasses with simple nested dictionary types and call it a day. I am currently teaching (typed) Python to a team of Windows sysadmins and it's been incredibly difficult to explain when to use a dataclass, a NamedTuple, a Pydantic model, or a dictionary. ¹)…

Do you seriously have difficulties explaining when to use a class and when to use a dictionary?!

Re: A Python dict that can report which keys you did not use

#28

Only tangentially related but I am really excited about PEP 764¹ (inline typed dictionaries). If it gets accepted, we can finally replace entire hierarchies of dataclasses with simple nested dictionary types and call it a day. I am currently teaching (typed) Python to a team of Windows sysadmins and it's been incredibly difficult to explain when to use a dataclass, a NamedTuple, a Pydantic model, or a dictionary. ¹)…

Do you seriously have difficulties explaining when to use a class and when to use a dictionary?!

You can create dictionaries on the fly. But dataclass objects require defining that dataclass first. The type safety (and LSP support) story for accessing individual dataclass fields is better than for accessing dict items (sometimes even when they are TypedDicts), but for iterating over all fields it's worse. dataclasses are nominal types and can contain additional logic, TypedDicts are structural ones, overall simpler, can be more convenient and lead to looser coupling. Dataclasses use metaclass and decorator magic while TypedDics are just plain dicts. Etc.

Let me make this more concrete: Those sysadmins frequently need to process and pass around complex (as in heavily nested) structured data. The data often comes in the form of singleton objects, i.e. they are built in single place, then used in another place and then thrown away (or merged into some other structure). In other words, any class hierarchy you build represents boilerplate code you'll only ever use once and which will be annoying to maintain as you refactor your code. Do you pick dataclasses or TypedDicts (or something else) for your map data structures?

In TypeScript you would just use `const data = as const` and be done with it.

Re: A Python dict that can report which keys you did not use

#29

Only tangentially related but I am really excited about PEP 764¹ (inline typed dictionaries). If it gets accepted, we can finally replace entire hierarchies of dataclasses with simple nested dictionary types and call it a day. I am currently teaching (typed) Python to a team of Windows sysadmins and it's been incredibly difficult to explain when to use a dataclass, a NamedTuple, a Pydantic model, or a dictionary. ¹)…

Do you seriously have difficulties explaining when to use a class and when to use a dictionary?!

The line is seriously blurred.

Re: A Python dict that can report which keys you did not use

#30
AI front: We have models to generate pictures, videos and code. We have the best devs and are so fskin rich!

Rust front: Here's a faster ls called ls-rs with different defaults, you should use this!

Go front: Here's reverse proxy #145728283 it is an open source project that has slightly different parameters than all the others.

Python hobo front: Uhh guys here's a dict that kinda might remember what you've accessed if you used it in a particular way.

Post reply on HN