Live data from Hacker News

Don't let dicts spoil your code (2020)

roman.pt

11–20 of 91 posts

Re: Don't let dicts spoil your code (2020)

#11
post #8

I don't program professionally, and I struggle with dicts and classes. On one hand, I want to avoid the Java world of needing to learn 8 new classes to use any library. So dicts are lightweight and extensible and feel like the modern way of doing things. One the other hand, all the problems listed in the article are right. You really do need to document different expected dicts somehow, which is basically structs/cla…

> You really do need to document different expected dicts somehow, which is basically structs/classes.

As someone who does program professionally, my experience is that if you find yourself needing to do this (and the expectations aren't runtime-dynamic) you've almost certainly gone astray, and you'll eventually find yourself implementing a much poorer version of a type system anyway. Dict contents should always be treated as optional. If you have a case where you have 2 required keys and then a bunch of optional ones, define a struct/class that has those 2 fields and then a dict/map for extra values.

The only reason one might want a "schema" for maps is when you're dealing with something config-driven; for instance, in implementing a SQL engine, or assembling inputs to an ML model. Even then, your code shouldn't have expectations on specific contents, other than that the keys must be the same as some other map/list.

A few other similar rules I follow with weak/primitive types:

* Strings are opaque blobs. The only valid operation on strings is to test two strings for equality. No parsing, no checking "does it have a prefix", no concatenation - if you need comparisons that take multiple elements into account, write a richer type. The exceptions are in implementing wire protocols, and rendering data for human consumption.

* Booleans are not allowed as function arguments or member fields. Define a custom enum instead. You'll almost always end up wanting at least a third possible value. Even if not, it's useful to make the default state some form of invalid, so that you don't have to guess at whether a field is false because you meant false, or false because you forgot to set it.

* If there's a value property that you rely on (e.g., lists being sorted, strings being capitalized, integers being in some range), and that property needs to be preserved across a function-call boundary, wrap it in a type. It doesn't force correctness (unless you're using a language with dependent types), but it's at least stickier than a comment.

Re: Don't let dicts spoil your code (2020)

#12
post #7

The article is not explaining the point, which I believe is: type your dicts if you want to provide strict guarantees to your downstream about data shape. If you know precisely what the data is used for - great, go ahead - type system is your friend. If you don't know how the data should be used, it's often a different story. Wrapping data in hand typed classes is a terrible idea in the typical data engineering scena…

This sounds specific to a particular company's organization where there are at least three different systems involved and no single source of truth. It seems like that's a problem in itself - how do you get everyone to refer to and update the same document?

Ideally everyone would be using a single type definition. Admittedly that's more common with protobufs, though, where you can't send any data that's not in the definition.

Come to think of that, it's true of plain old structs too.

Re: Don't let dicts spoil your code (2020)

#14
post #3

Earlier quoted context omitted.

Who needs hashes when you've got variable variables? ~ me, 20 years ago, learning the hard way

I still don't know how I feel about the fact that in PHP $$var (a) works, and (b) does exactly what you'd expect.

Early days of PHP were total madness. GET vars initialized variables in the global namespace. Uninitialized variables treated as zero/empty string. People were rolling their own query sanitizers in the best case. Misconfigured servers often horked out script sources. Why not pour some gas on it with variable variables?

Re: Don't let dicts spoil your code (2020)

#15
Yes to the principle. But typed dict is useful for more than just "the wire".

There are places where you just dont need the overhead of a class. Yes slotted classes make this much cheaper but so do named tuples.

If the behavior of a thing is to map values then it should stay a dict.

If the behavior is a bag of attributes then yes pick something better.

Re: Don't let dicts spoil your code (2020)

#16
Python's strapped on type annotations have been designed around traditional OOP, and it feels like a bad fit for the language. Duck typing is a tremendously powerful form of polymorphism, and none of the PEPs for type annotations do a great job of supporting it. Protocols don't work well with dataclasses and not at all with dicts. TypedDicts could have been perfect, but they explicitly disallow extra keys. Why even use a TypedDict instead of a dataclass? Why make yet another traditional OOP abstraction that was already well served by multiple other features of the language? Even more frustratingly, TypedDicts show that it could have been done. They just decided to break it on purpose.

TFA accidentally even brings up the reason by dicts are so powerful: they enable easy interoperability between libraries (like a wire format). Using two libraries together that insist on their own bespoke class hierarchy is an exercise in data conversion pain. Further, if I want a point to be an object containing fields for "x" and "y", I'd much rather just use a dict rather than construct an object in some incompatible inheritance nightmare.

Re: Don't let dicts spoil your code (2020)

#17
post #2

Took me a really long time to learn this lesson. IMO this is a variation of the primitive obsession code smell, although I'd say it's way more harmful. I was really reluctant to add data classes to my code when the good old PHP array could get the job done without holding me up with a bunch of beaurocracy. Of course they give no guarantees and enforce no structure, so inevitably you get slight variations depending on…

> so inevitably you get slight variations depending on what you need If you have a generic collection, you know it's generic. It does remove a class of errors when you start adding types, but it also adds problems in making changes as a tradeoff. Now I have to make a PR that is the change I want AND I have to modify the type, which comes with explaining/understanding that there isn't a reason to use 2 different types…

I'm not sure if I'm misunderstanding your point or you're understanding mine, so I'll just carefully say that the PHP array is really a dictionary + array combination type, and I was referring specifically to its use as a dictionary (since that's what TFA is about). If you're returning a list of things that are all the same type I agree that an array, or an array of a certain type if generics are available is totally fine and serviceable.

but if you're passing in/out some monstrosity which has a structure that you can only really find out by reading the code, often from top to bottom if different parts of the dictionary are referenced in different parts of the code, you are really setting yourself up for trouble down the line.

Re: Don't let dicts spoil your code (2020)

#18
post #16

Python's strapped on type annotations have been designed around traditional OOP, and it feels like a bad fit for the language. Duck typing is a tremendously powerful form of polymorphism, and none of the PEPs for type annotations do a great job of supporting it. Protocols don't work well with dataclasses and not at all with dicts. TypedDicts could have been perfect, but they explicitly disallow extra keys. Why even u…

Prior to dataclasses, didn’t the library attrs come about to address a gap, and then dataclasses were added from inspiration from attrs? I mean yea, ideally the best structures were designed from the start, but the history is understandable.

Re: Don't let dicts spoil your code (2020)

#19
post #16

Python's strapped on type annotations have been designed around traditional OOP, and it feels like a bad fit for the language. Duck typing is a tremendously powerful form of polymorphism, and none of the PEPs for type annotations do a great job of supporting it. Protocols don't work well with dataclasses and not at all with dicts. TypedDicts could have been perfect, but they explicitly disallow extra keys. Why even u…

Prior to dataclasses, didn’t the library attrs come about to address a gap, and then dataclasses were added from inspiration from attrs? I mean yea, ideally the best structures were designed from the start, but the history is understandable.

Dataclasses came out in 3.7, and TypedDicts and Protocols in 3.8. I had to check. I knew they were pretty close.

Re: Don't let dicts spoil your code (2020)

#20
When JavaScript added hash/object deconstruction (both at the argument level and assigning variables) I noticed code has been using Dict-like function arguments everywhere. It makes typing them a bit more of a pain in the ass (especially without default arguments).

I haven’t decided if I like it better than just breaking up objects into arguments in a more simple functional style.

On one hand it’s more predictable but on the other most complex apps start passing around objects for everything. Typescript of course helps with that, as does nearly modularized code (ie not passing in full typed objects outside of the parent module which owns/generates them unless they uniquely operate on the full object).

These are the small rescissions you end up making a hundred times.

Post reply on HN