Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't let dicts spoil your code (2020)

roman.pt

Post reply on HN
Print view View original

21–30 of 91 posts

Re: Don't let dicts spoil your code (2020)

#21

Post by djrobstep »

oivey wrote: post #16

Python's strapped on type annotations have been designed around traditional OOP, and it feels like a bad fit for the language. Duck typing is a tremendously powerful form of polymorphism, and none of the PEPs for type annotations do a great job of supporting it. Protocols don't work well with dataclasses and not at all with dicts. TypedDicts could have been perfect, but they explicitly disallow extra keys. Why even u…

Dealing with all these differences is one of the most frustrating, stupid things about programming today.

99% of the data i deal with on a day-to-day basis is lists and mappings.

Very conceptually simple, but with a million different implementations. Particularly in python where we have dicts, namedtuples, dataclasses, regular objects, etc etc etc, then you deal with databases (which are really just mappings of keys to rows), where the interaction works completely differently again (with annoying differences for each database of course). Then hundreds of different encodings again to send things across a network or save them to files.

None of this complexity is inherent to the problems being solved - it's all accumulated cruft and bullshit.

Re: Don't let dicts spoil your code (2020)

#22

Post by oivey »

skybrian wrote: post #12
snidane wrote: post #7

The article is not explaining the point, which I believe is: type your dicts if you want to provide strict guarantees to your downstream about data shape. If you know precisely what the data is used for - great, go ahead - type system is your friend. If you don't know how the data should be used, it's often a different story. Wrapping data in hand typed classes is a terrible idea in the typical data engineering scena…

This sounds specific to a particular company's organization where there are at least three different systems involved and no single source of truth. It seems like that's a problem in itself - how do you get everyone to refer to and update the same document? Ideally everyone would be using a single type definition. Admittedly that's more common with protobufs, though, where you can't send any data that's not in the de…

Sometimes defining who should have authority over a singular original type definition isn't possible. This is sometimes true at companies, and it's even more true in open source projects. Even when possible, single type definitions in those cases often end up as Homer-car monstrosities that are too big and difficult to construct when only a small subset of fields are needed.

Re: Don't let dicts spoil your code (2020)

#23

Post by oivey »

djrobstep wrote: post #21
oivey wrote: post #16

Python's strapped on type annotations have been designed around traditional OOP, and it feels like a bad fit for the language. Duck typing is a tremendously powerful form of polymorphism, and none of the PEPs for type annotations do a great job of supporting it. Protocols don't work well with dataclasses and not at all with dicts. TypedDicts could have been perfect, but they explicitly disallow extra keys. Why even u…

Dealing with all these differences is one of the most frustrating, stupid things about programming today. 99% of the data i deal with on a day-to-day basis is lists and mappings. Very conceptually simple, but with a million different implementations. Particularly in python where we have dicts, namedtuples, dataclasses, regular objects, etc etc etc, then you deal with databases (which are really just mappings of keys…

At least with things like databases and Pandas you can claim that there might be a valid performance reason for a different abstraction. Regular objects allow for inheritance which I usually find bad, but lots of people do like it. NamedTuples, TypedDicts, and Dataclasses are basically all rapid iterations on the same idea with the same purpose.

Re: Don't let dicts spoil your code (2020)

#24

Post by XorNot »

oivey wrote: post #22
skybrian wrote: post #12

Earlier quoted context omitted.

This sounds specific to a particular company's organization where there are at least three different systems involved and no single source of truth. It seems like that's a problem in itself - how do you get everyone to refer to and update the same document? Ideally everyone would be using a single type definition. Admittedly that's more common with protobufs, though, where you can't send any data that's not in the de…

Sometimes defining who should have authority over a singular original type definition isn't possible. This is sometimes true at companies, and it's even more true in open source projects. Even when possible, single type definitions in those cases often end up as Homer-car monstrosities that are too big and difficult to construct when only a small subset of fields are needed.

The normal way to handle this is to deserialize into your application specific type, and store extraneous data in an extra field that is private but included in reserializations.

Because your application will fail if fields you need aren't there.

Re: Don't let dicts spoil your code (2020)

#25

Post by fastball »

andreareina wrote: post #10
valbaca wrote: post #4

Interesting how Clojure takes the complete opposite approach by simply making dicts immutable. https://chasemerick.files.wordpress.com/2011/07/choosingtype...

Clojure also makes working with hashes a whole lot more ergonomic with destructuring and symbol keys.

Could you clarify a bit here? Python also has destructuring for its dicts and I'm not entirely sure what you mean by symbol keys.

Re: Don't let dicts spoil your code (2020)

#26

Post by oivey »

XorNot wrote: post #24
oivey wrote: post #22

Earlier quoted context omitted.

Sometimes defining who should have authority over a singular original type definition isn't possible. This is sometimes true at companies, and it's even more true in open source projects. Even when possible, single type definitions in those cases often end up as Homer-car monstrosities that are too big and difficult to construct when only a small subset of fields are needed.

The normal way to handle this is to deserialize into your application specific type, and store extraneous data in an extra field that is private but included in reserializations. Because your application will fail if fields you need aren't there.

That can turn into an enormous amount of work to provide all the permutations of type conversions between 3+ classes, and then manually shuffle between them over and over. It's even harder when you don't have the power to add similar conversions for the classes you're trying to convert to/from.

Classes aren't a great abstraction when enforcing program invariants like "this object must at least have fields a and b." With a dict you can just have "dict(a=1, b=2, c=3)" and it works everywhere without serialization/deserialization and manual type conversions. Python's type checkers can't provide any safety for you if you do that, but that's a deficiency in the language, not the concept.

Re: Don't let dicts spoil your code (2020)

#27

Post by lkrubner »

valbaca wrote: post #4

Interesting how Clojure takes the complete opposite approach by simply making dicts immutable. https://chasemerick.files.wordpress.com/2011/07/choosingtype...

Clojure has established the gold standard for beautiful abstractions that unify broad categories of data types. It's seq interface is elegant and powerful. Python's efforts towards option data typing or strict data typing looks especially clunky, awkward, forced, and painful when compared to Clojure.

Re: Don't let dicts spoil your code (2020)

#28

Post by geysersam »

fastball wrote: post #25
andreareina wrote: post #10

Earlier quoted context omitted.

Clojure also makes working with hashes a whole lot more ergonomic with destructuring and symbol keys.

Could you clarify a bit here? Python also has destructuring for its dicts and I'm not entirely sure what you mean by symbol keys.

Symbol keys are string keys for the more sophisticated among us. See :some-name vs "some_name". ;)

Re: Don't let dicts spoil your code (2020)

#29

Post by tomazio »

skybrian wrote: post #12
snidane wrote: post #7

The article is not explaining the point, which I believe is: type your dicts if you want to provide strict guarantees to your downstream about data shape. If you know precisely what the data is used for - great, go ahead - type system is your friend. If you don't know how the data should be used, it's often a different story. Wrapping data in hand typed classes is a terrible idea in the typical data engineering scena…

This sounds specific to a particular company's organization where there are at least three different systems involved and no single source of truth. It seems like that's a problem in itself - how do you get everyone to refer to and update the same document? Ideally everyone would be using a single type definition. Admittedly that's more common with protobufs, though, where you can't send any data that's not in the de…

This is more common than you might otherwise think. I've worked at multiple companies that have multiple systems/sources of truth for various reasons. One example of that is my current company has stored and handled all its transactional data in a legacy point of sale system from the early 90s. They decided to upgrade to a modern ERP system a couple years ago, but it takes a while to fully implement and roll over to a new source system. Especially in a high transaction system that cannot go down otherwise the company will start losing a lot of money. Thus its being incrementally rolled out, resulting in both systems running together and being read and written to simultaneously.

Re: Don't let dicts spoil your code (2020)

#30

Post by andreareina »

fastball wrote: post #25
andreareina wrote: post #10

Earlier quoted context omitted.

Clojure also makes working with hashes a whole lot more ergonomic with destructuring and symbol keys.

Could you clarify a bit here? Python also has destructuring for its dicts and I'm not entirely sure what you mean by symbol keys.

Python has tuple destructuring, which can be used with dict.items(). I'm talking about being able to destructure by name not position:

    name, address = some_dict
You could fake it if you could ask for a tuple of specific entries:

    name, address = some_dict.pluck("name", "address")
Post reply on HN