Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't let dicts spoil your code

roman.pt

Post reply on HN
Print view View original

41–50 of 121 posts

Re: Don't let dicts spoil your code

#41

Post by mjr00 »

cle wrote: post #36

Dicts can be a problem, but this particular example isn't that great, like in this diagram from the article: External API Ser/De Business Logic Life's all great until "External API" adds a field that your model doesn't know about, it gets dropped when you deserialize it, and then when you send it back (or around somewhere else) it's missing a field. There's config for this in Pydantic, but it's not the default, and i…

If external API adds a new field but your software already worked, you didn't need it in the first place, so why should it matter?

Dropping unknown/unused fields makes sense in 99% of cases.

Re: Don't let dicts spoil your code

#42

Post by xenoxcs »

I'm a big fan of using Protobuf for the third-party API validation task. After some slightly finniky initial schema definition (helped by things like json-to-proto.github.io), I can be sure the data I'm consuming from an external API is strongly typed, and the functions included in Protobuf which convert JSON to a Proto message instance blows up by default if there's an unexpected field in the API data it's consuming.

I use it to parse and validate incoming webhook data in my Python AWS Lambda functions, then re-use the protobuf types when I later ship the webhook data to our Flutter-based frontend. Adding extensions to the protobuf fields gives me a nice, structured way to add flags and metadata to different fields in the webhook message. For example, I can add table & column names to the protobuf message fields, and have them automatically be populated from the DB with some simple helper functions. Avoids me needing to write many lines of code that look like:

MyProtoClass.field1 = DB.table.column1.val

MyProtoClass.field2 = DB.table.column2.val

Re: Don't let dicts spoil your code

#43

Post by nesarkvechnep »

pmarreck wrote: post #6

Less important in Elixir (where they are "maps") due to the immutable nature of them as well as the Struct type which is a structured map.

Yes, usually my APIs in Elixir receive their arguments as a well-typed map, not stringly keyed, and transform them to structs which the core business logic expects.

Re: Don't let dicts spoil your code

#44

Post by pistoleer »

turnsout wrote: post #18
Waterluvian wrote: post #3

I think one really nice thing about Python is duck typing. Your interfaces are rarely asking for a dict as much as they’re asking for a dict-like. It’s pretty great how often you can worry about this kind of problem at the appropriate time (now, later, never) without much pain. There’s useful ideas in this post but I’d be careful not to throw the baby out with the bath water. Dicts are right there. There’s dict liter…

Duck typing is so fragile… Once you have implementations that are depending on your naming or property structure, you can’t update the model without breaking them all. If you use a real type, you never have to worry about this.

You would still have to update everything if you rename a field in a struct, what do you mean you never have to worry?

Re: Don't let dicts spoil your code

#45

Post by a57721 »

est wrote: post #31
Aditya_Garg wrote: post #26

Earlier quoted context omitted.

Thats only because your list has different types. Its a badly formed API and if you really need to support that use case then you can use maps and reflection to handle it.

The problem is, programmers can't dictate what JSON should look like in the wild. We used to have strict typed XML. Nobody even bothered.

> The problem is, programmers can't dictate what JSON should look like in the wild.

Not JSONs in general, but a sane API would never return something like that.

> We used to have strict typed XML. Nobody even bothered.

Nowadays there is OpenAPI, GraphQL, protobuf, etc. and people do bother about such things.

Re: Don't let dicts spoil your code

#46

Post by shiroiushi »

est wrote: post #31
Aditya_Garg wrote: post #26

Earlier quoted context omitted.

Thats only because your list has different types. Its a badly formed API and if you really need to support that use case then you can use maps and reflection to handle it.

The problem is, programmers can't dictate what JSON should look like in the wild. We used to have strict typed XML. Nobody even bothered.

>We used to have strict typed XML. Nobody even bothered.

Yeah, because it was ugly as hell and not human-readable.

Re: Don't let dicts spoil your code

#47

Post by mook »

a57721 wrote: post #45
est wrote: post #31

Earlier quoted context omitted.

The problem is, programmers can't dictate what JSON should look like in the wild. We used to have strict typed XML. Nobody even bothered.

> The problem is, programmers can't dictate what JSON should look like in the wild. Not JSONs in general, but a sane API would never return something like that. > We used to have strict typed XML. Nobody even bothered. Nowadays there is OpenAPI, GraphQL, protobuf, etc. and people do bother about such things.

Unfortunately, a lot of the time you need to deal with other people's APIs.

Re: Don't let dicts spoil your code

#48

Post by buzer »

mjr00 wrote: post #41
cle wrote: post #36

Dicts can be a problem, but this particular example isn't that great, like in this diagram from the article: External API Ser/De Business Logic Life's all great until "External API" adds a field that your model doesn't know about, it gets dropped when you deserialize it, and then when you send it back (or around somewhere else) it's missing a field. There's config for this in Pydantic, but it's not the default, and i…

If external API adds a new field but your software already worked, you didn't need it in the first place, so why should it matter? Dropping unknown/unused fields makes sense in 99% of cases.

Unfortunately some APIs assume that they will get all the fields as part of the update. If field doesn't exist in the input it gets it will drop the original value during the update.

Re: Don't let dicts spoil your code

#49

Post by travisjungroth »

jimmytucson wrote: post #14

Here’s an out-there take, but one I’ve held loosely for a long time and haven’t shed yet: dicts are not appropriate for what people mostly use them for, which is named access to member attributes. dict is an implementation of a hash table. Hash table are designed for o(1) lookup of items. As such, they are arrays which are much bigger than the number of items they store, to allow hashing items into integers and sides…

I think I once heard a Clojure talk where they were referred to as big and small maps. Small ones are what you’re comparing to arrays.

A place where dicts for hard coded keys makes sense is notebooks. The convenience is worth it and it’s unlikely to get out of hand.

Re: Don't let dicts spoil your code

#50

Post by cranium »

Python dataclasses are a good start for internal use. They are just a bit of a pain to serialize/deserialize natively. When it comes to that, I prefer to use Pydantic objects and have all the goodies, at the cost of some complexity.
Post reply on HN