Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Loading Pydantic models from JSON without running out of memory

pythonspeed.com

Post reply on HN
Print view View original

21–30 of 51 posts

Re: Loading Pydantic models from JSON without running out of memory

#21

Post by shakna »

cbcoutinho wrote: post #15
jmugan wrote: post #8

My problem isn't running out of memory; it's loading in a complex model where the fields are BaseModels and unions of BaseModels multiple levels deep. It doesn't load it all the way and leaves some of the deeper parts as dictionaries. I need like almost a parser to search the space of different loads. Anyone have any ideas for software that does that?

At some point, we have to admit we're asking too much from our tools. I know nothing about your context, but in what context would a single model need to support so many permutations of a data structure? Just because software can, doesn't mean it should.

Anything multi-tenant? There's a reason Salesforce is used for so many large organisations. The multi-nesting lets you account for all the descrepancies that come with scale.

Just tracking payments through multiple tax regions will explode the places where things need to be tweaked.

Re: Loading Pydantic models from JSON without running out of memory

#22

Post by not_skynet »

jmugan wrote: post #8

My problem isn't running out of memory; it's loading in a complex model where the fields are BaseModels and unions of BaseModels multiple levels deep. It doesn't load it all the way and leaves some of the deeper parts as dictionaries. I need like almost a parser to search the space of different loads. Anyone have any ideas for software that does that?

going to shamelessly plug my own library here: https://github.com/mivanit/ZANJ

You can have nested dataclasses, as well as specify custom serializers/loaders for things which aren't natively supported by json.

Re: Loading Pydantic models from JSON without running out of memory

#23

Post by jmugan »

causasui wrote: post #12
jmugan wrote: post #8

My problem isn't running out of memory; it's loading in a complex model where the fields are BaseModels and unions of BaseModels multiple levels deep. It doesn't load it all the way and leaves some of the deeper parts as dictionaries. I need like almost a parser to search the space of different loads. Anyone have any ideas for software that does that?

You probably want to use Discriminated Unions https://docs.pydantic.dev/latest/concepts/unions/#discrimina...

Yeah, I'm doing that

Re: Loading Pydantic models from JSON without running out of memory

#24

Post by jmugan »

enragedcacti wrote: post #13
jmugan wrote: post #8

My problem isn't running out of memory; it's loading in a complex model where the fields are BaseModels and unions of BaseModels multiple levels deep. It doesn't load it all the way and leaves some of the deeper parts as dictionaries. I need like almost a parser to search the space of different loads. Anyone have any ideas for software that does that?

The only reason I can think of for the behavior you are describing is if one of the unioned types at some level of the hierarchy is equivalent to Dict[str, Any]. My understanding is that Pydantic will explore every option provided recursively and raise a ValidationError if none match but will never just give up and hand you a partially validated object. Are you able to share a snippet that reproduces what you're seei…

That's an interesting idea. It's possible there's a Dict[str,Any] in there. And yeah, my assumption was that it tried everything recursively, but I just wasn't seeing that, and my LLM council said that it did not. But I'll check for a Dict[str,Any]. Unfortunately, I don't have a minimal example, but making one should be my next step.

Re: Loading Pydantic models from JSON without running out of memory

#25

Post by jmugan »

not_skynet wrote: post #22
jmugan wrote: post #8

My problem isn't running out of memory; it's loading in a complex model where the fields are BaseModels and unions of BaseModels multiple levels deep. It doesn't load it all the way and leaves some of the deeper parts as dictionaries. I need like almost a parser to search the space of different loads. Anyone have any ideas for software that does that?

going to shamelessly plug my own library here: https://github.com/mivanit/ZANJ You can have nested dataclasses, as well as specify custom serializers/loaders for things which aren't natively supported by json.

Ah, but I need something JSON-based.

Re: Loading Pydantic models from JSON without running out of memory

#26

Post by fidotron »

Having only recently encountered this, does anyone have any insight as to why it takes 2GB to handle a 100MB file?

This looks highly reminiscent (though not exactly the same, pedants) of why people used to get excited about using SAX instead of DOM for xml parsing.

Re: Loading Pydantic models from JSON without running out of memory

#27

Post by enragedcacti »

jmugan wrote: post #24
enragedcacti wrote: post #13

Earlier quoted context omitted.

The only reason I can think of for the behavior you are describing is if one of the unioned types at some level of the hierarchy is equivalent to Dict[str, Any]. My understanding is that Pydantic will explore every option provided recursively and raise a ValidationError if none match but will never just give up and hand you a partially validated object. Are you able to share a snippet that reproduces what you're seei…

That's an interesting idea. It's possible there's a Dict[str,Any] in there. And yeah, my assumption was that it tried everything recursively, but I just wasn't seeing that, and my LLM council said that it did not. But I'll check for a Dict[str,Any]. Unfortunately, I don't have a minimal example, but making one should be my next step.

One thing to watch out for while you debug is that the default 'smart' mode for union discrimination can be very unintuitive. As you can see in this example, an int vs a string can cause a different model to be chosen two layers up even though both are valid. You may have perfectly valid uses of Dict within your model that are being chosen in error because they result in less type coercion. left_to_right mode (or ideally discriminated unions if your data has easy discriminators) will be much more consistent.

    >>> class A(BaseModel):
    >>>     a: int
    >>> class B(BaseModel):
    >>>     b: A
    >>> class C(BaseModel):
    >>>     c: B | Dict[str, Any]

    >>> C.model_validate({'c':{'b':{'a':1}}})
    
    C(c=B(b=A(a=1)))

    >>> C.model_validate({'c':{'b':{'a':"1"}}})

    C(c={'b': {'a': '1'}})

    >>> class C(BaseModel):
    >>>     c: B | Dict[str, Any] = Field(union_mode='left_to_right')
    
    >>> C.model_validate({'c':{'b':{'a':"1"}}})

    C(c=B(b=A(a=1)))

Re: Loading Pydantic models from JSON without running out of memory

#28

Post by not_skynet »

jmugan wrote: post #25
not_skynet wrote: post #22

Earlier quoted context omitted.

going to shamelessly plug my own library here: https://github.com/mivanit/ZANJ You can have nested dataclasses, as well as specify custom serializers/loaders for things which aren't natively supported by json.

Ah, but I need something JSON-based.

It does allow dumping to/recovering from json, apologies if that isn't well documented.

Calling `x: str = json.dumps(MyClass(...).serialize())` will get you json you can recover to the original object, nested classes and custom types and all, with `MyClass.load(json.loads(x))`

Re: Loading Pydantic models from JSON without running out of memory

#29

Post by itamarst »

fidotron wrote: post #26

Having only recently encountered this, does anyone have any insight as to why it takes 2GB to handle a 100MB file? This looks highly reminiscent (though not exactly the same, pedants) of why people used to get excited about using SAX instead of DOM for xml parsing.

I talk about this more explicitly in the PyCon talk (https://pythonspeed.com/pycon2025/slides/ - video soon) though that's not specifically about Pydantic, but basically:

1. Inefficient parser implementation. It's just... very easy to allocate way too much memory if you don't think about large-scale documents, and very difficult to measure. Common problem with many (but not all) JSON parsers.

2. CPython in-memory representation is large compared to compiled languages. So e.g. 4-digit integer is 5-6 bytes in JSON, 8 in Rust if you do i64, 25ish in CPython. An empty dictionary is 64 bytes.

Re: Loading Pydantic models from JSON without running out of memory

#30

Post by deepsquirrelnet »

Alternatively, if you had to go with json, you could consider using jsonl. I think I’d start by evaluating whether this is a good application for json. I tend to only want to use it for small files. Binary formats are usually much better in this scenario.
Post reply on HN