Loading Pydantic models from JSON without running out of memory
1–10 of 51 posts
Re: Loading Pydantic models from JSON without running out of memory
#2[deleted]
Re: Loading Pydantic models from JSON without running out of memory
#3[deleted]
Re: Loading Pydantic models from JSON without running out of memory
#4I'd like to see a comparison of ijson vs just `json.load(f)`. `ujson` would also be interesting to see.
Re: Loading Pydantic models from JSON without running out of memory
#5So are there downsides to just always setting slots=True on all of my python data types?
Re: Loading Pydantic models from JSON without running out of memory
#6I'd like to see a comparison of ijson vs just `json.load(f)`. `ujson` would also be interesting to see.
For my PyCon 2025 talk I did this. Video isn't up yet, but slides are here: https://pythonspeed.com/pycon2025/slides/
The linked-from-original-article ijson article was the inspiration for the talk: https://pythonspeed.com/articles/json-memory-streaming/
Re: Loading Pydantic models from JSON without running out of memory
#7So are there downsides to just always setting slots=True on all of my python data types?
You can't add extra attributes that weren't part of the original dataclass definition:
>>> from dataclasses import dataclass
>>> @dataclass
... class C: pass
...
>>> C().x = 1
>>> @dataclass(slots=True)
... class D: pass
...
>>> D().x = 1
Traceback (most recent call last):
File "", line 1, in
D().x = 1
^^^^^
AttributeError: 'D' object has no attribute 'x' and no __dict__ for setting new attributes
Re: Loading Pydantic models from JSON without running out of memory
#8My problem isn't running out of memory; it's loading in a complex model where the fields are BaseModels and unions of BaseModels multiple levels deep. It doesn't load it all the way and leaves some of the deeper parts as dictionaries. I need like almost a parser to search the space of different loads. Anyone have any ideas for software that does that?
Re: Loading Pydantic models from JSON without running out of memory
#9Or just dump pydantic and use msgspec instead: https://jcristharif.com/msgspec/
Re: Loading Pydantic models from JSON without running out of memory
#10Or just dump pydantic and use msgspec instead: https://jcristharif.com/msgspec/
msgspec is much more memory efficient out of the box, yes. Also quite fast.