Or just dump pydantic and use msgspec instead: https://jcristharif.com/msgspec/
[deleted]
Loading Pydantic models from JSON without running out of memory
31–40 of 51 posts
Re: Loading Pydantic models from JSON without running out of memory
#32Having only recently encountered this, does anyone have any insight as to why it takes 2GB to handle a 100MB file? This looks highly reminiscent (though not exactly the same, pedants) of why people used to get excited about using SAX instead of DOM for xml parsing.
I talk about this more explicitly in the PyCon talk ( https://pythonspeed.com/pycon2025/slides/ - video soon) though that's not specifically about Pydantic, but basically: 1. Inefficient parser implementation. It's just... very easy to allocate way too much memory if you don't think about large-scale documents, and very difficult to measure. Common problem with many (but not all) JSON parsers. 2. CPython in-memory re…
Funny to see awkward array in this context! (And... do people really store giant datasets in json?!?).
Re: Loading Pydantic models from JSON without running out of memory
#33Earlier quoted context omitted.
I talk about this more explicitly in the PyCon talk ( https://pythonspeed.com/pycon2025/slides/ - video soon) though that's not specifically about Pydantic, but basically: 1. Inefficient parser implementation. It's just... very easy to allocate way too much memory if you don't think about large-scale documents, and very difficult to measure. Common problem with many (but not all) JSON parsers. 2. CPython in-memory re…
Funny to see awkward array in this context! (And... do people really store giant datasets in json?!?).
My sweet summer child
Re: Loading Pydantic models from JSON without running out of memory
#34How does the speed of the dataclass version compare?
Re: Loading Pydantic models from JSON without running out of memory
#35Earlier quoted context omitted.
I talk about this more explicitly in the PyCon talk ( https://pythonspeed.com/pycon2025/slides/ - video soon) though that's not specifically about Pydantic, but basically: 1. Inefficient parser implementation. It's just... very easy to allocate way too much memory if you don't think about large-scale documents, and very difficult to measure. Common problem with many (but not all) JSON parsers. 2. CPython in-memory re…
Funny to see awkward array in this context! (And... do people really store giant datasets in json?!?).
Often the legacy of an engineer (or team) who "did what they had to do" to meet a deadline, and if they wanted to migrate to something better post-launch, weren't allowed to allocate time to go back and do so.
At least JSON or CSV is better than the ad hoc homegrown formats you found at medium-sized companies that came out of the 90's and 00's.
Re: Loading Pydantic models from JSON without running out of memory
#36Earlier quoted context omitted.
You can't add extra attributes that weren't part of the original dataclass definition: >>> from dataclasses import dataclass >>> @dataclass ... class C: pass ... >>> C().x = 1 >>> @dataclass(slots=True) ... class D: pass ... >>> D().x = 1 Traceback (most recent call last): File " ", line 1, in D().x = 1 ^^^^^ AttributeError: 'D' object has no attribute 'x' and no __dict__ for setting new attributes Most of the time t…
I rarely need to dynamically add attributes myself on dataclasses like this but unfortunately this also means things like `@cached_property` won't work because it can't internally cache the method result anywhere.
IIRC you can just include a __dict__ slot and @cached_property should start working again. I
Re: Loading Pydantic models from JSON without running out of memory
#37Having only recently encountered this, does anyone have any insight as to why it takes 2GB to handle a 100MB file? This looks highly reminiscent (though not exactly the same, pedants) of why people used to get excited about using SAX instead of DOM for xml parsing.
[deleted]
Re: Loading Pydantic models from JSON without running out of memory
#38Or just dump pydantic and use msgspec instead: https://jcristharif.com/msgspec/
Can it do incremental parsing? Cant tell from a brief look.
Re: Loading Pydantic models from JSON without running out of memory
#39Earlier quoted context omitted.
I talk about this more explicitly in the PyCon talk ( https://pythonspeed.com/pycon2025/slides/ - video soon) though that's not specifically about Pydantic, but basically: 1. Inefficient parser implementation. It's just... very easy to allocate way too much memory if you don't think about large-scale documents, and very difficult to measure. Common problem with many (but not all) JSON parsers. 2. CPython in-memory re…
Funny to see awkward array in this context! (And... do people really store giant datasets in json?!?).
Some people even use AI-generated JSON as a semantic layer over their SQL.
Re: Loading Pydantic models from JSON without running out of memory
#40Having only recently encountered this, does anyone have any insight as to why it takes 2GB to handle a 100MB file? This looks highly reminiscent (though not exactly the same, pedants) of why people used to get excited about using SAX instead of DOM for xml parsing.
To take 2GB to parse a 100MB file, we increase file size 20x
Let's imagine the file is mostly full of single digit numbers with no spaces (so lists like 2,4,1,0,9,3...).
We need to spend 40 bytes storing a number.
Make a minimal sized class to store an integer:
class JsonInt:
x = 1
Usually we store floats from JSON, the size of 1 as a float in python is 24 bytes.
Now, you can get smaller, but as soon as you introduce any kind of class structure or not parsing numbers until they are used (in case you want people to be able to intrepret them as ints or floats), you blow through 20x memory size increase.