Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Loading Pydantic models from JSON without running out of memory

pythonspeed.com

Post reply on HN
Print view View original

41–50 of 51 posts

Re: Loading Pydantic models from JSON without running out of memory

#41

Post by scolvin »

Pydantic author here. We have plans for an improvement to pydantic where JSON is parsed iteratively, which will make way for reading a file as we parse it. Details in https://github.com/pydantic/pydantic/issues/10032.

Our JSON parser, jiter (https://github.com/pydantic/jiter) already supports iterative parsing, so it's "just" a matter of solving the lifetimes in pydantic-core to validate as we parse.

This should make pydantic around 3x faster at parsing JSON and significantly reduce the memory overhead.

Re: Loading Pydantic models from JSON without running out of memory

#42

Post by Lucasoato »

scolvin wrote: post #41

Pydantic author here. We have plans for an improvement to pydantic where JSON is parsed iteratively, which will make way for reading a file as we parse it. Details in https://github.com/pydantic/pydantic/issues/10032 . Our JSON parser, jiter ( https://github.com/pydantic/jiter ) already supports iterative parsing, so it's "just" a matter of solving the lifetimes in pydantic-core to validate as we parse. This should m…

Pydantic is a life changing library, thanks so much for your work!

Re: Loading Pydantic models from JSON without running out of memory

#43

Post by fidotron »

CJefferson wrote: post #40
fidotron wrote: post #26

Having only recently encountered this, does anyone have any insight as to why it takes 2GB to handle a 100MB file? This looks highly reminiscent (though not exactly the same, pedants) of why people used to get excited about using SAX instead of DOM for xml parsing.

To take 2GB to parse a 100MB file, we increase file size 20x Let's imagine the file is mostly full of single digit numbers with no spaces (so lists like 2,4,1,0,9,3...). We need to spend 40 bytes storing a number. Make a minimal sized class to store an integer: class JsonInt: x = 1 That object's size is already 48 bytes. Usually we store floats from JSON, the size of 1 as a float in python is 24 bytes. Now, you can g…

> We need to spend 40 bytes storing a number.

But . . . why? Assuming they aren't BigInts or similar these are maximum 8 bytes of actual data. This overhead is ridiculous.

Using classes should enable you to be much smaller than the JSON representation, not larger. For example, V8 does it like https://v8.dev/docs/hidden-classes

> not parsing numbers until they are used

Doesn't this defeat the point of pydantic? It's supposed to be checking the model is valid as it's loaded using jiter. If the data is valid it can be loaded into an efficient representation, and if it's not the errors can be emitted during iterating over it.

Re: Loading Pydantic models from JSON without running out of memory

#44

Post by tomrod »

itamarst wrote: post #6
thisguy47 wrote: post #4

I'd like to see a comparison of ijson vs just `json.load(f)`. `ujson` would also be interesting to see.

For my PyCon 2025 talk I did this. Video isn't up yet, but slides are here: https://pythonspeed.com/pycon2025/slides/ The linked-from-original-article ijson article was the inspiration for the talk: https://pythonspeed.com/articles/json-memory-streaming/

I have a side question -- what did you use for slides?

Re: Loading Pydantic models from JSON without running out of memory

#45

Post by adeeshaek »

Lucasoato wrote: post #42
scolvin wrote: post #41

Pydantic author here. We have plans for an improvement to pydantic where JSON is parsed iteratively, which will make way for reading a file as we parse it. Details in https://github.com/pydantic/pydantic/issues/10032 . Our JSON parser, jiter ( https://github.com/pydantic/jiter ) already supports iterative parsing, so it's "just" a matter of solving the lifetimes in pydantic-core to validate as we parse. This should m…

Pydantic is a life changing library, thanks so much for your work!

Seconded. Please keep up the awesome work!

Re: Loading Pydantic models from JSON without running out of memory

#46

Post by jerf »

fidotron wrote: post #43
CJefferson wrote: post #40

Earlier quoted context omitted.

To take 2GB to parse a 100MB file, we increase file size 20x Let's imagine the file is mostly full of single digit numbers with no spaces (so lists like 2,4,1,0,9,3...). We need to spend 40 bytes storing a number. Make a minimal sized class to store an integer: class JsonInt: x = 1 That object's size is already 48 bytes. Usually we store floats from JSON, the size of 1 as a float in python is 24 bytes. Now, you can g…

> We need to spend 40 bytes storing a number. But . . . why? Assuming they aren't BigInts or similar these are maximum 8 bytes of actual data. This overhead is ridiculous. Using classes should enable you to be much smaller than the JSON representation, not larger. For example, V8 does it like https://v8.dev/docs/hidden-classes > not parsing numbers until they are used Doesn't this defeat the point of pydantic? It's s…

"But . . . why?"

This is CPython. This is how it works. It's not particularly related to JSON. That sort of overhead is put on everything. It just hurts the most when the thing you're putting the overhead on is a single integer. It hurts less when you're doing it to, say, a multi-kilobyte string.

Even in your v8 example, that's a JIT optimization, not "how the language works". You break that optimization, which you can do at any moment with any change in your code base, you're back to similar sizes.

Boxing everything lets you easily implement the dynamic scripting language's way of treating everything as an Object of some sort, but it comes at a price. There's a reason dynamic scripting languages, even after the JIT has come through, are generally substantially slower languages. This isn't the only reason, but it's a significant part of it.

Re: Loading Pydantic models from JSON without running out of memory

#47

Post by itamarst »

tomrod wrote: post #44
itamarst wrote: post #6

Earlier quoted context omitted.

For my PyCon 2025 talk I did this. Video isn't up yet, but slides are here: https://pythonspeed.com/pycon2025/slides/ The linked-from-original-article ijson article was the inspiration for the talk: https://pythonspeed.com/articles/json-memory-streaming/

I have a side question -- what did you use for slides?

https://remarkjs.com/

Re: Loading Pydantic models from JSON without running out of memory

#48

Post by fidotron »

jerf wrote: post #46
fidotron wrote: post #43

Earlier quoted context omitted.

> We need to spend 40 bytes storing a number. But . . . why? Assuming they aren't BigInts or similar these are maximum 8 bytes of actual data. This overhead is ridiculous. Using classes should enable you to be much smaller than the JSON representation, not larger. For example, V8 does it like https://v8.dev/docs/hidden-classes > not parsing numbers until they are used Doesn't this defeat the point of pydantic? It's s…

"But . . . why?" This is CPython. This is how it works. It's not particularly related to JSON. That sort of overhead is put on everything. It just hurts the most when the thing you're putting the overhead on is a single integer. It hurts less when you're doing it to, say, a multi-kilobyte string. Even in your v8 example, that's a JIT optimization, not "how the language works". You break that optimization, which you c…

> Even in your v8 example, that's a JIT optimization, not "how the language works". You break that optimization, which you can do at any moment with any change in your code base, you're back to similar sizes.

The whole point of the v8 optimization is it works in the face of prototype chains that merge etc. as you add new fields dynamically so if you change your code base it adapts.

Re: Loading Pydantic models from JSON without running out of memory

#49

Post by jtmcivor »

aitchnyu wrote: post #38
m_ke wrote: post #9

Or just dump pydantic and use msgspec instead: https://jcristharif.com/msgspec/

Can it do incremental parsing? Cant tell from a brief look.

IIUC:

* You still need to load all the bytes into memory before passing to msgspec decoding

* You can decode a subset of fields, which is really helpful

* Reusing msgspec decoders saves some cpu cycles https://jcristharif.com/msgspec/perf-tips.html#reuse-encoder...

Slides 17, 18, 19 have an example of the first two points https://pythonspeed.com/pycon2025/slides/#17

Re: Loading Pydantic models from JSON without running out of memory

#50

Post by [deleted] »

chao- wrote: post #35
cozzyd wrote: post #32

Earlier quoted context omitted.

Funny to see awkward array in this context! (And... do people really store giant datasets in json?!?).

Often the legacy of an engineer (or team) who "did what they had to do" to meet a deadline, and if they wanted to migrate to something better post-launch, weren't allowed to allocate time to go back and do so. At least JSON or CSV is better than the ad hoc homegrown formats you found at medium-sized companies that came out of the 90's and 00's.

[deleted]
Post reply on HN