Pydantic author here. We have plans for an improvement to pydantic where JSON is parsed iteratively, which will make way for reading a file as we parse it. Details in https://github.com/pydantic/pydantic/issues/10032 . Our JSON parser, jiter ( https://github.com/pydantic/jiter ) already supports iterative parsing, so it's "just" a matter of solving the lifetimes in pydantic-core to validate as we parse. This should m…
That's great! Would also be cool (separately from Pydantic use case) to add jiter backend to ijson.