11–20 of 51 posts
Or just dump pydantic and use msgspec instead: https://jcristharif.com/msgspec/
For example if you are querying a DB that returns a column as a JSON string, trivial with Pydantic to json parse the column are part of deser with an annotation.
Pydantic is definitely slower and not a 'zero cost abstraction', but you do get a lot for it.
My problem isn't running out of memory; it's loading in a complex model where the fields are BaseModels and unions of BaseModels multiple levels deep. It doesn't load it all the way and leaves some of the deeper parts as dictionaries. I need like almost a parser to search the space of different loads. Anyone have any ideas for software that does that?
My problem isn't running out of memory; it's loading in a complex model where the fields are BaseModels and unions of BaseModels multiple levels deep. It doesn't load it all the way and leaves some of the deeper parts as dictionaries. I need like almost a parser to search the space of different loads. Anyone have any ideas for software that does that?
Are you able to share a snippet that reproduces what you're seeing?
So are there downsides to just always setting slots=True on all of my python data types?
You can't add extra attributes that weren't part of the original dataclass definition: >>> from dataclasses import dataclass >>> @dataclass ... class C: pass ... >>> C().x = 1 >>> @dataclass(slots=True) ... class D: pass ... >>> D().x = 1 Traceback (most recent call last): File " ", line 1, in D().x = 1 ^^^^^ AttributeError: 'D' object has no attribute 'x' and no __dict__ for setting new attributes Most of the time t…
If you're using dataclasses it's less of an issue because dataclasses.asdict.
My problem isn't running out of memory; it's loading in a complex model where the fields are BaseModels and unions of BaseModels multiple levels deep. It doesn't load it all the way and leaves some of the deeper parts as dictionaries. I need like almost a parser to search the space of different loads. Anyone have any ideas for software that does that?
I know nothing about your context, but in what context would a single model need to support so many permutations of a data structure? Just because software can, doesn't mean it should.
Automatic, statically typed deserialization is worth the trouble in my opinion
So are there downsides to just always setting slots=True on all of my python data types?
You can't add extra attributes that weren't part of the original dataclass definition: >>> from dataclasses import dataclass >>> @dataclass ... class C: pass ... >>> C().x = 1 >>> @dataclass(slots=True) ... class D: pass ... >>> D().x = 1 Traceback (most recent call last): File " ", line 1, in D().x = 1 ^^^^^ AttributeError: 'D' object has no attribute 'x' and no __dict__ for setting new attributes Most of the time t…
Maybe using mmap would also save some memory, I'm not quite sure if this can be implemented in Python.
Or just dump pydantic and use msgspec instead: https://jcristharif.com/msgspec/
A great feature of pydantic are the validation hooks that let you intercept serialization/deserialization of specific fields and augment behavior. For example if you are querying a DB that returns a column as a JSON string, trivial with Pydantic to json parse the column are part of deser with an annotation. Pydantic is definitely slower and not a 'zero cost abstraction', but you do get a lot for it.