Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Kim – A Python serialization and marshaling framework

kim.readthedocs.io

Post reply on HN
Print view View original

51–60 of 61 posts

Re: Show HN: Kim – A Python serialization and marshaling framework

#51

Post by tinnet »

pekk wrote: post #19
tinnet wrote: post #4

Earlier quoted context omitted.

Obviously no OS developer owes anybody an explanation, but man would I appreciate if more projects had a "why you should use this over related projects" (like e.g. pendulum does https://github.com/sdispater/pendulum/blob/master/README.rst... )

I know the pain of searching for software to meet your requirements. But unless you have a friend you can really trust to provide informed recommendations, nobody can take this pain away for you. If all require projects to say negative things about other people's projects while talking up their own, a lot of projects are going to distort the facts. In the end, if we don't have the ability to evaluate the software our…

I think you're right, when actually using it for production software it's probably wise to not be a trailblazer :)

For me this wish for a comparison (that I'd love to be objective and in god spirit of course - naive?) is probably coming more from "shopping around" between projects. Or just when seeing a new thing on HN and wondering if I should investigate adding this particular thing to my toolbox.

Re: Show HN: Kim – A Python serialization and marshaling framework

#52

Post by [deleted] »

pekk wrote: post #19
tinnet wrote: post #4

Earlier quoted context omitted.

Obviously no OS developer owes anybody an explanation, but man would I appreciate if more projects had a "why you should use this over related projects" (like e.g. pendulum does https://github.com/sdispater/pendulum/blob/master/README.rst... )

I know the pain of searching for software to meet your requirements. But unless you have a friend you can really trust to provide informed recommendations, nobody can take this pain away for you. If all require projects to say negative things about other people's projects while talking up their own, a lot of projects are going to distort the facts. In the end, if we don't have the ability to evaluate the software our…

[deleted]

Re: Show HN: Kim – A Python serialization and marshaling framework

#54

Post by vosper »

sidlls wrote: post #45
joshuamorton wrote: post #34

Earlier quoted context omitted.

To expand on fnord, to my knowledge, pickle handles all of these things. Its still a bad solution, but it does everything you want. pickle.dump(f, anyobject) anyobject = pickle.load(f)

Pickle had size constraints that make it unsuitable in certain ML applications.

Does using protocol version 4 help with this?

Re: Show HN: Kim – A Python serialization and marshaling framework

#55

Post by RussianCow »

voidfiles wrote: post #50

I added Kim to my ongoing set of python serialization framework benchmarks here is how it ranks. Library Many Objects One Object --------------------- -------------- ------------ Custom 0.0187769 0.00682402 Strainer 0.0603201 0.0337129 serpy 0.073787 0.038656 Lollipop 0.47821 0.231566 Marshmallow 1.14844 0.598486 Django REST Framework 1.94096 1.3277 kim 2.28477 1.15237 Comments on how to improve the benchmark are app…

Just a minor note: It seems you don't mention anywhere what those numbers actually mean. I'm assuming they are seconds, but I can't know for certain, which makes it really unclear if Kim is the fastest or the slowest.

Re: Show HN: Kim – A Python serialization and marshaling framework

#56

Post by auxym »

fnord123 wrote: post #31
mhneu wrote: post #17

Earlier quoted context omitted.

Python's data infrastructure has a huge problem: serialization and thus saving data results. A good serialization library should serialize: - classes/objects (best practice: objects for holding data) - pandas/numpy objects (must have: minimizing space) - namedtuples (currently: a mess, factory implementation) - dicts and lists of dicts (must have: space efficiency) Compare to Matlab: save(f, 'anyobject'); anyobject=l…

> Compare to Matlab: save(f, 'anyobject'); anyobject=load(f) If you want matlab files in Python you can use `scipy.io.loadmat('file.mat')`. PyTables (built on hdf5) is a better solution since the hdf5 format is a lot more flexible than matlab's (ime). But Parquet is looking to be the best solution moving forward as it's gaining a lot of mindshare as the go-to flexible format for data and will be / is used in Arrow. B…

Actually, since Matlab v7.3, .mat files are actually hdf5 files.

Re: Show HN: Kim – A Python serialization and marshaling framework

#57

Post by digisth »

makmanalp wrote: post #35
sandGorgon wrote: post #14

We are really looking for serialization libraries that will work with pandas and scikit. This stuff is really all over the place - PMML, Arrow, Dill, pickle. Some stuff won't work with one or the other. I will actually pay for consistency versus performance. There are way too many primitive serialization libraries. Surprisingly none for the higher order ML, etc stuff. Give the kind of people behind Arrow, I would lov…

So stuff like this or marshmallow is more for cases when you have some database / ORM objects and you want to serialize them out to a json object, or you want to process form/POST data into a well-structured json or database object. For your use case, it's more about large amounts of tabular data and efficient (binary / columnar / compressed) serialization and queryability. I'd say that the defacto standard for that…

Do you know of a source that compares these different libraries in terms of capabilities, focus/use cases, size limits, performance, format support, etc.?

Googling turned up very little for me.

TIA

Edit: libraries mentioned in thread:

PMML, Arrow, Dill, marshmallow, pytables, parquet/fastparquet (and pickle, obviously)

Re: Show HN: Kim – A Python serialization and marshaling framework

#58

Post by makmanalp »

digisth wrote: post #57
makmanalp wrote: post #35

Earlier quoted context omitted.

So stuff like this or marshmallow is more for cases when you have some database / ORM objects and you want to serialize them out to a json object, or you want to process form/POST data into a well-structured json or database object. For your use case, it's more about large amounts of tabular data and efficient (binary / columnar / compressed) serialization and queryability. I'd say that the defacto standard for that…

Do you know of a source that compares these different libraries in terms of capabilities, focus/use cases, size limits, performance, format support, etc.? Googling turned up very little for me. TIA Edit: libraries mentioned in thread: PMML, Arrow, Dill, marshmallow, pytables, parquet/fastparquet (and pickle, obviously)

No, I don't, but some of these are apples and oranges, that was part of my point. You're conflating many different types of things.

Specifically, the ones I talked about are for storing large tabular datasets on disk. Stuff that lays out data on disk so that it's easy and efficient to query only a part of the dataset, e.g. only certain columns or only certain rows that match a predicate or within a range of indexes. These can store hundreds of gb, no problem. They often have some sort of compression, like LZ, snappy or blosc that has relatively low CPU overhead while giving decent compression. I tried to separate the file formats (which are readable from other languages) from the python libraries that write them. For this, I'd default to pytables / HDF5, barring some specific use case where you'd already know what other one you need.

Dill / pickle are for serializing generic python objects. I wouldn't really use them to store anything big, but it's very convenient for complicated data structures, like hierarchies of objects and classes. E.g. to save the current running state of your program. You don't have to think about storage formats and layouts and serialization routines, if you have a list of python objects you can pickle it. Pickle is built in, while dill is an external library that nicely handles a bunch more edge cases.

PMML seems like an XML based format specifically for trained machine learning models. Don't really know much about this.

Re: Show HN: Kim – A Python serialization and marshaling framework

#59

Post by makmanalp »

voidfiles wrote: post #50

I added Kim to my ongoing set of python serialization framework benchmarks here is how it ranks. Library Many Objects One Object --------------------- -------------- ------------ Custom 0.0187769 0.00682402 Strainer 0.0603201 0.0337129 serpy 0.073787 0.038656 Lollipop 0.47821 0.231566 Marshmallow 1.14844 0.598486 Django REST Framework 1.94096 1.3277 kim 2.28477 1.15237 Comments on how to improve the benchmark are app…

This is brilliant, exactly what I was looking for. I did a profile recently on some API calls and found that 40-50% was being spent on serialization with marshmallow, which I'm looking to drop.

I'll be doing this stuff for myself, but would you be curious in having:

a) Support for lima: https://lima.readthedocs.io/en/latest/

b) more benchmark cases (serializing a larger list of objects)

Re: Show HN: Kim – A Python serialization and marshaling framework

#60

Post by mikeywaites »

voidfiles wrote: post #50

I added Kim to my ongoing set of python serialization framework benchmarks here is how it ranks. Library Many Objects One Object --------------------- -------------- ------------ Custom 0.0187769 0.00682402 Strainer 0.0603201 0.0337129 serpy 0.073787 0.038656 Lollipop 0.47821 0.231566 Marshmallow 1.14844 0.598486 Django REST Framework 1.94096 1.3277 kim 2.28477 1.15237 Comments on how to improve the benchmark are app…

Thanks so much for this Voidfiles. We were under no illusions that we weren't the most performant library out there (yet)

This is a great start for us understanding where we need to get to! We've got some work to do :)

Post reply on HN