Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Kim – A Python serialization and marshaling framework

kim.readthedocs.io

Post reply on HN
Print view View original

31–40 of 61 posts

Re: Show HN: Kim – A Python serialization and marshaling framework

#31

Post by fnord123 »

mhneu wrote: post #17
sandGorgon wrote: post #14

We are really looking for serialization libraries that will work with pandas and scikit. This stuff is really all over the place - PMML, Arrow, Dill, pickle. Some stuff won't work with one or the other. I will actually pay for consistency versus performance. There are way too many primitive serialization libraries. Surprisingly none for the higher order ML, etc stuff. Give the kind of people behind Arrow, I would lov…

Python's data infrastructure has a huge problem: serialization and thus saving data results. A good serialization library should serialize: - classes/objects (best practice: objects for holding data) - pandas/numpy objects (must have: minimizing space) - namedtuples (currently: a mess, factory implementation) - dicts and lists of dicts (must have: space efficiency) Compare to Matlab: save(f, 'anyobject'); anyobject=l…

> Compare to Matlab: save(f, 'anyobject'); anyobject=load(f)

If you want matlab files in Python you can use `scipy.io.loadmat('file.mat')`. PyTables (built on hdf5) is a better solution since the hdf5 format is a lot more flexible than matlab's (ime). But Parquet is looking to be the best solution moving forward as it's gaining a lot of mindshare as the go-to flexible format for data and will be / is used in Arrow.

But really, Matlab is on par with pickles when it comes to serialisation. It's a trap solution.

Re: Show HN: Kim – A Python serialization and marshaling framework

#32

Post by amelius »

jackqu7 wrote: post #15
amelius wrote: post #12

Earlier quoted context omitted.

Roughly speaking, by cycles I mean a structure that refers to itself somehow. For example: A = {} B = {} A["ref"] = B B["ref"] = A So would it be possible to serialize A and B, and of course to deserialize them? Note that print A gives {'ref': {'ref': {...}}} which is of course not a suitable serialization, since you can't recover the original structure from it.

Yes, this is possible as long as the second level nested object has a role to stop infinite recursion from occurring. Cycles are not automatically detected. class BaseMapper(Mapper): __type__ = TestType score = Integer() nest = Nested('NestedMapper') __roles__ = {'nested': blacklist('nest')} class NestedMapper(Mapper): __type__ = TestType back = Nested('BaseMapper', role='nested') name = String() obj2 = TestType(name…

The output isn't clear to me:

    {'nest': {'back': {'score': 5}, 'name': 'test'}, 'score': 5}
How can this be mapped back unambiguously to a cyclic structure?

(I might be wrong but it seems to me that the act of serialization has simply expanded the cycle one level deep.)

Re: Show HN: Kim – A Python serialization and marshaling framework

#34

Post by joshuamorton »

mhneu wrote: post #17
sandGorgon wrote: post #14

We are really looking for serialization libraries that will work with pandas and scikit. This stuff is really all over the place - PMML, Arrow, Dill, pickle. Some stuff won't work with one or the other. I will actually pay for consistency versus performance. There are way too many primitive serialization libraries. Surprisingly none for the higher order ML, etc stuff. Give the kind of people behind Arrow, I would lov…

Python's data infrastructure has a huge problem: serialization and thus saving data results. A good serialization library should serialize: - classes/objects (best practice: objects for holding data) - pandas/numpy objects (must have: minimizing space) - namedtuples (currently: a mess, factory implementation) - dicts and lists of dicts (must have: space efficiency) Compare to Matlab: save(f, 'anyobject'); anyobject=l…

To expand on fnord, to my knowledge, pickle handles all of these things. Its still a bad solution, but it does everything you want.

    pickle.dump(f, anyobject)
    anyobject = pickle.load(f)

Re: Show HN: Kim – A Python serialization and marshaling framework

#35

Post by makmanalp »

sandGorgon wrote: post #14

We are really looking for serialization libraries that will work with pandas and scikit. This stuff is really all over the place - PMML, Arrow, Dill, pickle. Some stuff won't work with one or the other. I will actually pay for consistency versus performance. There are way too many primitive serialization libraries. Surprisingly none for the higher order ML, etc stuff. Give the kind of people behind Arrow, I would lov…

So stuff like this or marshmallow is more for cases when you have some database / ORM objects and you want to serialize them out to a json object, or you want to process form/POST data into a well-structured json or database object.

For your use case, it's more about large amounts of tabular data and efficient (binary / columnar / compressed) serialization and queryability. I'd say that the defacto standard for that is the HDF5 standard, which PyTables supports (http://www.pytables.org/). This is what pandas uses under the hood and I've been using this with hundreds of millions of rows with no problem.

Arrow is slightly more different - it's a specification for the in-memory layout of data that enables faster computation. This is more about what happens if you have data in memory and you want to use it with another tool - serializing / deserializing, munging formats is a waste of time if tools can standardize how they store dataframes in memory and can work on each other's tables. As far as I understand, Feather is not an implementation of arrow (that would be up to the processing tools like pandas), but supports a way of saving and loading that in-memory format to and from disk efficiently and in an interoperable way. (https://github.com/wesm/feather)

Also of note is parquet, which has similar goals to HDF and feather, but the continuum / dask people have been working on a wrapper for that called fastparquet (https://github.com/dask/fastparquet). In my experience it has a few hitches right now but works darn well, and gives me better performance than HDF. This is also one of the hadoop ecosystem defacto standards for storage formats, which again is good for interop.

Re: Show HN: Kim – A Python serialization and marshaling framework

#36

Post by mafro »

BuuQu9hu wrote: post #33

Sorry, I must be harsh. No. This fundamentally doesn't offer much advantage over a .toJSON() instance method and a .fromJSON() class method. Don't say "security-focused" if you can't handle cyclic object graphs.

Please elaborate on the reasons for your opinion :)

Re: Show HN: Kim – A Python serialization and marshaling framework

#37

Post by dtjon »

fnord123 wrote: post #30
sandGorgon wrote: post #14

We are really looking for serialization libraries that will work with pandas and scikit. This stuff is really all over the place - PMML, Arrow, Dill, pickle. Some stuff won't work with one or the other. I will actually pay for consistency versus performance. There are way too many primitive serialization libraries. Surprisingly none for the higher order ML, etc stuff. Give the kind of people behind Arrow, I would lov…

McKinney has been hard at work getting parquet and arrow support in pandas. http://wesmckinney.com/blog/outlook-for-2017/ >Give the kind of people behind Arrow, I would love wrapper that will use Arrow to do all of this...But doesn't matter at the end of the day. pyarrow; pyarrow.parquet (which uses parquet-cpp).

Wow this is great. I've been working around the jvm to integrate sklearn and some spark jobs that produce Parquet. This is a huge relief

Re: Show HN: Kim – A Python serialization and marshaling framework

#39

Post by sandGorgon »

fnord123 wrote: post #30
sandGorgon wrote: post #14

We are really looking for serialization libraries that will work with pandas and scikit. This stuff is really all over the place - PMML, Arrow, Dill, pickle. Some stuff won't work with one or the other. I will actually pay for consistency versus performance. There are way too many primitive serialization libraries. Surprisingly none for the higher order ML, etc stuff. Give the kind of people behind Arrow, I would lov…

McKinney has been hard at work getting parquet and arrow support in pandas. http://wesmckinney.com/blog/outlook-for-2017/ >Give the kind of people behind Arrow, I would love wrapper that will use Arrow to do all of this...But doesn't matter at the end of the day. pyarrow; pyarrow.parquet (which uses parquet-cpp).

Arrow doesn't do scikit - atleast last time I checked . Has it changed ?

Re: Show HN: Kim – A Python serialization and marshaling framework

#40

Post by dtjon »

sandGorgon wrote: post #39
fnord123 wrote: post #30

Earlier quoted context omitted.

McKinney has been hard at work getting parquet and arrow support in pandas. http://wesmckinney.com/blog/outlook-for-2017/ >Give the kind of people behind Arrow, I would love wrapper that will use Arrow to do all of this...But doesn't matter at the end of the day. pyarrow; pyarrow.parquet (which uses parquet-cpp).

Arrow doesn't do scikit - atleast last time I checked . Has it changed ?

pyarrow has methods to convert to pandas, which scikit supports

http://pyarrow.readthedocs.io/en/latest/pandas.html

Post reply on HN