41–50 of 61 posts
I'd like to congratulate the authors regarding the clever naming. I totally get the Eminem's reference. Disclaimer: Posting this comment because my colleague pointed out that I could get some points.
We are really looking for serialization libraries that will work with pandas and scikit. This stuff is really all over the place - PMML, Arrow, Dill, pickle. Some stuff won't work with one or the other. I will actually pay for consistency versus performance. There are way too many primitive serialization libraries. Surprisingly none for the higher order ML, etc stuff. Give the kind of people behind Arrow, I would lov…
Python's data infrastructure has a huge problem: serialization and thus saving data results. A good serialization library should serialize: - classes/objects (best practice: objects for holding data) - pandas/numpy objects (must have: minimizing space) - namedtuples (currently: a mess, factory implementation) - dicts and lists of dicts (must have: space efficiency) Compare to Matlab: save(f, 'anyobject'); anyobject=l…
Works pretty well - I know of large financial firms that are using this in production to load large trained models of size hundreds of GB
Earlier quoted context omitted.
Roughly speaking, by cycles I mean a structure that refers to itself somehow. For example: A = {} B = {} A["ref"] = B B["ref"] = A So would it be possible to serialize A and B, and of course to deserialize them? Note that print A gives {'ref': {'ref': {...}}} which is of course not a suitable serialization, since you can't recover the original structure from it.
Yes, this is possible as long as the second level nested object has a role to stop infinite recursion from occurring. Cycles are not automatically detected. class BaseMapper(Mapper): __type__ = TestType score = Integer() nest = Nested('NestedMapper') __roles__ = {'nested': blacklist('nest')} class NestedMapper(Mapper): __type__ = TestType back = Nested('BaseMapper', role='nested') name = String() obj2 = TestType(name…
One way is to to store a table of objects (as identified by id()) encountered during serialization, indexed by the order you encounter them. If you encounter an object you have already serialized, serialize an index into that table. On deserialization, construct the same kind of table, and deserialize an index with a reference to the same object.
See e.g. AMF for an example format that does this: https://en.wikipedia.org/wiki/Action_Message_Format
Earlier quoted context omitted.
(I'm Jack, another developer at OSL.) We started writing Kim around the same time as the Marshmallow project began as we found it wasn't suitable for our needs at that time, though it has come a long way since then. They are very similar projects and have similar functionality, but Kim has a focus on making it relatively simple to do unusual or 'advanced' things. For example, Kim supports polymorphism out of the box,…
I've been saddened by Marshmallow on many occasions (I have gripes with the particular way defaults/validation play together. This is true for WTForms too). I'm excited to try out Kim. I've been very close to just writing my own serialization lib on many occasions. It looks like your pipelines might bring a bit of sanity to it. :) It looks like you support a few sorts of validation, but the docs aren't super clear as…
It's great you asked this question as we noticed part of the documentation was actually broken. here's a link to a pretty basic example of adding extra validation "pipes" to a pipeline
http://kim.readthedocs.io/en/latest/user/advanced.html#custo...
We'd be more than happy to discuss how to solve more complex requirements if there's something specific you had in mind though.
Thanks for the message!
Earlier quoted context omitted.
Python's data infrastructure has a huge problem: serialization and thus saving data results. A good serialization library should serialize: - classes/objects (best practice: objects for holding data) - pandas/numpy objects (must have: minimizing space) - namedtuples (currently: a mess, factory implementation) - dicts and lists of dicts (must have: space efficiency) Compare to Matlab: save(f, 'anyobject'); anyobject=l…
To expand on fnord, to my knowledge, pickle handles all of these things. Its still a bad solution, but it does everything you want. pickle.dump(f, anyobject) anyobject = pickle.load(f)
Earlier quoted context omitted.
To expand on fnord, to my knowledge, pickle handles all of these things. Its still a bad solution, but it does everything you want. pickle.dump(f, anyobject) anyobject = pickle.load(f)
Pickle had size constraints that make it unsuitable in certain ML applications.
Silly question, what happens with Unicode?
Earlier quoted context omitted.
I've been saddened by Marshmallow on many occasions (I have gripes with the particular way defaults/validation play together. This is true for WTForms too). I'm excited to try out Kim. I've been very close to just writing my own serialization lib on many occasions. It looks like your pipelines might bring a bit of sanity to it. :) It looks like you support a few sorts of validation, but the docs aren't super clear as…
Hey that's really great to hear. (that you're keen to use Kim) WTF-Forms and Marshmallow both solve problems and they do it well but it seems like us you wanted something that offered just a bit more flexibility. That's totally the idea behind pipelines in Kim. They are like tiny little computer programmes and are really capable of anything (providing it's possible in Python of course :D ) It's great you asked this q…
Earlier quoted context omitted.
Arrow doesn't do scikit - atleast last time I checked . Has it changed ?
pyarrow has methods to convert to pandas, which scikit supports http://pyarrow.readthedocs.io/en/latest/pandas.html
Take a look at this to understand what I mean . http://stackoverflow.com/questions/32757656/what-are-the-pit...
Library Many Objects One Object
--------------------- -------------- ------------
Custom 0.0187769 0.00682402
Strainer 0.0603201 0.0337129
serpy 0.073787 0.038656
Lollipop 0.47821 0.231566
Marshmallow 1.14844 0.598486
Django REST Framework 1.94096 1.3277
kim 2.28477 1.15237
source: https://voidfiles.github.io/python-serialization-benchmark/