Can it serialize cycles?
Hey Amelius, thanks for the message. Gonna be honest, I'm not sure what you mean by cycles. Can you elaborate a bit?
a = {}
b = {}
a["b"] = b
b["a"] = a
a == deserialize(serialize(a))Show HN: Kim – A Python serialization and marshaling framework
11–20 of 61 posts
Re: Show HN: Kim – A Python serialization and marshaling framework
#12Can it serialize cycles?
Hey Amelius, thanks for the message. Gonna be honest, I'm not sure what you mean by cycles. Can you elaborate a bit?
Roughly speaking, by cycles I mean a structure that refers to itself somehow. For example:
A = {}
B = {}
A["ref"] = B
B["ref"] = A
Note that
print A
{'ref': {'ref': {...}}}
Re: Show HN: Kim – A Python serialization and marshaling framework
#13It does look like marshmalllow[1]. How does relate Kim with it? [1]: https://github.com/marshmallow-code/marshmallow/
I think marshmallow primary use case is to unserialize to nested dicts/lists while kim outputs full classes. Did I understood that right ?
Re: Show HN: Kim – A Python serialization and marshaling framework
#14We are really looking for serialization libraries that will work with pandas and scikit.
This stuff is really all over the place - PMML, Arrow, Dill, pickle.
Some stuff won't work with one or the other. I will actually pay for consistency versus performance.
There are way too many primitive serialization libraries. Surprisingly none for the higher order ML, etc stuff.
Give the kind of people behind Arrow, I would love wrapper that will use Arrow to do all of this...But doesn't matter at the end of the day.
Re: Show HN: Kim – A Python serialization and marshaling framework
#15Earlier quoted context omitted.
Hey Amelius, thanks for the message. Gonna be honest, I'm not sure what you mean by cycles. Can you elaborate a bit?
Roughly speaking, by cycles I mean a structure that refers to itself somehow. For example: A = {} B = {} A["ref"] = B B["ref"] = A So would it be possible to serialize A and B, and of course to deserialize them? Note that print A gives {'ref': {'ref': {...}}} which is of course not a suitable serialization, since you can't recover the original structure from it.
Yes, this is possible as long as the second level nested object has a role to stop infinite recursion from occurring. Cycles are not automatically detected.
class BaseMapper(Mapper):
__type__ = TestType
score = Integer()
nest = Nested('NestedMapper')
__roles__ = {'nested': blacklist('nest')}
class NestedMapper(Mapper):
__type__ = TestType
back = Nested('BaseMapper', role='nested')
name = String()
obj2 = TestType(name='test')
obj = TestType(score=5, nest=obj2)
obj2.back = obj
>> BaseMapper(obj=obj).serialize()
{'nest': {'back': {'score': 5}, 'name': 'test'}, 'score': 5}Re: Show HN: Kim – A Python serialization and marshaling framework
#16We are really looking for serialization libraries that will work with pandas and scikit. This stuff is really all over the place - PMML, Arrow, Dill, pickle. Some stuff won't work with one or the other. I will actually pay for consistency versus performance. There are way too many primitive serialization libraries. Surprisingly none for the higher order ML, etc stuff. Give the kind of people behind Arrow, I would lov…
One of the things we felt very strongly about when developing Kim was that Simple things should be simple. Complex things should be possible. To that end the Pipeline system behind the Field objects really does allow anything to be achieved. Wether thats producing values from composite fields or handling unique or non standard data types.
it would be great if you can share some ways that you specifically need serialization to work for something like pandas, or better yet, some ways existing solutions don’t work with pandas. We’ve had some pretty unique requirements ourselves and have not found any blockers yet.
Thanks for the message.
Re: Show HN: Kim – A Python serialization and marshaling framework
#17We are really looking for serialization libraries that will work with pandas and scikit. This stuff is really all over the place - PMML, Arrow, Dill, pickle. Some stuff won't work with one or the other. I will actually pay for consistency versus performance. There are way too many primitive serialization libraries. Surprisingly none for the higher order ML, etc stuff. Give the kind of people behind Arrow, I would lov…
Python's data infrastructure has a huge problem: serialization and thus saving data results.
A good serialization library should serialize:
- classes/objects (best practice: objects for holding data)
- pandas/numpy objects (must have: minimizing space)
- namedtuples (currently: a mess, factory implementation)
- dicts and lists of dicts (must have: space efficiency)
Python is terrible at this and it limits use in real data analysis environments and limits competition with matlab.
Re: Show HN: Kim – A Python serialization and marshaling framework
#18We are really looking for serialization libraries that will work with pandas and scikit. This stuff is really all over the place - PMML, Arrow, Dill, pickle. Some stuff won't work with one or the other. I will actually pay for consistency versus performance. There are way too many primitive serialization libraries. Surprisingly none for the higher order ML, etc stuff. Give the kind of people behind Arrow, I would lov…
Python's data infrastructure has a huge problem: serialization and thus saving data results. A good serialization library should serialize: - classes/objects (best practice: objects for holding data) - pandas/numpy objects (must have: minimizing space) - namedtuples (currently: a mess, factory implementation) - dicts and lists of dicts (must have: space efficiency) Compare to Matlab: save(f, 'anyobject'); anyobject=l…
Thanks for expanding on that mhneu. So our primary focus with Kim has certainly been around serializing/marshaling JSON though we've used it for plenty of other uses cases.
It's great to get a view of other problems people are experiencing.
Now we've finished wrapping up 1.0.0 we're going to be spending some time on the roadmap of new features. I personally feel variation in use cases from our own is only going to help make Kim better so we'll defo look into this problem some more in the near future. Right now though i couldn't say for sure what Kim would have to offer when working with Pandas etc as we've simply never tried.
Re: Show HN: Kim – A Python serialization and marshaling framework
#19It does look like marshmalllow[1]. How does relate Kim with it? [1]: https://github.com/marshmallow-code/marshmallow/
Obviously no OS developer owes anybody an explanation, but man would I appreciate if more projects had a "why you should use this over related projects" (like e.g. pendulum does https://github.com/sdispater/pendulum/blob/master/README.rst... )
I know the pain of searching for software to meet your requirements. But unless you have a friend you can really trust to provide informed recommendations, nobody can take this pain away for you.
If all require projects to say negative things about other people's projects while talking up their own, a lot of projects are going to distort the facts. In the end, if we don't have the ability to evaluate the software ourselves, then all we are measuring is who can shout the loudest and who is the most aggressive against other projects. Quiet projects will still be good, but now those would be overlooked even more because they aren't shouting. With this requirement you are making your life easier but you are making life harder on open source developers by forcing them to deal with unnecessary inter-project drama and to divert lots of effort into marketing that could have been put into code. That might make sense in proprietary products, but in open source this kind of demand just hurts the ecosystem.
If the pain of choosing is too much then choose something that is standardized, or the most popular thing, or what your trusted friend recommends. People will seek out the very specific projects they need. If you don't even know why you are using something, it isn't the responsibility of someone else to tell you why you are using it!
Re: Show HN: Kim – A Python serialization and marshaling framework
#20Earlier quoted context omitted.
Python's data infrastructure has a huge problem: serialization and thus saving data results. A good serialization library should serialize: - classes/objects (best practice: objects for holding data) - pandas/numpy objects (must have: minimizing space) - namedtuples (currently: a mess, factory implementation) - dicts and lists of dicts (must have: space efficiency) Compare to Matlab: save(f, 'anyobject'); anyobject=l…
Thanks for expanding on that mhneu. So our primary focus with Kim has certainly been around serializing/marshaling JSON though we've used it for plenty of other uses cases. It's great to get a view of other problems people are experiencing. Now we've finished wrapping up 1.0.0 we're going to be spending some time on the roadmap of new features. I personally feel variation in use cases from our own is only going to he…
defo?