21–30 of 61 posts
Earlier quoted context omitted.
Thanks for expanding on that mhneu. So our primary focus with Kim has certainly been around serializing/marshaling JSON though we've used it for plenty of other uses cases. It's great to get a view of other problems people are experiencing. Now we've finished wrapping up 1.0.0 we're going to be spending some time on the roadmap of new features. I personally feel variation in use cases from our own is only going to he…
defo?
In the case of serialization libraries,unless you are validating as part of your (de)serialization, I'd recommend avoiding schema-driven serialization libraries. These Kim-like libraries, such as Marshmallow, introduce quite a bit of overhead. If validation isn't required and performance matters, I recommend choosing a lighter-weight serialization/marshalling alternative, such as that provided by asphalt-serialization: https://github.com/asphalt-framework/asphalt-serialization
Asphalt-serialization supports cbor, msgpack, json, ... and is easy to wire up
This recommendation is based on my own experience using Marshmallow for Yosai, analyzing its performance and then refactoring to a ported version of asphalt-serialization.
Cool project! In the case of serialization libraries,unless you are validating as part of your (de)serialization, I'd recommend avoiding schema-driven serialization libraries. These Kim-like libraries, such as Marshmallow, introduce quite a bit of overhead. If validation isn't required and performance matters, I recommend choosing a lighter-weight serialization/marshalling alternative, such as that provided by asphal…
That's a great point and an important distinction to make. As I mentioned in some of the other comments, we have certainly been focussed on features over performance so far but we are actively working on dramatically improving the performance of Kim.
I guess it's almost important to pick the right tool for the job. Thanks for sharing the link to asphalt too. I'd not see that before.
Cool project! In the case of serialization libraries,unless you are validating as part of your (de)serialization, I'd recommend avoiding schema-driven serialization libraries. These Kim-like libraries, such as Marshmallow, introduce quite a bit of overhead. If validation isn't required and performance matters, I recommend choosing a lighter-weight serialization/marshalling alternative, such as that provided by asphal…
Hey Dowwie! That's a great point and an important distinction to make. As I mentioned in some of the other comments, we have certainly been focussed on features over performance so far but we are actively working on dramatically improving the performance of Kim. I guess it's almost important to pick the right tool for the job. Thanks for sharing the link to asphalt too. I'd not see that before.
Earlier quoted context omitted.
Hey Dowwie! That's a great point and an important distinction to make. As I mentioned in some of the other comments, we have certainly been focussed on features over performance so far but we are actively working on dramatically improving the performance of Kim. I guess it's almost important to pick the right tool for the job. Thanks for sharing the link to asphalt too. I'd not see that before.
Keep up the good work, Mikey. :) See you at PyCon, maybe?
Nice, but I recommend closing issues https://github.com/mikeywaites/kim/issues which have fixes (some of them show 'merge'). It's one thing I as a user look at choosing whether to adopt a project or not.
Nice, but I recommend closing issues https://github.com/mikeywaites/kim/issues which have fixes (some of them show 'merge'). It's one thing I as a user look at choosing whether to adopt a project or not.
absolutely. Im a bit annoyed at myself that I hadn't got round to that yet but thanks for raising it.
It does look like marshmalllow[1]. How does relate Kim with it? [1]: https://github.com/marshmallow-code/marshmallow/
(I'm Jack, another developer at OSL.) We started writing Kim around the same time as the Marshmallow project began as we found it wasn't suitable for our needs at that time, though it has come a long way since then. They are very similar projects and have similar functionality, but Kim has a focus on making it relatively simple to do unusual or 'advanced' things. For example, Kim supports polymorphism out of the box,…
I'm excited to try out Kim. I've been very close to just writing my own serialization lib on many occasions.
It looks like your pipelines might bring a bit of sanity to it. :)
It looks like you support a few sorts of validation, but the docs aren't super clear as to what the expected validation strategy is. Could you elaborate on what that looks like?
My typical strategy I'd like to do is to just a list of functions that take the input and return a boolean as far as validation goes.
We are really looking for serialization libraries that will work with pandas and scikit. This stuff is really all over the place - PMML, Arrow, Dill, pickle. Some stuff won't work with one or the other. I will actually pay for consistency versus performance. There are way too many primitive serialization libraries. Surprisingly none for the higher order ML, etc stuff. Give the kind of people behind Arrow, I would lov…
http://wesmckinney.com/blog/outlook-for-2017/
>Give the kind of people behind Arrow, I would love wrapper that will use Arrow to do all of this...But doesn't matter at the end of the day.
pyarrow; pyarrow.parquet (which uses parquet-cpp).