I think that compression is a must for the serialization library. Protobuf uses almost twice less memory than Cap'n Proto. Using an external compression is not an option in some cases. E.g. consider building tcp-server that communicates with thousands of clients simultaneously. Each client connection will have its own LZ4 context that should be heap allocated. I believe it's about 16KB per connection + buffers. This results in large memory consumption and a lot of random memory access and TLB misses.
Cap’n Proto
171–180 of 194 posts
Re: Cap’n Proto
#172Earlier quoted context omitted.
Nitpick: I think you meant "&c". The t is superfluous.
Interesting; I've always written &ct, which wikipedia informs me is considered "archaic" Thanks for that note, it will save me one character of typing 3 years from now when I've finally broken the muscle memory of the old way :)
I'm still trying to break the muscle memory of typing &ct ...
Re: Cap’n Proto
#173Earlier quoted context omitted.
It lets you put data on the wire, in a structured format, right out of memory. Asking the question "how much faster is it" isn't even valid here, because it skips the usual serialization process. Cap'n Proto generates you some code that contains some data structures. You put data into these structures, and they will automatically be in the right shape to put directly on the wire. And then that data can be pulled righ…
It's infinitely faster if you have control over the data layout in your application - which most likely means your are developing your application in C/C++, or maybe Rust. In the case of JS you don't have, so accessing and writing the data is slow (since you would need [de]deserialization there to write it in a byte array). In fact any serializations in JS are slower than JSON, since this is natively implemented in t…
I have a similar question re the Python implementation
Is it still possible to realise benefits of the encoding when translating Python objects?
Re: Cap’n Proto
#174> The Cap’n Proto encoding is appropriate both as a data interchange format and an in-memory representation, so once your structure is built, you can simply write the bytes straight out to disk! Eh, I'd rather pay the cost of serialisation once and deserialisation once, and then access my data for as close to free as possible, rather than relying on a compiler to actually inline calls properly. > Integers use little-…
> > Integers use little-endian byte order because most CPUs are little-endian, and even big-endian CPUs usually have instructions for reading little-endian data. > sob There are a lot of things Intel has to account for, and frankly little-endian byte order isn't the worst of them, but it's pretty rotten. Writing 'EFCDAB8967452301' for 0x0123456789ABCDEF is perverse in the extreme. Why? Why? Little endian means that a…
Hah. According to Wikipedia: "The Datapoint 2200 used simple bit-serial logic with little-endian to facilitate carry propagation. When Intel developed the 8008 microprocessor for Datapoint, they used little-endian for compatibility. However, as Intel was unable to deliver the 8008 in time, Datapoint used a medium scale integration equivalent, but the little-endianness was retained in most Intel designs." Which I guess is a reason to use little-endian, though not one that's really relevant to anyone that's ever actually used an Intel CPU. (ARM, incidentally, was little-endian by default because the 6502 was.)
Re: Cap’n Proto
#175Hi all, Cap'n Proto author here. Thanks for the post. Just wanted to note that although Cap'n Proto hasn't had a blog post or official release in a while, development is active as part of the Sandstorm project ( https://sandstorm.io ). Cap'n Proto -- including the RPC system -- is used extensively in Sandstorm. Sandboxed Sandstorm apps in fact do all their communications with the outside world through a single Cap'n…
From reading the examples, it looks like one can only serialise to a file descriptor. What if i wan to serialise to a byte array? Am I perhaps missing something obvious?
Re: Cap’n Proto
#176Earlier quoted context omitted.
Also I might add that I very quickly became quite disenchanted with the lack of types in JSON; and I'm sure many of us have come to that same conclusion. Really hope that it can be fixed at some point, but not holding my breath on that one. I think it will take a major effort to reform that format although I am hopeful now that we at least have UInt8Array and friends that are starting to expose a broader set of machi…
I've actually seen people use types with JSON. JSON objects are clearly more than sufficient to represent the data you need, and you can enforce the schema in a library. It's just a much less efficient form of something like protobufs.
I think it depends on what data you need based on the problem space, but yes it does suffice for many use cases.
The thing is that it is least common denominator, and when you are dealing with high perf, cross language systems, it really isn't a good wire format or storage format.
It takes ages to parse, it's lossy, lacks commonly used types (or you have to annotate it with non standard attributes)... or worse guess the intention, and it's pretty verbose.
But again, that said it is a widely used standard and one that we have to live with. So there is that.
Re: Cap’n Proto
#177For some reason, the banner (infinitely faster?), name, and introductory FAQ-style responses made me think the whole thing is a joke - similar to Vanilla JS [1]. Anyways, it seems like a cool project, so I'll be sure to follow its development closely. [1]: http://vanilla-js.com/
If Cap'n Proto is a joke, JSON beat it to the punchline. What's an efficient binary representation? C Structs. What's an efficient text representation? Javascript objects. But casting arbitrary data to a struct is a horrible idea. And eval'ing anonymous javascript is a horrible idea. Back to the drawing board. Then N years later someone has the brilliant idea of just... not parsing these formats that idiotic way. And…
I'd love to read the full story there - how JSON got so much momentum behind it. Do you know of a good overview blog post?
Re: Cap’n Proto
#178Earlier quoted context omitted.
Like Flask.
It was a joke aimed at Bottle, right? And now it's probably the second most popular Python web framework. Armin knows his shit.
Yeah, it was an April fool's joke satirizing web microframeworks. He does, but I don't know if that was more because Flask is a better framework or because of better marketing. I don't think Bottle is really any worse than Flask.
Re: Cap’n Proto
#179Fractalide ( http://githib.com/fractalide/fractalide ) is an implementation of dataflow programming (specifically flow based programming). Component build hierarchies are coordinated via the Nix package manager. Capnproto contracts are weaved into each component just before build time. These contracts are the only way compenents talk to each other. Thanks Sandstorm.io for this great software.
Re: Cap’n Proto
#180Earlier quoted context omitted.
> and you don't care if you're implementing someone else's ideas and working for an advertising company FTFY
I'd be fascinated if someone who downvoted this would also be brave enough to explain why they're in denial that google is an advertising company. It may not be that your ideas are crazy and bad, but rather that your sane good ideas simply aren't appropriate in the context of advertising. Enough with the Stockholm syndrome. There are pleanty of perfectly great ideas that Google would never pay their employees to work…
Personally, I find it very difficult to motivate myself to apply for Google. Because of your reasons, and also becoming the 50,000th engineer. Will take almost ~2 yrs to work on anything interesting (Let's face it, most engineers aren't the world famous engineer in the Google Brain team).