Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cap’n Proto

capnproto.org

Post reply on HN
Print view View original

141–150 of 194 posts

Re: Cap’n Proto

#141

Post by kentonv »

jcoffland wrote: post #79
kentonv wrote: post #11

Hi all, Cap'n Proto author here. Thanks for the post. Just wanted to note that although Cap'n Proto hasn't had a blog post or official release in a while, development is active as part of the Sandstorm project ( https://sandstorm.io ). Cap'n Proto -- including the RPC system -- is used extensively in Sandstorm. Sandboxed Sandstorm apps in fact do all their communications with the outside world through a single Cap'n…

Recently I've started compiling more of my code for Windows using Mingw (or clang). Performance is very good and you can build in Linux for Windows inside a Docker container. You can even build Windows installers using NSIS. I've also tried to do this for OSX but with less success.

Cap'n Proto supports MinGW as a target platform. (It has also supported Cygwin since very early on.) However, Cap'n Proto is a library, so it needs to support whatever compiler the users of that library are using, and for the vast majority of Windows developers that's MSVC.

Re: Cap’n Proto

#142

Post by kentonv »

heavenlyhash wrote: post #108
kentonv wrote: post #47

Earlier quoted context omitted.

I haven't looked at Ion before, but it appears to be similar to BSON or Msgpack in that it's a binary format that encodes field names as textual identifiers, to be "self-describing" and avoid the need for an external schema. I generally like schemas, because I like static typing. Of course, static types vs. dynamic types are another ancient flamewar and I'm unlikely to cover any new ground by stating arguments here.…

I like schemas for validation and strong types. I also like self-describing systems when all goes to hell and I'm debugging either my own protocols or someone else's. I have this crazy idea that there's a middle ground: self-describing and schemas? We can kind of glue this together (there's lots of json schemas floating around out there now, it seems), but it would be awfully interesting to see these well-supported a…

When I'm forced to use JSON, I often do it by defining a Cap'n Proto schema and using capnp::JsonCodec to parse / serialize that schema as JSON.

https://github.com/sandstorm-io/capnproto/blob/master/c++/sr...

(There are also similar libraries for Protobuf.)

Re: Cap’n Proto

#143

Post by kentonv »

DenisM wrote: post #131
kentonv wrote: post #11

Hi all, Cap'n Proto author here. Thanks for the post. Just wanted to note that although Cap'n Proto hasn't had a blog post or official release in a while, development is active as part of the Sandstorm project ( https://sandstorm.io ). Cap'n Proto -- including the RPC system -- is used extensively in Sandstorm. Sandboxed Sandstorm apps in fact do all their communications with the outside world through a single Cap'n…

How about first-class map / dictionary support? That is, const or log time retrieval of key-value pairs. Right now I can only see a way to create lists, so access will be in linear time.

The "lists" are actually arrays, so access by index is constant-time. You can build a hashtable on top of that. I'd like to add built-in support for maps at some point, but it's one of a very large number of wishlist items...

Re: Cap’n Proto

#144

Post by kentonv »

dekhn wrote: post #117
kentonv wrote: post #29

Earlier quoted context omitted.

Proto3 is not a large change. In fact it shares most of its code with proto2, as I understand it. The underlying encoding is the same. Proto3 removes some features from proto2 which were deemed overcomplicated relative to their value (unknown field retention, non-zero default values, extensions, required fields) and adds some features that people have wanted for a long time (maps). But all of these features are thing…

proto2 has maps, but like proto3 they aren't maps, they're an randomly ordered sequence of key value pairs ugh .

Presumably the lookup table is built at parse time?

(Proto2 definitely didn't have any built-in notion of maps when I was working on it. I thought maps were added as a proto3 feature...)

Re: Cap’n Proto

#145

Post by kentonv »

_0w8t wrote: post #95
kentonv wrote: post #11

Hi all, Cap'n Proto author here. Thanks for the post. Just wanted to note that although Cap'n Proto hasn't had a blog post or official release in a while, development is active as part of the Sandstorm project ( https://sandstorm.io ). Cap'n Proto -- including the RPC system -- is used extensively in Sandstorm. Sandboxed Sandstorm apps in fact do all their communications with the outside world through a single Cap'n…

I am curious, what is your take on Thrift? Is it really an improvement on protocol buffers?

Thrift to me is a Protobuf clone. Very similar design. Early on it was not very well optimized compared to protobuf, but I imagine they've fixed that by now. The big advantage they had was that they included an RPC system in their first release -- although it was a FIFO RPC system which struck me as an odd design choice (I think they may have fixed this more recently?). But now GRPC exists, so I don't think there's much reason to choose Thrift over Protobuf/GRPC.

However, I am obviously very biased. :)

Re: Cap’n Proto

#146

Post by kentonv »

Matthias247 wrote: post #140
striking wrote: post #109

Earlier quoted context omitted.

It lets you put data on the wire, in a structured format, right out of memory. Asking the question "how much faster is it" isn't even valid here, because it skips the usual serialization process. Cap'n Proto generates you some code that contains some data structures. You put data into these structures, and they will automatically be in the right shape to put directly on the wire. And then that data can be pulled righ…

It's infinitely faster if you have control over the data layout in your application - which most likely means your are developing your application in C/C++, or maybe Rust. In the case of JS you don't have, so accessing and writing the data is slow (since you would need [de]deserialization there to write it in a byte array). In fact any serializations in JS are slower than JSON, since this is natively implemented in t…

No no, this is a common misunderstanding about Cap'n Proto. It does not take your regular in-memory data structures from your regular programming language (even C++) and put them on the wire. What it does is defines its own specific data layout which happens to be appropriate both for in-memory random-access use and for transmission.

Cap'n Proto generates classes which wrap a byte buffer and give you accessor methods that read the fields straight out of the buffer.

That actually works equally well in C++ and Javascript.

Re: Cap’n Proto

#147

Post by DenisM »

kentonv wrote: post #143
DenisM wrote: post #131

Earlier quoted context omitted.

How about first-class map / dictionary support? That is, const or log time retrieval of key-value pairs. Right now I can only see a way to create lists, so access will be in linear time.

The "lists" are actually arrays, so access by index is constant-time. You can build a hashtable on top of that. I'd like to add built-in support for maps at some point, but it's one of a very large number of wishlist items...

The simplest implementation would be a sorted list with binary search for access. Segments create problems though.

I thought of the hashtable, but that means I have to keep the hash-table in RAM. By contrast, the entire capnproto structure itself could be memory mapped, thus not having any RAM constraints.

I'm thinking larger datasets here, large enough where overhead of e.g. JSON becomes a problem with RAM (on mobile devices), but not yet large enough to have to use SQLite.

I might be overthinking it, and could move to SQLite straight away. Just trying to keep my stuff as simple as possible. Conceptually simple, that is. Capnproto is conceptually simple: a tree dumped to disk, allowing on-demand memory mapped access to requested parts.

Re: Cap’n Proto

#148

Post by kentonv »

DenisM wrote: post #147
kentonv wrote: post #143

Earlier quoted context omitted.

The "lists" are actually arrays, so access by index is constant-time. You can build a hashtable on top of that. I'd like to add built-in support for maps at some point, but it's one of a very large number of wishlist items...

The simplest implementation would be a sorted list with binary search for access. Segments create problems though. I thought of the hashtable, but that means I have to keep the hash-table in RAM. By contrast, the entire capnproto structure itself could be memory mapped, thus not having any RAM constraints. I'm thinking larger datasets here, large enough where overhead of e.g. JSON becomes a problem with RAM (on mobil…

The capnp structure itself could be a hashtable, and doesn't need to be (entirely) loaded into RAM.

I'm saying you create a capnp list, and then you store elements into the list at position according to their hash -- i.e. how you'd build a hashtable, but the capnp list itself is the backing array.

You would have to do this at write time, of course, and make sure the hashing is consistent between runs.

Re: Cap’n Proto

#150

Post by Matthias247 »

kentonv wrote: post #146
Matthias247 wrote: post #140

Earlier quoted context omitted.

It's infinitely faster if you have control over the data layout in your application - which most likely means your are developing your application in C/C++, or maybe Rust. In the case of JS you don't have, so accessing and writing the data is slow (since you would need [de]deserialization there to write it in a byte array). In fact any serializations in JS are slower than JSON, since this is natively implemented in t…

No no, this is a common misunderstanding about Cap'n Proto. It does not take your regular in-memory data structures from your regular programming language (even C++) and put them on the wire. What it does is defines its own specific data layout which happens to be appropriate both for in-memory random-access use and for transmission. Cap'n Proto generates classes which wrap a byte buffer and give you accessor methods…

Ok, but these accessor methods will still have a very different performance. In C++ copying a UTF8 string from one byte array into an std::string is super fast. Whereas in JS it's really slow, since you need to read from an ArrayBuffer, convert code points to UTF16 in JS and then store these in a string (which is not efficient, since the strings are immutable). In node you could at least speed that up through some native extension, but even then it would most likely be slower than JSON. In the browser it would be in any case. But that's really a speciality of JS.

In general I then think the difference (for non-C++) between your method and others (protobuf, thrift, ...) is that yours would require the cost of a field serialization in the moment the field is accessed. In others all fields are deserialized at once. But in the end it should have the same cost if I need all fields, e.g. in order to convert the data into a plain Java/Javascript/C#/... object, or am I missing something there? For C++ is absolutely believe that you can have a byte-array backed proxy-object with accessor methods that have the same properties as accessing native C++ structures.

Post reply on HN