Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cap’n Proto

capnproto.org

Post reply on HN
Print view View original

151–160 of 194 posts

Re: Cap’n Proto

#151

Post by anongoogler2 »

kentonv wrote: post #11

Hi all, Cap'n Proto author here. Thanks for the post. Just wanted to note that although Cap'n Proto hasn't had a blog post or official release in a while, development is active as part of the Sandstorm project ( https://sandstorm.io ). Cap'n Proto -- including the RPC system -- is used extensively in Sandstorm. Sandboxed Sandstorm apps in fact do all their communications with the outside world through a single Cap'n…

I could imagine implementing something like this as an alternate wire format and client library for protocol buffers. Can you outline the main reason you chose not to go this route? In particular, what aspects of the proto descriptor language don't fit Cap'n Proto? Or what aspects did you feel needed to be changed for some other reason?

Re: Cap’n Proto

#152

Post by DenisM »

kentonv wrote: post #148
DenisM wrote: post #147

Earlier quoted context omitted.

The simplest implementation would be a sorted list with binary search for access. Segments create problems though. I thought of the hashtable, but that means I have to keep the hash-table in RAM. By contrast, the entire capnproto structure itself could be memory mapped, thus not having any RAM constraints. I'm thinking larger datasets here, large enough where overhead of e.g. JSON becomes a problem with RAM (on mobil…

The capnp structure itself could be a hashtable, and doesn't need to be (entirely) loaded into RAM. I'm saying you create a capnp list, and then you store elements into the list at position according to their hash -- i.e. how you'd build a hashtable, but the capnp list itself is the backing array. You would have to do this at write time, of course, and make sure the hashing is consistent between runs.

Do'h. Of course.

I'd still prefer the framework do it for me. It seems quite involved.

Thanks for taking time to answer my questions!

Re: Cap’n Proto

#153

Post by dekhn »

kentonv wrote: post #144
dekhn wrote: post #117

Earlier quoted context omitted.

proto2 has maps, but like proto3 they aren't maps, they're an randomly ordered sequence of key value pairs ugh .

Presumably the lookup table is built at parse time? (Proto2 definitely didn't have any built-in notion of maps when I was working on it. I thought maps were added as a proto3 feature...)

https://developers.google.com/protocol-buffers/docs/proto#ma...

I don't think any lookup table is provided (the wire order of entries is undefined). They are not lookup maps, they are syntactic sugar for repeated key/value pairs.

Re: Cap’n Proto

#154

Post by heavenlyhash »

kentonv wrote: post #142
heavenlyhash wrote: post #108

Earlier quoted context omitted.

I like schemas for validation and strong types. I also like self-describing systems when all goes to hell and I'm debugging either my own protocols or someone else's. I have this crazy idea that there's a middle ground: self-describing and schemas? We can kind of glue this together (there's lots of json schemas floating around out there now, it seems), but it would be awfully interesting to see these well-supported a…

When I'm forced to use JSON, I often do it by defining a Cap'n Proto schema and using capnp::JsonCodec to parse / serialize that schema as JSON. https://github.com/sandstorm-io/capnproto/blob/master/c++/sr... (There are also similar libraries for Protobuf.)

That's..... I want that in my lang.

Re: Cap’n Proto

#155

Post by kentonv »

dekhn wrote: post #153
kentonv wrote: post #144

Earlier quoted context omitted.

Presumably the lookup table is built at parse time? (Proto2 definitely didn't have any built-in notion of maps when I was working on it. I thought maps were added as a proto3 feature...)

https://developers.google.com/protocol-buffers/docs/proto#ma... I don't think any lookup table is provided (the wire order of entries is undefined). They are not lookup maps, they are syntactic sugar for repeated key/value pairs.

The "lookup table" is constructed at parse time. Yes, the items are just key/values on the wire, but when parsed (in C++, at least) they are placed in a google::protobuf::Map, which is hashtable-based. I guess it could differ across languages -- some might not have any specific support for maps yet.

Re: Cap’n Proto

#156

Post by heavenlyhash »

DenisM wrote: post #112
heavenlyhash wrote: post #108

Earlier quoted context omitted.

I like schemas for validation and strong types. I also like self-describing systems when all goes to hell and I'm debugging either my own protocols or someone else's. I have this crazy idea that there's a middle ground: self-describing and schemas? We can kind of glue this together (there's lots of json schemas floating around out there now, it seems), but it would be awfully interesting to see these well-supported a…

http://cbor.io

Doesn't really have an implementation-agnostic schema spec, as far as I'm aware.

(I use CBOR a lot -- I'm otherwise quite happy with it!)

EDIT: I guess there's a "CDDL" listed on the tools page, but... It's still single implementation (ruby) and I don't see a clear link to a grammar for it.

Re: Cap’n Proto

#157

Post by kentonv »

Matthias247 wrote: post #150
kentonv wrote: post #146

Earlier quoted context omitted.

No no, this is a common misunderstanding about Cap'n Proto. It does not take your regular in-memory data structures from your regular programming language (even C++) and put them on the wire. What it does is defines its own specific data layout which happens to be appropriate both for in-memory random-access use and for transmission. Cap'n Proto generates classes which wrap a byte buffer and give you accessor methods…

Ok, but these accessor methods will still have a very different performance. In C++ copying a UTF8 string from one byte array into an std::string is super fast. Whereas in JS it's really slow, since you need to read from an ArrayBuffer, convert code points to UTF16 in JS and then store these in a string (which is not efficient, since the strings are immutable). In node you could at least speed that up through some na…

Even if you access every field, Cap'n Proto's approach should still be faster in theory because:

- Making one pass instead of two is better for the cache. When dealing with messages larger than the CPU cache, memory bandwidth can easily be the program's main bottleneck, at which point using one pass instead of two can actually double your performance.

- Along similar lines, when you parse a protobuf upfront, you have to parse it into some intermediate structure. That intermediate structure takes memory, which adds cache pressure. Cap'n Proto has no intermediate structure.

- Protobuf and many formats like it are branch-heavy. For example, protobuf likes to encode integers as "varints" (variable-width integers), which require a branch on every byte to check if it's the last byte. Also, protobuf is a tag-value stream, which means the parser has to be a switch-in-a-loop, which is a notoriously CPU-unfriendly pattern. Cap'n Proto uses fixed widths and fixed offsets, which means there are very few branches. As a result, an upfront Cap'n Proto parser would be expected to outperform a Protobuf parser. The fact that parsing happens lazily at time of use is a bonus.

All that said, it's true that if you are reading every field of your structure, then Cap'n Proto serialization is more of an incremental improvement, not a paradigm shift.

Re: Cap’n Proto

#158

Post by socmag »

aidenn0 wrote: post #77
kentonv wrote: post #47

Earlier quoted context omitted.

I haven't looked at Ion before, but it appears to be similar to BSON or Msgpack in that it's a binary format that encodes field names as textual identifiers, to be "self-describing" and avoid the need for an external schema. I generally like schemas, because I like static typing. Of course, static types vs. dynamic types are another ancient flamewar and I'm unlikely to cover any new ground by stating arguments here.…

Schemas are IMO the only way to go when you are transferring between heterogenous languages, even when all languages involved are untyped. Consider javascript talking to common lisp. Of course JSON has a canonical mapping to javascript, but it does not for common lisp. Should a JS array be a lisp list or vector? Should lisp's NIL be false or null? Should a JS object decode to an alist, plist, or hash-table? &ct.

I think there is a lot in common between Common Lisp, Python and JavaScript in general.

For many years I was in the schemaless camp before JS came along. Then for a number of years I was in the self-describing camp because I was thinking that if we don't accept JavaScript and JSON are pretty fundamental on the web we're fools and everyone seemed to be passing around JSON. So in that period was thinking that MsgPack was pretty damn good.

Recently I've switched back to the schemaless view, but with strict order preservation and richly typed fields. Very "tuple" based... so works well with Lisp and JavaScript but also C++. Highly inspired by Linda.

I don't do what Cap'n Proto does and lay out the fields and all that good stuff so that you can kind of memory map it onto structs.

That is nice and I understand the motivation for sure, and I have worked on systems that do that in the past with very good results, but currently my thinking is that compactness without additional compression is a good balance.

Also, since the protocol is order preserving (where it matters) you can do radix sort operations or hash maps on the server side extremely quickly. That was the ultimate motivating factor.

[SomeDateTime, "hello", 1.0f, [10,20], "Foo", false, [SomeMatrix]] etc etc...

Inner tuples or BLOB's are length prefixed of course so you can skip them, but basic types and strings are not. Strings are zero terminated while Ints/Floats/Doubles are BE and complement encoded to preserve sorts, and also Integers are packed to minimum size.

Memory mapping is possible on this system too, but it's very "functional"... there's no attempt at pointer preservation. I don't go that far and Cap'n P seems to be preserving some of the semantics of ProtoBuf at least in that regard, which I'm sure is a good thing for many scenarios.

You could still do that with what I'm doing but it would be at the application level. Same with self-description actually, you could easily build something that looks like JSON if you wanted... either:

[["foo",1.0],["boo","cat"]...] etc

or

[[1.0,"cat"],["foo",boo"]

the choice is really up to the application programmer.

That might be a bit "loose" for a lot of people to stomach, but it works very well for what I'm doing, has a lot of flexibility and packs really well

Like I was suggesting, protocols are something of an art and I don't think we're at a final solution yet, which is why many people are constantly inventing new ones :-)

Hopefully we will get to some consensus one day!

Re: Cap’n Proto

#159

Post by kentonv »

anongoogler2 wrote: post #151
kentonv wrote: post #11

Hi all, Cap'n Proto author here. Thanks for the post. Just wanted to note that although Cap'n Proto hasn't had a blog post or official release in a while, development is active as part of the Sandstorm project ( https://sandstorm.io ). Cap'n Proto -- including the RPC system -- is used extensively in Sandstorm. Sandboxed Sandstorm apps in fact do all their communications with the outside world through a single Cap'n…

I could imagine implementing something like this as an alternate wire format and client library for protocol buffers. Can you outline the main reason you chose not to go this route? In particular, what aspects of the proto descriptor language don't fit Cap'n Proto? Or what aspects did you feel needed to be changed for some other reason?

To be clear, by "alternate wire format and client library for protobuf" you mean "reuse the protobuf IDL but change everything else", right?

There are a few major reasons I didn't go that route:

* The .proto language has a lot of weird quirks that I don't like. Some of the quirks are specific to the protobuf encoding (e.g. int32 vs. sint32 vs. fixed32 being different types), while other quirks have no particular rationale behind them. I didn't want that baggage.

* The .proto language does not treat interfaces (aka services) as a first-class type. That is, you cannot define a field whose type is an RPC interface type -- a reference to a remote object. The ability to do this is a critical part of Cap'n Proto's interface design.

* It's highly unlikely that the protobuf team would be interested in accepting changes to the language which were not actually supported by protobuf. This means that if I shared the language, I would have my hands tied when it comes to new features -- or I'd also have to implement Protobuf equivalents to make them happy.

Re: Cap’n Proto

#160

Post by socmag »

socmag wrote: post #158
aidenn0 wrote: post #77

Earlier quoted context omitted.

Schemas are IMO the only way to go when you are transferring between heterogenous languages, even when all languages involved are untyped. Consider javascript talking to common lisp. Of course JSON has a canonical mapping to javascript, but it does not for common lisp. Should a JS array be a lisp list or vector? Should lisp's NIL be false or null? Should a JS object decode to an alist, plist, or hash-table? &ct.

I think there is a lot in common between Common Lisp, Python and JavaScript in general. For many years I was in the schemaless camp before JS came along. Then for a number of years I was in the self-describing camp because I was thinking that if we don't accept JavaScript and JSON are pretty fundamental on the web we're fools and everyone seemed to be passing around JSON. So in that period was thinking that MsgPack w…

Also I might add that I very quickly became quite disenchanted with the lack of types in JSON; and I'm sure many of us have come to that same conclusion.

Really hope that it can be fixed at some point, but not holding my breath on that one.

I think it will take a major effort to reform that format although I am hopeful now that we at least have UInt8Array and friends that are starting to expose a broader set of machine friendly types.

Post reply on HN