Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cap’n Proto

capnproto.org

Post reply on HN
Print view View original

31–40 of 194 posts

Re: Cap’n Proto

#31

Post by wtbob »

> The Cap’n Proto encoding is appropriate both as a data interchange format and an in-memory representation, so once your structure is built, you can simply write the bytes straight out to disk!

Eh, I'd rather pay the cost of serialisation once and deserialisation once, and then access my data for as close to free as possible, rather than relying on a compiler to actually inline calls properly.

> Integers use little-endian byte order because most CPUs are little-endian, and even big-endian CPUs usually have instructions for reading little-endian data.

sob There are a lot of things Intel has to account for, and frankly little-endian byte order isn't the worst of them, but it's pretty rotten. Writing 'EFCDAB8967452301' for 0x0123456789ABCDEF is perverse in the extreme. Why? Why?

As pragmatic design choices go, Cap'n Proto's is a good one (although it violates the standard network byte order). Intel effectively won the CPU war, and we'll never be free of the little-endian plague.

It's all so depressing.

Re: Cap’n Proto

#32

Post by oneloop »

kentonv wrote: post #11

Hi all, Cap'n Proto author here. Thanks for the post. Just wanted to note that although Cap'n Proto hasn't had a blog post or official release in a while, development is active as part of the Sandstorm project ( https://sandstorm.io ). Cap'n Proto -- including the RPC system -- is used extensively in Sandstorm. Sandboxed Sandstorm apps in fact do all their communications with the outside world through a single Cap'n…

What a great pitch, that website is. I don't even understand what it is, but I'm stocked.

Re: Cap’n Proto

#33

Post by kentonv »

lobster_johnson wrote: post #27
kentonv wrote: post #11

Hi all, Cap'n Proto author here. Thanks for the post. Just wanted to note that although Cap'n Proto hasn't had a blog post or official release in a while, development is active as part of the Sandstorm project ( https://sandstorm.io ). Cap'n Proto -- including the RPC system -- is used extensively in Sandstorm. Sandboxed Sandstorm apps in fact do all their communications with the outside world through a single Cap'n…

I'd like to see Ruby bindings (not just serialization). We are using Go, Node.js and Ruby in production, and we have been looking to move from plain HTTP to gRPC. If Cap'n Proto is better, we might use it, but not without Ruby bindings.

I don't think anyone is currently working on Ruby bindings, but if you're interested, feel free to take it on!

(Note that Sandstorm is focused on making Cap'n Proto work well for Sandstorm. We welcome contributions, but we generally don't have resources available ourselves to work on third-party feature requests unrelated to Sandstorm. That is, unless you want to pay us a bunch of money, in which case, feel free to contact me. ;) )

Re: Cap’n Proto

#34

Post by morecoffee »

wtbob wrote: post #31

> The Cap’n Proto encoding is appropriate both as a data interchange format and an in-memory representation, so once your structure is built, you can simply write the bytes straight out to disk! Eh, I'd rather pay the cost of serialisation once and deserialisation once, and then access my data for as close to free as possible, rather than relying on a compiler to actually inline calls properly. > Integers use little-…

> Eh, I'd rather pay the cost of serialisation once and deserialisation once, and then access my data for as close to free as possible, rather than relying on a compiler to actually inline calls properly.

The thing is, as close to free as possible is surprisingly expensive. Protobuf's varint encoding is extremely branchy, and hurts performance in a datacenter environment (where bandwidth is free, and CPU is expensive).

> As pragmatic design choices go, Cap'n Proto's is a good one (although it violates the standard network byte order). Intel effectively won the CPU war, and we'll never be free of the little-endian plague.

Did they though? Arguably there are far more ARM CPUs (like the one in your pocket) than there are server CPUs. Since cellphones and other low power devices are almost all big endian, it seems like network byte order would have been better to use. High powered servers can pay the cost of coding them, but battery powered devices cannot afford to do so.

Re: Cap’n Proto

#35

Post by morecoffee »

> capability-based RPC system.

This sounds like a cool idea, but so far I haven't seen any good explanation of how it works, and why it will save me from rolling my own ACL system. For bragging about it in the very first sentence, there is surprisingly little detail about how it works.

Re: Cap’n Proto

#36

Post by niftich »

morecoffee wrote: post #34
wtbob wrote: post #31

> The Cap’n Proto encoding is appropriate both as a data interchange format and an in-memory representation, so once your structure is built, you can simply write the bytes straight out to disk! Eh, I'd rather pay the cost of serialisation once and deserialisation once, and then access my data for as close to free as possible, rather than relying on a compiler to actually inline calls properly. > Integers use little-…

> Eh, I'd rather pay the cost of serialisation once and deserialisation once, and then access my data for as close to free as possible, rather than relying on a compiler to actually inline calls properly. The thing is, as close to free as possible is surprisingly expensive. Protobuf's varint encoding is extremely branchy, and hurts performance in a datacenter environment (where bandwidth is free, and CPU is expensive…

ARM (since v3) is bi-endian for data accesses and defaults to little. You can confirm this by searching the ARM Information Center [1] for 'support for mixed-endian data', but I can't get a working URL for that exact page.

[1] http://infocenter.arm.com/help/index.jsp

Re: Cap’n Proto

#37

Post by kentonv »

wtbob wrote: post #31

> The Cap’n Proto encoding is appropriate both as a data interchange format and an in-memory representation, so once your structure is built, you can simply write the bytes straight out to disk! Eh, I'd rather pay the cost of serialisation once and deserialisation once, and then access my data for as close to free as possible, rather than relying on a compiler to actually inline calls properly. > Integers use little-…

Big-endian vs. little-endian is an ancient flamewar that isn't going to go away any time soon, but sure, let's argue.

Once you've spent as much time twiddling bits as I have (as the author of proto2 and Cap'n Proto), you start to realize that little endian is much easier to work with than big-endian.

For example:

- To reinterpret a 64-bit number as 32-bit in BE, you have to add 4 bytes to your pointer. In LE, the pointer doesn't change.

- Just about any arithmetic operation on integers (e.g. adding) starts from the least-significant bits and moves up. It's nice if that can mean iterating forward from the start instead of backwards from the end, e.g. when implementing a "bignum" library.

- Which of the following is simpler?

    // Extract nth bit from byte array, assuming LE order.
    (bytes[n/8] >> (n%8)) & 1

    // Extract nth bit from byte array, assuming BE order.
    (bytes[n/8] >> (7 - n%8)) & 1
There's really no good argument for big-endian encoding except that it's the ordering that we humans use in writing.

I think the correct answer won here.

Re: Cap’n Proto

#38

Post by jackmott »

niftich wrote: post #14

I've always liked Cap'n Proto because it was (quite literally) the ideas behind Protobuf taken to an extreme, or, depending on your point-of-view, reduced to its most basic components: data structures already have to sit in memory looking a certain way, why can't we just squirt that on the wire instead of some fancy bespoke type-length-value struct? Of course, the hardest part is convincing everyone that it's not you…

>data structures already have to sit in memory looking a certain way, why can't we just squirt that on the wire instead of some fancy bespoke type-length-value struct?

In C/C++ ya can! When making games in college that is exactly what we did. Take the struct, dump it into the socket. I was rather shocked when trying to recreate the same system in C#. "I can't? I CAN'T?"

Re: Cap’n Proto

#39

Post by kentonv »

morecoffee wrote: post #35

> capability-based RPC system. This sounds like a cool idea, but so far I haven't seen any good explanation of how it works, and why it will save me from rolling my own ACL system. For bragging about it in the very first sentence, there is surprisingly little detail about how it works.

It's a complicated topic -- it requires thinking about things in a different way, and tends not to make a lot of sense until at some point it "clicks" and you realize all sorts of patterns you were already using are actually special cases of capabilities.

Here is some reading:

https://capnproto.org/rpc.html#security

https://sandstorm.io/how-it-works#capabilities

http://zesty.ca/capmyths/usenix.pdf

Re: Cap’n Proto

#40

Post by teddyh »

kentonv wrote: post #37
wtbob wrote: post #31

> The Cap’n Proto encoding is appropriate both as a data interchange format and an in-memory representation, so once your structure is built, you can simply write the bytes straight out to disk! Eh, I'd rather pay the cost of serialisation once and deserialisation once, and then access my data for as close to free as possible, rather than relying on a compiler to actually inline calls properly. > Integers use little-…

Big-endian vs. little-endian is an ancient flamewar that isn't going to go away any time soon, but sure, let's argue. Once you've spent as much time twiddling bits as I have (as the author of proto2 and Cap'n Proto), you start to realize that little endian is much easier to work with than big-endian. For example: - To reinterpret a 64-bit number as 32-bit in BE, you have to add 4 bytes to your pointer. In LE, the poi…

I thought the best summary of the big-endian/little-endian question was written in 1980 in Internet Experiment Note (IEN) 137:

https://www.ietf.org/rfc/ien/ien137.txt

Also more reader-friendly here: https://www.computer.org/csdl/mags/co/1981/10/01667115.pdf

Post reply on HN