Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cap’n Proto

capnproto.org

Post reply on HN
Print view View original

121–130 of 194 posts

Re: Cap’n Proto

#121

Post by adrianratnapala »

lux wrote: post #115
kentonv wrote: post #11

Hi all, Cap'n Proto author here. Thanks for the post. Just wanted to note that although Cap'n Proto hasn't had a blog post or official release in a while, development is active as part of the Sandstorm project ( https://sandstorm.io ). Cap'n Proto -- including the RPC system -- is used extensively in Sandstorm. Sandboxed Sandstorm apps in fact do all their communications with the outside world through a single Cap'n…

I recently ran into flatbuffers https://google.github.io/flatbuffers ), which claims the same no-serialization step. Curious how the two compare?

Does flatbuffers have an RPC mechanism? Or is it purely a serialisation tool?

Cap'n Proto is both. I.e. you have the serialisation, which can be used on its own, but you can also define interfaces with methods that can pass those serialised structs as parameters, and which return asynchronous promises.

Re: Cap’n Proto

#122

Post by adrianratnapala »

kentonv wrote: post #37
wtbob wrote: post #31

> The Cap’n Proto encoding is appropriate both as a data interchange format and an in-memory representation, so once your structure is built, you can simply write the bytes straight out to disk! Eh, I'd rather pay the cost of serialisation once and deserialisation once, and then access my data for as close to free as possible, rather than relying on a compiler to actually inline calls properly. > Integers use little-…

Big-endian vs. little-endian is an ancient flamewar that isn't going to go away any time soon, but sure, let's argue. Once you've spent as much time twiddling bits as I have (as the author of proto2 and Cap'n Proto), you start to realize that little endian is much easier to work with than big-endian. For example: - To reinterpret a 64-bit number as 32-bit in BE, you have to add 4 bytes to your pointer. In LE, the poi…

There's really no good argument for big-endian encoding except that it's the ordering that we humans use in writing. And not even always that. We call our numbering system "Arabic", but for the Arabs, it little-endian.

For some reason humans seem to want high powers on the left, even if it makes no sense in a left-to-right language.

Take polynomials, they are typically written big-endian

      ax^2 + bx + c
But infinite series have to be little-endian.

      c_0  +  c_1*x  + c_2*x^2 ....
If you think for a moment about how you would write multiplication, you will see the latter form is much easier to reason about and program with.

Re: Cap’n Proto

#123

Post by mrfusion »

striking wrote: post #109
mrfusion wrote: post #106

I hate to ask but can anyone explain this like I'm from 2000? I guess it's a way to send data to your front end java script but not use json and this compresses it so it's faster? How much better than using json is it?

It lets you put data on the wire, in a structured format, right out of memory. Asking the question "how much faster is it" isn't even valid here, because it skips the usual serialization process. Cap'n Proto generates you some code that contains some data structures. You put data into these structures, and they will automatically be in the right shape to put directly on the wire. And then that data can be pulled righ…

So could django send data to jquery with this? Or what are some simple use cases?

Re: Cap’n Proto

#124

Post by scott_karana »

mrfusion wrote: post #123
striking wrote: post #109

Earlier quoted context omitted.

It lets you put data on the wire, in a structured format, right out of memory. Asking the question "how much faster is it" isn't even valid here, because it skips the usual serialization process. Cap'n Proto generates you some code that contains some data structures. You put data into these structures, and they will automatically be in the right shape to put directly on the wire. And then that data can be pulled righ…

So could django send data to jquery with this? Or what are some simple use cases?

I think this is aimed at lower-level problems: RPC between interoperating daemons, connector protocols for databases, that sort of thing.

Lack of serialization is probably useful for JavaScript, but "exact in-memory data formats" probably don't fit well with dynamic-typed languages ;)

Re: Cap’n Proto

#126

Post by lmm »

kentonv wrote: post #11

Hi all, Cap'n Proto author here. Thanks for the post. Just wanted to note that although Cap'n Proto hasn't had a blog post or official release in a while, development is active as part of the Sandstorm project ( https://sandstorm.io ). Cap'n Proto -- including the RPC system -- is used extensively in Sandstorm. Sandboxed Sandstorm apps in fact do all their communications with the outside world through a single Cap'n…

If those platforms are causing issues would it be worth downgrading them to a second-class "LTS releases only" state or some such?

Re: Cap’n Proto

#127

Post by adrianratnapala »

kentonv wrote: post #51
zump wrote: post #42

Earlier quoted context omitted.

Why did you leave Google?

Well, I could write a lot about this, but... In general, because I felt there was too much resistance to me pursuing ideas that I wanted to pursue. Google has become increasingly top-down, whereas it was fairly bottom-up when I started in 2005. That's not necessarily a bad thing, but I felt that I personally would be happier running a startup where I could call the shots. FWIW, if you just want to write code, be comf…

... some people at Google would surely argue that my problem is that my ideas are crazy and bad ...

To what extent are such disagreements purely technical (e.g. byte-oriented wire encodings vs. roughly-like-memory) and to what extent are they "political".

By "political" I am talking about your claims about decentralisation and diversity, as if you are selling Sandstorm as the IBM-PC for the cloud millennium: a neutral platform any old software developer. That is very different from Google's way of doing doing business.

Re: Cap’n Proto

#128

Post by lmm »

mrfusion wrote: post #106

I hate to ask but can anyone explain this like I'm from 2000? I guess it's a way to send data to your front end java script but not use json and this compresses it so it's faster? How much better than using json is it?

It's a lot higher-performance, because you virtually don't have to parse. Remember when MySQL started supporting the memcached protocol because it'd reached the point where form simple pkey lookups it was spending more time parsing an SQL query than actually executing it? It's like doing that for your program.

But for me at least, the real advantage over JSON isn't the performance but the schema compatibility. You have a spec for your data and generate code from that, which means the spec is guaranteed to be correct, and there's clear documentation about what changes to the spec are or aren't forward or backward compatibile. (You get the same thing from the original Protocol Buffers though).

Re: Cap’n Proto

#129

Post by networked »

david-given wrote: post #94
Hydraulix989 wrote: post #93

Earlier quoted context omitted.

There's still a bunch of minor differences/annoyances with the MS toolchain that require me to go in and make a bunch of small changes everywhere to my project, which compiles fine on clang/g++ (and I wouldn't say I'm doing anything _crazy_ like the aforementioned "Expression SFINAE"). Right now, my project's rather small 100 kLOC codebase compiles (with very minimal #ifdef hackery) on Android, iOS, Mac, and Linux; b…

I used to develop the Windows port of WordGrinder (my console word processor) using Mingw for Linux, testing with Wine and packaging using nsys for Linux. This actually worked pretty well and allowed a complete Windows development flow from Linux using the same rolling automation for both platforms. The big problem was Wine, which can behave quite differently to real Windows in some places. It's certainly not bug-for…

You might mean the differences between MinGW and MinGW-w64. The former is the version hosted at http://mingw.org that used to be called mingw32 (not to be confused with mingw-w64-i686). The latter is what a lot of Linux distributions package today and what you install with MSYS2 [1] on Windows. It was initially forked from MinGW to integrate support for the 64-bit Windows API, hence the name, but it has gained other features since and is more actively maintained.

I've run into a difference between the two recently. It turns out that MinGW implemented POSIX glob() post-fork, which hasn't been ported to MinGW-w64.

[1] As I side note, I found that MSYS2 makes a huge difference in how you develop software for Windows with a POSIXy toolchain. Anyone who uses MinGW on Windows should try it.

Re: Cap’n Proto

#130

Post by Drdrdrq »

striking wrote: post #109
mrfusion wrote: post #106

I hate to ask but can anyone explain this like I'm from 2000? I guess it's a way to send data to your front end java script but not use json and this compresses it so it's faster? How much better than using json is it?

It lets you put data on the wire, in a structured format, right out of memory. Asking the question "how much faster is it" isn't even valid here, because it skips the usual serialization process. Cap'n Proto generates you some code that contains some data structures. You put data into these structures, and they will automatically be in the right shape to put directly on the wire. And then that data can be pulled righ…

Well, to be fair, serialization is still performed - but interchange format is cleverly picked out to be as similar as possible to common memory representation.
Post reply on HN