Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cap’n Proto

capnproto.org

Post reply on HN
Print view View original

181–190 of 194 posts

Re: Cap’n Proto

#181

Post by kentonv »

DonHopkins wrote: post #170
DonHopkins wrote: post #84

Earlier quoted context omitted.

> and you don't care if you're implementing someone else's ideas and working for an advertising company FTFY

I'd be fascinated if someone who downvoted this would also be brave enough to explain why they're in denial that google is an advertising company. It may not be that your ideas are crazy and bad, but rather that your sane good ideas simply aren't appropriate in the context of advertising. Enough with the Stockholm syndrome. There are pleanty of perfectly great ideas that Google would never pay their employees to work…

I didn't downvote you (in fact, I can't, since you were replying to me), but:

Your post was snide. Snide remarks are commonly downvoted on HN, regardless of their accuracy.

Regarding your follow-up, you're taking a very simplistic view of Google that doesn't really capture reality. Google obviously builds lots of technology that doesn't directly affect advertising revenue. Most decision-makers at Google are not asking "how does this affect advertising revenue" for every decision. You'd know that if you ever worked there.

Re: Cap’n Proto

#182

Post by alfalfasprout »

throwaway13337 wrote: post #4

To get an overview of the area of binary interchange formats that are language agnostic, the author of Cap'n Proto does a good job in this: https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-... "Protocol Buffers" has been the go-to for a long time but there are more options now. For uses where serialization/deserialization CPU time is a concern, it seems to really a question of Cap'n Proto versus flatbuffer…

For our use (serializing streaming market data from direct NASDAQ and BATS feeds) we found both Capn'Proto and Flatbuffers to perform similarly.

Ultimately we went with flatbuffers because we found the API much cleaner across languages (C++/Go), but it's ultimately going to depend on your use case which one you use. Performance is pretty much identical. We populate our own custom structs from the capn'proto/flatbuffers structs anyways so we're never really doing zero-copy. That said, since both of these formats transmit numerical data as little-endian memory-representations of ints/float/longs/doubles their performance is fantastic.

Re: Cap’n Proto

#183

Post by heywire »

QuercusMax wrote: post #72
jackmott wrote: post #38

Earlier quoted context omitted.

>data structures already have to sit in memory looking a certain way, why can't we just squirt that on the wire instead of some fancy bespoke type-length-value struct? In C/C++ ya can! When making games in college that is exactly what we did. Take the struct, dump it into the socket. I was rather shocked when trying to recreate the same system in C#. "I can't? I CAN'T ?"

That's an absolutely terrible idea if you want any kind of forward / backward compatibility. Just about any system is better than dumping structs over the network, once you're dealing with a system that can evolve.

You might be surprised just how many successful applications simply dump a packed struct to disk as a serialization format (or even transmit it over the wire). If the operating environment (OS/hardware) is known to be of a certain type, endianness is not a concern.

Re: Cap’n Proto

#184

Post by dragonwriter »

DonHopkins wrote: post #170
DonHopkins wrote: post #84

Earlier quoted context omitted.

> and you don't care if you're implementing someone else's ideas and working for an advertising company FTFY

I'd be fascinated if someone who downvoted this would also be brave enough to explain why they're in denial that google is an advertising company. It may not be that your ideas are crazy and bad, but rather that your sane good ideas simply aren't appropriate in the context of advertising. Enough with the Stockholm syndrome. There are pleanty of perfectly great ideas that Google would never pay their employees to work…

> I'd be fascinated if someone who downvoted this would also be brave enough to explain why they're in denial that google is an advertising company.

Several reasons, but most critically:

The phrase "advertising company" generally refers to a firm in the business of creating advertising. Google's main business is an online media company selling advertising space (both in its own advertising-supported services and alongside online media provided by others.)

Re: Cap’n Proto

#185

Post by kentonv »

edraferi wrote: post #177
jtolmar wrote: post #44

Earlier quoted context omitted.

If Cap'n Proto is a joke, JSON beat it to the punchline. What's an efficient binary representation? C Structs. What's an efficient text representation? Javascript objects. But casting arbitrary data to a struct is a horrible idea. And eval'ing anonymous javascript is a horrible idea. Back to the drawing board. Then N years later someone has the brilliant idea of just... not parsing these formats that idiotic way. And…

I'd love to read the full story there - how JSON got so much momentum behind it. Do you know of a good overview blog post?

In my view, JSON beat XML because it matched the way we structure data in code: records and lists. XML, meanwhile, wants us to represent data structures as text with tags, which isn't how we represent data internally at all, requiring laborious translation steps.

The fact that it was "built-in" to Javascript, of course, helped, by allowing it to compete on even footing (XML is also built in via XMLHttpRequest).

Re: Cap’n Proto

#186

Post by kentonv »

easytiger wrote: post #175
kentonv wrote: post #11

Hi all, Cap'n Proto author here. Thanks for the post. Just wanted to note that although Cap'n Proto hasn't had a blog post or official release in a while, development is active as part of the Sandstorm project ( https://sandstorm.io ). Cap'n Proto -- including the RPC system -- is used extensively in Sandstorm. Sandboxed Sandstorm apps in fact do all their communications with the outside world through a single Cap'n…

From reading the examples, it looks like one can only serialise to a file descriptor. What if i wan to serialise to a byte array? Am I perhaps missing something obvious?

You can certainly serialize to a byte array. The example uses a file descriptor but in fact the code defines an abstract stream type which you can implement any way you want, and you can also obtain pointers to the message's underlying storage in order to extract the bytes directly (avoiding a copy).

Re: Cap’n Proto

#187

Post by kentonv »

chaotic-good wrote: post #171

I think that compression is a must for the serialization library. Protobuf uses almost twice less memory than Cap'n Proto. Using an external compression is not an option in some cases. E.g. consider building tcp-server that communicates with thousands of clients simultaneously. Each client connection will have its own LZ4 context that should be heap allocated. I believe it's about 16KB per connection + buffers. This…

Cap'n Proto offers "packed" encoding which applies light compression (removing zero-valued padding bytes), brings it in-line with Protobuf, and ought to be much faster than the things Protobuf does for "compression" (varint is a very slow encoding!).

Re: Cap’n Proto

#188

Post by DonHopkins »

dragonwriter wrote: post #184
DonHopkins wrote: post #170

Earlier quoted context omitted.

I'd be fascinated if someone who downvoted this would also be brave enough to explain why they're in denial that google is an advertising company. It may not be that your ideas are crazy and bad, but rather that your sane good ideas simply aren't appropriate in the context of advertising. Enough with the Stockholm syndrome. There are pleanty of perfectly great ideas that Google would never pay their employees to work…

> I'd be fascinated if someone who downvoted this would also be brave enough to explain why they're in denial that google is an advertising company. Several reasons, but most critically: The phrase "advertising company" generally refers to a firm in the business of creating advertising . Google's main business is an online media company selling advertising space (both in its own advertising-supported services and alo…

Is what the article says not true that "Google makes more than 90 percent of its revenue from online advertising" and "Google alone makes about half of all global mobile advertising revenue"?

That sounds like a pretty major advertising company to me.

Can you name a bigger advertising company than Google?

If they're actually a technology company driven by the demands of their users and striving for technical excellence, then do you expect Google to support built-in ad-blocking in Chrome and Android like Apple already supports it in Safari and iOS any time soon?

Re: Cap’n Proto

#189

Post by QuercusMax »

heywire wrote: post #183
QuercusMax wrote: post #72

Earlier quoted context omitted.

That's an absolutely terrible idea if you want any kind of forward / backward compatibility. Just about any system is better than dumping structs over the network, once you're dealing with a system that can evolve.

You might be surprised just how many successful applications simply dump a packed struct to disk as a serialization format (or even transmit it over the wire). If the operating environment (OS/hardware) is known to be of a certain type, endianness is not a concern.

It's not just endianness; it's about maintainability. If you ever think you might change the struct, EVER, you need to worry about these things.

Re: Cap’n Proto

#190

Post by easytiger »

kentonv wrote: post #186
easytiger wrote: post #175

Earlier quoted context omitted.

From reading the examples, it looks like one can only serialise to a file descriptor. What if i wan to serialise to a byte array? Am I perhaps missing something obvious?

You can certainly serialize to a byte array. The example uses a file descriptor but in fact the code defines an abstract stream type which you can implement any way you want, and you can also obtain pointers to the message's underlying storage in order to extract the bytes directly (avoiding a copy).

Great thanks. I'll give it a go again for a current project I'm kicking off.
Post reply on HN