Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Arrow Flight: A Framework for Fast Data Transport

arrow.apache.org

Post reply on HN
Print view View original

1–10 of 20 posts

Re: Apache Arrow Flight: A Framework for Fast Data Transport

#3

Post by jumpingmice »

More people should try high performance services with non-traditional protobuf implementations. The fact that every language has a generated parser in no way preclude you from parsing them yourself. Hand-rolled serialization of your outbound messages can also be really fast, and the C++ gRPC stack will just accept preformatted messages and put them on the wire. Finally the existence of gRPC itself should not make you feel constrained against implementing the entire protocol yourself. It’s just HTTP/2 with conventional headers.

Re: Apache Arrow Flight: A Framework for Fast Data Transport

#5

Post by iRobbery »

RocketSyntax wrote: post #4

We are struggling with reliability when using mounting solutions for big data in S3. Would this help?

depends how you would use it i guess. And it seems quite bound to certain data formats.

I initially thought after reading the headline, data as in any kinds of bytes to replicate or something. But it is something else, mainly by reading "is focused on optimized transport of the Arrow columnar format (i.e. “Arrow record batches”) over gRPC"

Re: Apache Arrow Flight: A Framework for Fast Data Transport

#7

Post by wesm »

jumpingmice wrote: post #3

More people should try high performance services with non-traditional protobuf implementations. The fact that every language has a generated parser in no way preclude you from parsing them yourself. Hand-rolled serialization of your outbound messages can also be really fast, and the C++ gRPC stack will just accept preformatted messages and put them on the wire. Finally the existence of gRPC itself should not make you…

To be clear for anyone reading, we're parsing and generating the data-related protobufs ourselves, and retaining ownership of the memory returned by gRPC to obtain zero copy.

The C++ details are found in

https://github.com/apache/arrow/blob/master/cpp/src/arrow/fl...

Re: Apache Arrow Flight: A Framework for Fast Data Transport

#8

Post by jumpingmice »

wesm wrote: post #7
jumpingmice wrote: post #3

More people should try high performance services with non-traditional protobuf implementations. The fact that every language has a generated parser in no way preclude you from parsing them yourself. Hand-rolled serialization of your outbound messages can also be really fast, and the C++ gRPC stack will just accept preformatted messages and put them on the wire. Finally the existence of gRPC itself should not make you…

To be clear for anyone reading, we're parsing and generating the data-related protobufs ourselves, and retaining ownership of the memory returned by gRPC to obtain zero copy. The C++ details are found in https://github.com/apache/arrow/blob/master/cpp/src/arrow/fl...

Have you considered/tried using the new ParseContext instead of CodedInputStream? It is performance-oriented.

Edit: Apparently it's also the default in protobuf 3.10

Post reply on HN