Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Arrow: A new open source in-memory columnar data format

blogs.apache.org

Post reply on HN
Print view View original

41–47 of 47 posts

Re: Apache Arrow: A new open source in-memory columnar data format

#41

Post by okigan »

bcoates wrote: post #2

If you don't speak press-release, this is a cool project to create an in-memory interop format for columnar data, so various tools can share data without a serialize/deserialize step which is very expensive compared to copying or sharing memory on the same machine or within a local network. https://git-wip-us.apache.org/repos/asf?p=arrow.git;a=blob;f... (edited post because I fail reading git and didn't notice the ja…

Where are the API docs? this seem rather useless: https://github.com/apache/arrow

Re: Apache Arrow: A new open source in-memory columnar data format

#42

Post by NovaX »

"Modern CPUs are designed to exploit data-level parallelism via vectorized operations and SIMD instructions. Arrow facilitates such processing."

How will Arrow use vectorized instructions on the JVM? That seems to be only available to the JIT and JNI, which is a frustrating limitation.

Re: Apache Arrow: A new open source in-memory columnar data format

#43

Post by infinite8s »

xtacy wrote: post #7

Nice initiative. Cheap serde and cross-language compatibility with an eye towards data scan intensive workloads is an important component! Have you folks considered Supersonic engine from Google, which was designed with similar (but not as extensive as Arrow) goals in mind? https://github.com/google/supersonic

Do you have any experience with Supersonic? It appears to be abandoned (at least there's no activity on the mailing list as of 2014 - https://groups.google.com/forum/#!forum/supersonic-query-eng...)

Re: Apache Arrow: A new open source in-memory columnar data format

#45

Post by [deleted] »

jaltekruse wrote: post #3
bcoates wrote: post #2

If you don't speak press-release, this is a cool project to create an in-memory interop format for columnar data, so various tools can share data without a serialize/deserialize step which is very expensive compared to copying or sharing memory on the same machine or within a local network. https://git-wip-us.apache.org/repos/asf?p=arrow.git;a=blob;f... (edited post because I fail reading git and didn't notice the ja…

Disclosure I am a committer on Apache Drill and Apache Arrow. This isn't actually true. The java implementation has been complete and used in Apache Drill, a distributed SQL engine, for the past few years. While we anticipate a few small changes to make sure the standard works well across new systems, this is by no means an announcement without tested code. https://git-wip-us.apache.org/repos/asf?p=arrow.git;a=commit…

[deleted]

Re: Apache Arrow: A new open source in-memory columnar data format

#46

Post by vdm »

TallGuyShort wrote: post #26
perishabledave wrote: post #21

Earlier quoted context omitted.

I stumbled upon Google's whitepaper on Dremel. IIRC it explained how to store the data in columnar format, but I didn't quite get how that translated into quick queries. Happen to know where I can look to better understand how it works?

I'll take a stab at explaining it myself: "transactional queries" are faster in a traditional format because they access many columns in few rows. For instance, if you want to log in to a website, you access the username, password, and possibly other authentication factors for a single user: this ends up being faster if you can go to the user's row, and then read all those fields in a contiguous scan. "Analytical que…

> For a long time I was confused about why HBase was described as a columnar data store when access was still pretty row-based

The term was overloaded by column-family stores, which were often referred to as just 'column stores', probably by people who were not aware of systems like Vertica and MonetDB.

http://dbmsmusings.blogspot.co.uk/2010/03/distinguishing-two...

Re: Apache Arrow: A new open source in-memory columnar data format

#47

Post by tycho01 »

I'm interpreting this as saying they want to use the same representation to use in memory (for querying) and for 'serialization' (sending the same thing over the wire). This begs the question why separate serialized representations ever became a thing in the first place.

My understanding is that serialization became a thing because in-memory representations tend to use pointers to shared data structures that may thus be referenced multiple times while being stored only once. This would not translate 1:1 to serialized representations (where memory offsets would no longer hold meaning) -- much less in any language-agnostic way.

So I have this suspicion that Apache Arrow would not support reusing duplicate data while storing it only once. Would anyone mind clarifying on this point?

Post reply on HN