Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Arrow: A new open source in-memory columnar data format

blogs.apache.org

Post reply on HN
Print view View original

21–30 of 47 posts

Re: Apache Arrow: A new open source in-memory columnar data format

#21

Post by perishabledave »

jaltekruse wrote: post #3
bcoates wrote: post #2

If you don't speak press-release, this is a cool project to create an in-memory interop format for columnar data, so various tools can share data without a serialize/deserialize step which is very expensive compared to copying or sharing memory on the same machine or within a local network. https://git-wip-us.apache.org/repos/asf?p=arrow.git;a=blob;f... (edited post because I fail reading git and didn't notice the ja…

Disclosure I am a committer on Apache Drill and Apache Arrow. This isn't actually true. The java implementation has been complete and used in Apache Drill, a distributed SQL engine, for the past few years. While we anticipate a few small changes to make sure the standard works well across new systems, this is by no means an announcement without tested code. https://git-wip-us.apache.org/repos/asf?p=arrow.git;a=commit…

I stumbled upon Google's whitepaper on Dremel. IIRC it explained how to store the data in columnar format, but I didn't quite get how that translated into quick queries. Happen to know where I can look to better understand how it works?

Re: Apache Arrow: A new open source in-memory columnar data format

#22

Post by jaltekruse »

khc wrote: post #19
buremba wrote: post #18

Nice to see a new columnar data format alternative. Just a quick question though. The existing columnar data formats such as Parquet and ORC aim to be space-efficient since the data is stored in disk and IO operations are usually the bottleneck. The columnar data formats shine in big-data area so the amount of data will be huge. Given that columnar data formats can be compressed efficiently and that's of the main poi…

Random memory access isn't really constant time when you factor in hardware prefetch and cache lines. See "What Every Programmer Should Know About Memory"

Exactly this, the most basic example is an operation on a single column. If the data for a single column is mixed in with other data in a row-oriented layout, you are going to have to bring a new chunk of memory into the CPU cache sooner to process a given amount of data. If all of the data is packed together tightly you can read many more values out of the cache before you need to go get more from main memory.

Re: Apache Arrow: A new open source in-memory columnar data format

#23

Post by barneso »

david-given wrote: post #6

Is it streamable? Could I use this as an intermediate format to send columnar data between two processes via a pipe?

Better, if the processes are on the same machine you could use it to share the data via shared memory or a common memory mapping, to avoid having copies of the data on each end of the pipe.

Re: Apache Arrow: A new open source in-memory columnar data format

#24

Post by rch »

sandGorgon wrote: post #17
rch wrote: post #12

In-memory only... How is this better than the SFrame implementation from Dato (2015) that was posted here a couple of days ago? https://news.ycombinator.com/item?id=11106501

even i had that question - especially when people are talking about using SFrame as the underlying structure for Julia. but then I saw this: "Arrow's cross platform and cross system strengths will enable Python and R to become first-class languages across the entire Big Data stack," said Wes McKinney, creator of Pandas. Code committers to Apache Arrow include developers from Apache Big Data projects Calcite, Cassandr…

> this is pretty much nuke-from-orbit

That analogy might imply overkill, thus highlighting the tactical advantages of the SFrame approach in processing a month's worth of 1-10GB daily-generated SQLite files, for instance.

Re: Apache Arrow: A new open source in-memory columnar data format

#26

Post by TallGuyShort »

perishabledave wrote: post #21
jaltekruse wrote: post #3

Earlier quoted context omitted.

Disclosure I am a committer on Apache Drill and Apache Arrow. This isn't actually true. The java implementation has been complete and used in Apache Drill, a distributed SQL engine, for the past few years. While we anticipate a few small changes to make sure the standard works well across new systems, this is by no means an announcement without tested code. https://git-wip-us.apache.org/repos/asf?p=arrow.git;a=commit…

I stumbled upon Google's whitepaper on Dremel. IIRC it explained how to store the data in columnar format, but I didn't quite get how that translated into quick queries. Happen to know where I can look to better understand how it works?

I'll take a stab at explaining it myself: "transactional queries" are faster in a traditional format because they access many columns in few rows. For instance, if you want to log in to a website, you access the username, password, and possibly other authentication factors for a single user: this ends up being faster if you can go to the user's row, and then read all those fields in a contiguous scan.

"Analytical queries" are faster in a column-based format, because you're doing things like computing the correlation between 2 variables. Instead of looking at many columns in a specific row, you're looking at most of the data in a few columns. So instead of reading a whole row at once, it would be nice to skip the columns you don't care about and just grab big chunks of two or three columns.

Does that make sense?

edit: For a long time I was confused about why HBase was described as a columnar data store when access was still pretty row-based. I think the reason is because you group columns into column families which can be stored and retrieved separately, so you still get some of the benefit of a true column-oriented store.

Re: Apache Arrow: A new open source in-memory columnar data format

#27

Post by TallGuyShort »

khc wrote: post #19
buremba wrote: post #18

Nice to see a new columnar data format alternative. Just a quick question though. The existing columnar data formats such as Parquet and ORC aim to be space-efficient since the data is stored in disk and IO operations are usually the bottleneck. The columnar data formats shine in big-data area so the amount of data will be huge. Given that columnar data formats can be compressed efficiently and that's of the main poi…

Random memory access isn't really constant time when you factor in hardware prefetch and cache lines. See "What Every Programmer Should Know About Memory"

That, and regardless of row- or column-orientation, any common in-memory format that was actually as well-established as this looks like it will be in Big Data projects would be nice.

Re: Apache Arrow: A new open source in-memory columnar data format

#28

Post by tanlermin »

rch wrote: post #24
sandGorgon wrote: post #17

Earlier quoted context omitted.

even i had that question - especially when people are talking about using SFrame as the underlying structure for Julia. but then I saw this: "Arrow's cross platform and cross system strengths will enable Python and R to become first-class languages across the entire Big Data stack," said Wes McKinney, creator of Pandas. Code committers to Apache Arrow include developers from Apache Big Data projects Calcite, Cassandr…

> this is pretty much nuke-from-orbit That analogy might imply overkill, thus highlighting the tactical advantages of the SFrame approach in processing a month's worth of 1-10GB daily-generated SQLite files, for instance.

Python's Dask out of core dataframe can also do that.

Re: Apache Arrow: A new open source in-memory columnar data format

#29

Post by infinite8s »

perishabledave wrote: post #21
jaltekruse wrote: post #3

Earlier quoted context omitted.

Disclosure I am a committer on Apache Drill and Apache Arrow. This isn't actually true. The java implementation has been complete and used in Apache Drill, a distributed SQL engine, for the past few years. While we anticipate a few small changes to make sure the standard works well across new systems, this is by no means an announcement without tested code. https://git-wip-us.apache.org/repos/asf?p=arrow.git;a=commit…

I stumbled upon Google's whitepaper on Dremel. IIRC it explained how to store the data in columnar format, but I didn't quite get how that translated into quick queries. Happen to know where I can look to better understand how it works?

This is a good high level overview from Twitter (who created Parquet): https://blog.twitter.com/2013/dremel-made-simple-with-parque...

Re: Apache Arrow: A new open source in-memory columnar data format

#30

Post by infinite8s »

tanlermin wrote: post #28
rch wrote: post #24

Earlier quoted context omitted.

> this is pretty much nuke-from-orbit That analogy might imply overkill, thus highlighting the tactical advantages of the SFrame approach in processing a month's worth of 1-10GB daily-generated SQLite files, for instance.

Python's Dask out of core dataframe can also do that.

Dasks' out of core dataframes are just a thin wrapper around pandas dataframes (aided by the recent improvement in pandas to release the GIL on a bunch of operations)
Post reply on HN