Apache Arrow: A new open source in-memory columnar data format
blogs.apache.org
Apache Arrow: A new open source in-memory columnar data format
1–10 of 47 posts
Re: Apache Arrow: A new open source in-memory columnar data format
#2If you don't speak press-release, this is a cool project to create an in-memory interop format for columnar data, so various tools can share data without a serialize/deserialize step which is very expensive compared to copying or sharing memory on the same machine or within a local network.
https://git-wip-us.apache.org/repos/asf?p=arrow.git;a=blob;f...
(edited post because I fail reading git and didn't notice the java implementation)
Re: Apache Arrow: A new open source in-memory columnar data format
#3If you don't speak press-release, this is a cool project to create an in-memory interop format for columnar data, so various tools can share data without a serialize/deserialize step which is very expensive compared to copying or sharing memory on the same machine or within a local network. https://git-wip-us.apache.org/repos/asf?p=arrow.git;a=blob;f... (edited post because I fail reading git and didn't notice the ja…
Disclosure I am a committer on Apache Drill and Apache Arrow.
This isn't actually true. The java implementation has been complete and used in Apache Drill, a distributed SQL engine, for the past few years. While we anticipate a few small changes to make sure the standard works well across new systems, this is by no means an announcement without tested code.
https://git-wip-us.apache.org/repos/asf?p=arrow.git;a=commit...
Re: Apache Arrow: A new open source in-memory columnar data format
#4If you don't speak press-release, this is a cool project to create an in-memory interop format for columnar data, so various tools can share data without a serialize/deserialize step which is very expensive compared to copying or sharing memory on the same machine or within a local network. https://git-wip-us.apache.org/repos/asf?p=arrow.git;a=blob;f... (edited post because I fail reading git and didn't notice the ja…
To be fair this isn't the best UI for git. We were waiting for the mirror to Github to be set up, so we decided to just include the link to the Apache git repo UI to be safe the link was live when the press announcements went out.
The Github mirror is now live, you can see it here: https://github.com/apache/arrow
Re: Apache Arrow: A new open source in-memory columnar data format
#5If you don't speak press-release, this is a cool project to create an in-memory interop format for columnar data, so various tools can share data without a serialize/deserialize step which is very expensive compared to copying or sharing memory on the same machine or within a local network. https://git-wip-us.apache.org/repos/asf?p=arrow.git;a=blob;f... (edited post because I fail reading git and didn't notice the ja…
Does it mean that other engines could/should use that format?
Re: Apache Arrow: A new open source in-memory columnar data format
#6Is it streamable? Could I use this as an intermediate format to send columnar data between two processes via a pipe?
Re: Apache Arrow: A new open source in-memory columnar data format
#7Nice initiative. Cheap serde and cross-language compatibility with an eye towards data scan intensive workloads is an important component!
Have you folks considered Supersonic engine from Google, which was designed with similar (but not as extensive as Arrow) goals in mind?
Re: Apache Arrow: A new open source in-memory columnar data format
#8Could someone explain the difference between this and Avro or Parquet? Do they serve the same purpose?
Re: Apache Arrow: A new open source in-memory columnar data format
#9If you don't speak press-release, this is a cool project to create an in-memory interop format for columnar data, so various tools can share data without a serialize/deserialize step which is very expensive compared to copying or sharing memory on the same machine or within a local network. https://git-wip-us.apache.org/repos/asf?p=arrow.git;a=blob;f... (edited post because I fail reading git and didn't notice the ja…
To be fair this isn't the best UI for git. We were waiting for the mirror to Github to be set up, so we decided to just include the link to the Apache git repo UI to be safe the link was live when the press announcements went out. The Github mirror is now live, you can see it here: https://github.com/apache/arrow
As best I can tell, you don't have a "git clone" URL visible anywhere on the site. I have the vague impression that gitweb is capable of showing one but you have to configure it(?).
Re: Apache Arrow: A new open source in-memory columnar data format
#10Earlier quoted context omitted.
To be fair this isn't the best UI for git. We were waiting for the mirror to Github to be set up, so we decided to just include the link to the Apache git repo UI to be safe the link was live when the press announcements went out. The Github mirror is now live, you can see it here: https://github.com/apache/arrow
As best I can tell, you don't have a "git clone" URL visible anywhere on the site. I have the vague impression that gitweb is capable of showing one but you have to configure it(?).
We are just using the site as set up by the Apache Infrastructure team. I'll file a ticket to see if they can add the clone URL there.
Here is a page with the clone URLs: https://git.apache.org/