I'm done with Hacker News, you guys just upvote marketing and politics.
Apache Arrow 3.0
101–110 of 204 posts
Re: Apache Arrow 3.0
#102So if I understand this correctly from an application developers perspective: - for OLTP tasks, something row based like sqlite is great. Small to medium amounts of data mixed reading/writing with transactions - for OLAP tasks, arrow looks great. Big amounts of data, faster querying (datafusion) and more compact data files with parquet. Basically prevent the operational database from growing too large, offload older…
I think the best way to think about it is, you have teams where hadoop queries take 5+ hours to run. You port that same data into arrow, now that same query takes 30 seconds.
This is something I understood, but I ask specifically as a person at the intersection point between those worlds, owning the operational database and want to see if I can enhance systems for both, speedy operations _and_ seamless analytics using arrow.
Re: Apache Arrow 3.0
#103has anyone had success using arrow in js to feed tabular data to the frontend from the backend, such as a pandas data frame?
Re: Apache Arrow 3.0
#104Earlier quoted context omitted.
I challenge you to have a closer look at the project. Deserialization by definition requires bytes or bits to be relocated from their position in the wire protocol to other data structures which are used for processing. Arrow does not require any bytes or bits to be relocated. So if a "C array of doubles" is not native to the CPU, then I don't know what is.
CPUs come in many flavors. One area where they differ is in the way that bytes of a word are represented in memory. Two common formats are Big Endian and Little Endian. This is an example where a "C array of doubles" would be incompatible and some form of deserilaziation would be needed. My understanding is that an apache arrow library provides an API to manipulate the format in a platform agnostic way. But to claim…
Big endian is pretty rare among anything you’d be doing in-memory analytics on. Looks like you can choose the endianess of the format if you need to, but it’s little endian by default: https://arrow.apache.org/docs/format/Columnar.html. I’d suggest reading up on the format, it covers the properties it provides to be friendly to direct random access.
Re: Apache Arrow 3.0
#105Earlier quoted context omitted.
The premise around arrow is that when you want share data with another system, or even on the same machine between processes, most of the compute time spent is in serializing and deserializing data. Arrow removes that step by defining a common columnar format that can be used in many different programming languages. Theres more to arrow than just the file format that makes working with data even easier like better ov…
> most of the compute time spent is in serializing and deserializing data. This is to be viewed in light how hardware evolves now. CPU compute power is no longer growing as much (at least for individual cores). But one thing that's still doubling on a regular basis is memory capacity of all kinds (RAM, SSD, etc) and bandwidth of all kinds (PCIe lanes, networking, etc). This divide is getting large and will only conti…
> What you want is to have the CPU coordinate the SSD to copy chunks directly -and as is- to the NIC/app/etc.
Isn't that what DMA is supposed to be?
Also, there's work in getting GPUs to load data straight from NVME drives, bypassing both the CPU and system memory. So you could certainly do similar things with the PCIE bus.
https://developer.nvidia.com/blog/gpudirect-storage/
A big problem is that a lot of data isn't laid out in a way that's ready to be stuffed in memory. When you see a game spending a long time loading data, that's usually why. The CPU will do a bunch of processing to map on disk data structures to a more efficient memory representation.
If you can improve the on-disk representation to more closely match what's in memory, then CPUs are generally more than fast enough to copy bytes around. They are definitely faster than system RAM.
Re: Apache Arrow 3.0
#106Arrow is the most important thing happening in the data ecosystem right now. It's going to allow you to run your choice of execution engine, on top of your choice of data store, as though they are designed to work together. It will mostly be invisible to users, the key thing that needs to happen is that all the producers and consumers of batch data need to adopt Arrow as the common interchange format. BigQuery recent…
Are these execution engines internally using Arrow columnar format or are they just exposing Arrow as a client wire format? AFAIK Spark and Presto does not use Arrow as execution columnar format, but just data sources/sinks.
Re: Apache Arrow 3.0
#107Uh... 404 error for the link....
Re: Apache Arrow 3.0
#108Has anyone had success in getting the page to load? I'm on my desktop and can't see what's behind the link.
Re: Apache Arrow 3.0
#109Excited to see this release's official inclusion of the pure Julia Arrow implementation [1]! It's so cool to be able mmap Arrow memory and natively manipulate it from within Julia with virtually no performance overhead. Since the Julia compiler can specialize on the layout of Arrow-backed types at runtime (just as it can with any other type), the notion of needing to build/work with a separate "compiler for fast UDFs…
> a separate "compiler for fast UDFs" is rendered obsolete
Agreed. I am excited too. Thanks Jacob!
Re: Apache Arrow 3.0
#110Excited to see this release's official inclusion of the pure Julia Arrow implementation [1]! It's so cool to be able mmap Arrow memory and natively manipulate it from within Julia with virtually no performance overhead. Since the Julia compiler can specialize on the layout of Arrow-backed types at runtime (just as it can with any other type), the notion of needing to build/work with a separate "compiler for fast UDFs…
Especially impressive that the first official version ships with such broad feature coverage! Really great work by Jacob.