Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Arrow 3.0

arrow.apache.org

Post reply on HN
Print view View original

191–200 of 204 posts

Re: Apache Arrow 3.0

#191

Post by jhgb »

wesm wrote: post #130
nomel wrote: post #111

Earlier quoted context omitted.

Until arrow has proper support for multidimensional arrays [1], its not really appropriate for many use cases. Not having first class support for multidimensional arrays, in a modern framework, really surprised me and my sensor data. [1] https://lists.apache.org/x/thread.html/9b142c1709aa37dc35f1c...

Almost no database systems support multidimensional arrays. So they are not appropriate for many use cases? * BigQuery: no * Redshift: no * Spark SQL: no * Snowflake: no * Clickhouse: no * Dremio: no * Impala: no * Presto: no ... list continues We've invited developers to add the extension types for tensor data, but no one has contributed them yet. I'm not seeing a lot of tabular data with embedded tensors out in the…

I suspect that AllegroCache accepts arrays with rank>=2, although I never got around to trying it out. (At the very least its documentation has nothing to say about any limitations on what kinds of arrays can be stored, so I'm assuming it stores all of them.)

Re: Apache Arrow 3.0

#192

Post by davidkell »

georgewfraser wrote: post #2

Arrow is the most important thing happening in the data ecosystem right now. It's going to allow you to run your choice of execution engine, on top of your choice of data store, as though they are designed to work together. It will mostly be invisible to users, the key thing that needs to happen is that all the producers and consumers of batch data need to adopt Arrow as the common interchange format. BigQuery recent…

Any Snowflake developers reading this - the current snowflake-connector-python is pinned to 0.17, almost 1 year out of date now. Would be great to get that bumped to a more recent version :-)

Re: Apache Arrow 3.0

#193

Post by imtringued »

adgjlsfhk1 wrote: post #6
mushufasa wrote: post #3

Can someone ELI5 what problems are best solved by apache arrow?

The big thing is that it is one of the first standardized, cross language binary data formats. CSV is an OK text format, but parsing it is really slow because of string escaping. The files it produces are also pretty big since it's text. Arrow is really fast to parse (up to 1000x faster than CSV), supports data compression, enough data-types to be useful, and deals with metadata well. The closest competitor is probab…

The worst part about CSV is that it appears to be so simple that many people don't even try to understand how complicated it is and just roll their own broken dialect instead of just using a library. There is also no standardized encoding. Delimiters have to be negotiated out of band and people love getting them wrong.

Re: Apache Arrow 3.0

#194

Post by zackmorris »

chenster wrote: post #116

Why no love for PHP I wonder? Don't see a supported library there.

I notice PHP compatibility conspicuously absent from so many libraries. Which is kind of amazing to me, since I "think" in PHP and MATLAB in a declarative and data-driven way. I do everything in sync blocking functional style and mostly just pipe data around. So I've found that the runners up (like Javascript, Python and Ruby) mostly just get in the way and force me to adopt their style. They're all roughly equivalent in power and expressivity, but nothing lets me go between a spreadsheet and the shell quite as easily as PHP.

Re: Apache Arrow 3.0

#195

Post by gggtt »

georgewfraser wrote: post #60
jayd16 wrote: post #30

Can someone dig into the pros and cons of the columnar aspect of Arrow? To some degree there are many other data transfer formats but this one seems to promote its columnar orientation. Things like eg. protobuffers support hierarchical data which seems like a superset of columns. Is there a benefit to a column based format? Is it an enforced simplification to ensure greater compatibility or is there some other reason…

The main benefit of a columnar representation in memory is it's more cache friendly for a typical analytical workload. For example, if I have a dataframe: (A int, B int, C int, D int) And I write: A + B In a columnar representation, all the As are next to each other, and all the Bs are next to each other, so the process of (A and B in memory) => (A and B in CPU registers) => (addition) => (A + B result back to memory…

It might be a stretch , but isn't it kind of the same idea as why ECS is faster than OO in video games development ?

Re: Apache Arrow 3.0

#196

Post by juntan »

nevi-me wrote: post #74
liminal wrote: post #64

Would really love to see first class support for Javascript/Typescript for data visualization purposes. The columnar format would naturally lend itself to an Entity-Component style architecture with TypedArrays.

Have you seen https://github.com/finos/perspective ? Though they removed the JS library a few months ago in favour of a WASM build of the C++ library.

The JS library hasn't been removed - it's the main UI and API over the WASM library that allows for Perspective to be used in the browser: https://perspective.finos.org/

Re: Apache Arrow 3.0

#197

Post by IshKebab »

BadInformatics wrote: post #90
IshKebab wrote: post #78

Earlier quoted context omitted.

Yeah I do. For example Apache Arrow supports in memory compression. But Parquet does not support that. I had to look through the code to find that out, and I found many instances of basically `throw "Not supported"`. And yeah as I said the documentation is just non-existent. If you are already using Arrow, or you absolutely must use a columnar file format then it's probably a good option.

Is that a problem in the Parquet format or in PyArrow? My understanding is that Parquet is primarily meant for on-disk storage (hence the default on-disk compression), so you'd read into Arrow for in-memory compression or IPC.

I don't know whether it is a limitation of the format or of the implementation. I used the C++ library (or maybe it was C, I can't remember), which I assume PyArrow uses too.

> hence the default on-disk compression

No, Parquet doesn't support some compression formats that Arrow does.

Re: Apache Arrow 3.0

#198

Post by lmeyerov »

aldanor wrote: post #159
wesm wrote: post #130

Earlier quoted context omitted.

Almost no database systems support multidimensional arrays. So they are not appropriate for many use cases? * BigQuery: no * Redshift: no * Spark SQL: no * Snowflake: no * Clickhouse: no * Dremio: no * Impala: no * Presto: no ... list continues We've invited developers to add the extension types for tensor data, but no one has contributed them yet. I'm not seeing a lot of tabular data with embedded tensors out in the…

I think that implementing good ndim=2 support would already be a huge leap forward, it doesn't have to be something super generic. E.g., given that most of the classic machine learning is essentially using 2-dimensional data (samples x features) as inputs, this is a very common use case. E.g., as of right now, having to concatenate hundreds of columns manually just in order to pass them to some ml library in a contig…

This may help you do zero copy for a column of multi-dim without losing value types, just that it's encoding a multi-dim. This example is for values that are 3x3 of int8's:

```

import pyarrow as pa

my_col_of_3x3s = pa.struct([ (f'f_{x}_{y}', pa.int8()) for x in range(3) for y in range(3) ])

```

If using ndarrays, I think our helpers are another ~4 lines each. Interop with C is even easier, just cast. You can now pass this data through any Arrow-compatible compute stack / DB and not lose the value types. We do this for streaming into webgl's packed formats, for example.

What you don't get is a hint to the downstream systems that it is multidimensional. Tableau would just let you do individual bar charts, not say a heatmap, assuming they support rank 2's. To convert, you'd need to do that zero-copy cast to whatever they do support. I agree a targetable standard would avoid the need for that manual conversion, and increases the likelihood they use the same data rep.

Native support would also avoid some header bloat from using structs. However, we find that's fine in practice, it's metadata. E.g., our streaming code reads the schema at the beginning and then passes it along, so actual payloads are pure data, and skip resending metadata.

Re: Apache Arrow 3.0

#199

Post by infinite8s »

mushufasa wrote: post #121
neovintage wrote: post #7

Earlier quoted context omitted.

The premise around arrow is that when you want share data with another system, or even on the same machine between processes, most of the compute time spent is in serializing and deserializing data. Arrow removes that step by defining a common columnar format that can be used in many different programming languages. Theres more to arrow than just the file format that makes working with data even easier like better ov…

is there a strong use case around passing data from a backend to a frontend, e.g. from pandas data frame on the server into a js implementation on the client side, to use in a UI? As opposed to data pipelining among processing servers.

Yes definitely. For example, the perspective pivoting engine (https://github.com/finos/perspective) supports arrow natively, so you can stream arrow buffers from a big data system directly to be manipulated in your browser (treating the browser as a thick client application). It's a single copy from network buffer into the webassembly heap to get it into the library.

Re: Apache Arrow 3.0

#200

Post by infinite8s »

juntan wrote: post #196
nevi-me wrote: post #74

Earlier quoted context omitted.

Have you seen https://github.com/finos/perspective ? Though they removed the JS library a few months ago in favour of a WASM build of the C++ library.

The JS library hasn't been removed - it's the main UI and API over the WASM library that allows for Perspective to be used in the browser: https://perspective.finos.org/

I think the GP meant the Typescript arrow library.
Post reply on HN