Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Algorithms on billion-scale graph using 10GB RAM: I love DataFusion

semyonsinchenko.github.io

Post reply on HN
Print view View original

21–30 of 42 posts

Re: Algorithms on billion-scale graph using 10GB RAM: I love DataFusion

#21

Post by lmeyerov »

adsharma wrote: post #15

The idea of graph algorithms on Apache arrow at scale originated here. 100+ graph algorithms running on columnar memory. https://github.com/Ladybug-Memory/icebug Out of core with datafusion is the main innovation here in graphframes-rs. But it has only 2 algorithms so far. Icebug and LadybugDB can be tightly integrated to efficiently move tables encoded as compressed sparse row (CSR) into arrow memory. Jupyter notebo…

Not really ;-)

~10 years ago, we helped create apache arrow, helped create GPU data frames, and been running for the last decade the open source pygraphistry and now gfql cpu+gpu property graph engine for this. Likewise, Nvidia has been doing great with cuGraph (OSS) for GPU algs around this.

It is great you are finding success with this direction, but "shoulders of giants" merit credit - 100+ people.

Re: Algorithms on billion-scale graph using 10GB RAM: I love DataFusion

#22

Post by lmeyerov »

ratmice wrote: post #8

It would be nice if OP noted what caused the change in their opinion? did datafusion gain some feature that they noted was missing in the previous article, or did something in their understanding click so they could overcome the previous issues?

We shared with the author how databricks multi-node and single-node graphframes were wildly inefficient for this kind of thing: we were measuring doing billion-edge graph traversals & scans in single node in-memory in seconds with regular dataframe (cudf) libraries, so the core of pagerank, which is magnitudes more efficient than their original spark approach

So then the question became pandas/polars/datafusion/duckdb/etc, must of which are rust/native. I'm curious myself why datafusion vs others, it's an interesting project :)

Re: Algorithms on billion-scale graph using 10GB RAM: I love DataFusion

#23

Post by BigTTYGothGF »

cpdomina wrote: post #14

cool! you might be interested in graphchi (2012), also designed to do large scale graph operations on a single machine https://github.com/GraphChi/graphchi-cpp#performance

I got to use graphchi/graphlab back in the day (I think I still have the tshirt from the 2013 conference), wish it had caught on more.

Re: Algorithms on billion-scale graph using 10GB RAM: I love DataFusion

#24

Post by adsharma »

lmeyerov wrote: post #21
adsharma wrote: post #15

The idea of graph algorithms on Apache arrow at scale originated here. 100+ graph algorithms running on columnar memory. https://github.com/Ladybug-Memory/icebug Out of core with datafusion is the main innovation here in graphframes-rs. But it has only 2 algorithms so far. Icebug and LadybugDB can be tightly integrated to efficiently move tables encoded as compressed sparse row (CSR) into arrow memory. Jupyter notebo…

Not really ;-) ~10 years ago, we helped create apache arrow, helped create GPU data frames, and been running for the last decade the open source pygraphistry and now gfql cpu+gpu property graph engine for this. Likewise, Nvidia has been doing great with cuGraph (OSS) for GPU algs around this. It is great you are finding success with this direction, but "shoulders of giants" merit credit - 100+ people.

Ofc. I'm not claiming credit for Apache Arrow. Lots of people contributed to it.

I didn't write the graph algorithms in networkit. Somebody else did.

But if you're looking for 100-200 graph algorithms running efficiently on columnar memory, I haven't found a working implementation that's permissively open source.

Happy to work with cugraph or anyone else who builds on top of Parquet/Arrow ecosystem.

Re: Algorithms on billion-scale graph using 10GB RAM: I love DataFusion

#25

Post by lmeyerov »

adsharma wrote: post #24
lmeyerov wrote: post #21

Earlier quoted context omitted.

Not really ;-) ~10 years ago, we helped create apache arrow, helped create GPU data frames, and been running for the last decade the open source pygraphistry and now gfql cpu+gpu property graph engine for this. Likewise, Nvidia has been doing great with cuGraph (OSS) for GPU algs around this. It is great you are finding success with this direction, but "shoulders of giants" merit credit - 100+ people.

Ofc. I'm not claiming credit for Apache Arrow. Lots of people contributed to it. I didn't write the graph algorithms in networkit. Somebody else did. But if you're looking for 100-200 graph algorithms running efficiently on columnar memory, I haven't found a working implementation that's permissively open source. Happy to work with cugraph or anyone else who builds on top of Parquet/Arrow ecosystem.

cudf/cugraph is open source and helped fund the arrow ecosystem , I think it's good to give teams credit when they do work they didn't have to do and not misrepresent them. And yes, it works great, graph500 leaderboard level.

Re: Algorithms on billion-scale graph using 10GB RAM: I love DataFusion

#26

Post by adsharma »

lmeyerov wrote: post #25
adsharma wrote: post #24

Earlier quoted context omitted.

Ofc. I'm not claiming credit for Apache Arrow. Lots of people contributed to it. I didn't write the graph algorithms in networkit. Somebody else did. But if you're looking for 100-200 graph algorithms running efficiently on columnar memory, I haven't found a working implementation that's permissively open source. Happy to work with cugraph or anyone else who builds on top of Parquet/Arrow ecosystem.

cudf/cugraph is open source and helped fund the arrow ecosystem , I think it's good to give teams credit when they do work they didn't have to do and not misrepresent them. And yes, it works great, graph500 leaderboard level.

I'm happy to credit these people.

https://arrow.apache.org/docs/r/authors.html

CUDA and the ecosystem around it is more complicated.

I'll use it. I agree that it advanced the state of the art and helped fund some of the truly OSS projects.

Don't feel the need to bring it up on a HN comment. Neither does the parent article by Sem.

In fact, it's in the vendor's interest to transparently route the algorithms in Icebug to the GPU in a compatible way instead of writing their own APIs.

Re: Algorithms on billion-scale graph using 10GB RAM: I love DataFusion

#28

Post by adsharma »

adsharma wrote: post #26
lmeyerov wrote: post #25

Earlier quoted context omitted.

cudf/cugraph is open source and helped fund the arrow ecosystem , I think it's good to give teams credit when they do work they didn't have to do and not misrepresent them. And yes, it works great, graph500 leaderboard level.

I'm happy to credit these people. https://arrow.apache.org/docs/r/authors.html CUDA and the ecosystem around it is more complicated. I'll use it. I agree that it advanced the state of the art and helped fund some of the truly OSS projects. Don't feel the need to bring it up on a HN comment. Neither does the parent article by Sem. In fact, it's in the vendor's interest to transparently route the algorithms in Icebug t…

They use networkx today: https://github.com/rapidsai/nx-cugraph/ because it's more popular than networkit or icebug. But its popularity is based on ease of use, not performance:

https://github.com/timlrx/graph-benchmarks

Not clear if the author or anyone else has an updated version of these benchmarks. Opened an icebug issue on the repo.

Re: Algorithms on billion-scale graph using 10GB RAM: I love DataFusion

#29

Post by lmeyerov »

adsharma wrote: post #26
lmeyerov wrote: post #25

Earlier quoted context omitted.

cudf/cugraph is open source and helped fund the arrow ecosystem , I think it's good to give teams credit when they do work they didn't have to do and not misrepresent them. And yes, it works great, graph500 leaderboard level.

I'm happy to credit these people. https://arrow.apache.org/docs/r/authors.html CUDA and the ecosystem around it is more complicated. I'll use it. I agree that it advanced the state of the art and helped fund some of the truly OSS projects. Don't feel the need to bring it up on a HN comment. Neither does the parent article by Sem. In fact, it's in the vendor's interest to transparently route the algorithms in Icebug t…

cudf+cugraph are the table+graph algorithms that started around that same time to align with it, that's what helped get arrow funded from the nvidia side

can't reply on the below, but re:networkx, it was more of the reverse, they did a nice job of building a standalone embeddable table+graph arrow-friendly library over many years, and with networkx compatibility from the beginning. Later, they collaborated with the networkx team to eventually upstream it so networkx users can benefit more easily

as it is embeddable, you don't need to go through networkx to use it -- eg, we use it directly, as do various databases

and I'm not sure why you're saying people do not use networkx->cugraph because of performance, again, this is graph500 level performance that even graph database vendors now support because it is literally magnitudes faster than their cpu alternatives. We have had projects like court cases where the data science team switched to sitting on top of pygraphistry/gfql -> cudf/cugraph and using the GPU versions were the difference between hours and minutes, which for iterating over interactive analysis, is important. It's night and day going from CPU -> GPU, and a lot of butts were saved because of this.

We see the same thing with databricks+neo4j migrations.. most tools have their sweet spots, but also their comparative weak spots.

Re: Algorithms on billion-scale graph using 10GB RAM: I love DataFusion

#30

Post by adsharma »

ratmice wrote: post #8

It would be nice if OP noted what caused the change in their opinion? did datafusion gain some feature that they noted was missing in the previous article, or did something in their understanding click so they could overcome the previous issues?

There is a bit of healthy competition going on between graphframes and icebug. See:

https://github.com/Ladybug-Memory/icebug-graphframes-compari...

I thought of datafusion work and the rust implementation as a way to address the delta seen vs the Spark/JVM GraphFrames implementation (author is a major contributor).

Looking forward to more such innovations, which will benefit the ecosystem as a whole. Why would anyone want to use a pure python graph algorithm package unless they're dealing with toy graphs?

Post reply on HN