Live data from Hacker News

Algorithms on billion-scale graph using 10GB RAM: I love DataFusion

semyonsinchenko.github.io

41–42 of 42 posts

Re: Algorithms on billion-scale graph using 10GB RAM: I love DataFusion

#41
post #16

It's hard to take the article seriously when it has quotes like this: "The hardest part. 2B edges twitter graph is already huge (its edges are 30 GB in CSV !!!)." Who cares how big the graph is in CSV? That's not the representation you operate over in big data. All of this would have easily fit in memory on any reasonable modern system.

> All of this would have easily fit in memory on any reasonable modern system.

Understand me correctly. This is my research project and I only have a laptop, not a server with 256 GB of RAM. I tested my project on a 2B graph with a hard cap of 8–10 GB of RAM. Of course it fits in memory on any modern system with 64–128 GB of RAM. The whole idea was to conduct a stress test and check how my tool works in out-of-core mode, not to prove to anyone that 2 billion edges (30 GB CSV) constitutes "big data".

Re: Algorithms on billion-scale graph using 10GB RAM: I love DataFusion

#42

Datafusion is undoubtedly one of the best open source projects of all time, it's so incredibly powerful and well designed. The extensibility is insane, you can create your own query language that compiles to logical plans.

I agree 100%! DataFusion is beautiful and easy to extend in any direction. For the second version of my "out-of-core" graph algorithms project, for example, I implemented my own "co-partitioning" to speed up joins and achieved a performance improvement of two times! It was also easy to modify the physical plan and declare partitioning.

Co-partitioning of range-partitioned data is now supported on `main` too: see https://github.com/apache/datafusion/issues/23183 and the epic https://github.com/apache/datafusion/issues/22395
Post reply on HN