Algorithms on billion-scale graph using 10GB RAM: I love DataFusion
semyonsinchenko.github.io
Algorithms on billion-scale graph using 10GB RAM: I love DataFusion
1–10 of 42 posts
Re: Algorithms on billion-scale graph using 10GB RAM: I love DataFusion
#2Really cool visualization, amazing how it resembles a neural network.
Re: Algorithms on billion-scale graph using 10GB RAM: I love DataFusion
#3> "I can compute PageRank on a directed graph with one billion edges (graph500-26 from the Graphalytics dataset) using 5 GB of memory. Alternatively, I can identify all the weakly connected components in a graph with two billion edges (twitter_mpi from the same dataset collection) using 10 GB of memory. Neither NetworkX nor Igraph can do this; most existing graph algorithms require the graph to fit into memory. Previously, I thought you needed Apache Spark and GraphFrames for billion-scale graph analytics. Now, however, I think all you need is a laptop. I have completely changed my old opinion about using Apache DataFusion for graph analytics."
Impressive!
Re: Algorithms on billion-scale graph using 10GB RAM: I love DataFusion
#4Does it support out-of-core or multi-processor processing?
Re: Algorithms on billion-scale graph using 10GB RAM: I love DataFusion
#5[dead]
Re: Algorithms on billion-scale graph using 10GB RAM: I love DataFusion
#6DataFusion is really cool, it's kind of like the LLVM of the OLAP world.
Re: Algorithms on billion-scale graph using 10GB RAM: I love DataFusion
#7> "I can compute PageRank on a directed graph with one billion edges (graph500-26 from the Graphalytics dataset) using 5 GB of memory. Alternatively, I can identify all the weakly connected components in a graph with two billion edges (twitter_mpi from the same dataset collection) using 10 GB of memory. Neither NetworkX nor Igraph can do this; most existing graph algorithms require the graph to fit into memory. Previ…
> most existing graph algorithms require the graph to fit into memory.
You can get pretty far with sparse graphs, which are just arrays, in combination with memory mapping.
Re: Algorithms on billion-scale graph using 10GB RAM: I love DataFusion
#8It would be nice if OP noted what caused the change in their opinion?
did datafusion gain some feature that they noted was missing in the previous article, or did something in their understanding click so they could overcome the previous issues?
Re: Algorithms on billion-scale graph using 10GB RAM: I love DataFusion
#93.4 gb dataset on 10gb ram
Re: Algorithms on billion-scale graph using 10GB RAM: I love DataFusion
#10Hello, I am new to hacker news and finding it really resourceful. I found this article interesting (having learnt KG and Map Reduce (spark) as part of my masters' course), appreciate the effort to post this.
I am here to seek guidance from the community. I want to refresh my memory on knowledge graphs and algorithms for Big Data Mining and Processing.
I believe KG can solve problems on Agent attacks (LLM agency) in real-time - so want to build knowledge around the topic.
Interested to join any interest/ discussion groups if any. Thanks!