for curiosity - what wisdom do you intend to draw from visualising relations of single gut bacteria? Or is it grains of sand in the sea? How many of them will you zoom into? Maybe clustering may make things feasible.
Ask HN: What is the best software to visualize a graph with a billion nodes?
11–20 of 117 posts
Re: Ask HN: What is the best software to visualize a graph with a billion nodes?
#12It would be a great thing for open source if someone improved the performance of dot/graphviz!
Re: Ask HN: What is the best software to visualize a graph with a billion nodes?
#13Deck.gl’s PointCloudLayer
Re: Ask HN: What is the best software to visualize a graph with a billion nodes?
#14Re: Ask HN: What is the best software to visualize a graph with a billion nodes?
#15Datashader is good for rendering large amounts of data, I'd start with that
Re: Ask HN: What is the best software to visualize a graph with a billion nodes?
#16It really feels like an under defined task. Do you actually need to see those nodes? At that scale, you never want to render 100B of them. Instead you would need some kind of density aggregation when zoomed out and moving to LoD style k-d tree partitioning when zoomed in. That's almost the area of rendering engines like Unreal's Nanite. You can create your own renderer for data like this, but game engines are likely your closest inspiration. Then again, unless you already have x/y coordinates ready, (based on graphviz I'm assuming you don't) even laying out the points will be a very heavy task. (The usual iterative force directed layout would likely take days)
But if you were my coworker I'd really press on why do you want the visualisation and if you can get your answers in some other way. And whether you can create aggregates of your data that reduces it to thousands of groups instead. Your data is a minimum of ~800GB if the graph is a single line (position + 64bit value encoding each edge, no labels), so you're not doing anything real-time with it anyway.
Re: Ask HN: What is the best software to visualize a graph with a billion nodes?
#17It would be a great thing for open source if someone improved the performance of dot/graphviz!
They're not meant for anywhere close to that scale. They deal with text descriptions and rendering full views at the time. That's never going to be usable with more than hundreds of models.
Re: Ask HN: What is the best software to visualize a graph with a billion nodes?
#18Sigma.js is pretty good at rendering a ton of nodes and edges. I haven't tried it with a billion nodes though.
https://www.sigmajs.org/
Re: Ask HN: What is the best software to visualize a graph with a billion nodes?
#19I haven’t tried that many, but GraphPU was able to render 100s of millions for me in real time.
Re: Ask HN: What is the best software to visualize a graph with a billion nodes?
#20Earlier quoted context omitted.
If it's text-heavy I'd recommend Obsidian
So the goal would be to eventually move that graph into running code and query it. But it’s never going to be large - easily fit in memory. Obsidian is PKM right? Does it have the idea of labels on the edges?
Right, in that case Obsidian might not be the best choice.
I'd look for Knowledge Graph editors then, for RDF or OWL knowledge bases, I don't have any specific recommendation, there are many but most are rather old.
Alternatively, go for a graph database like Neo4j, it's primarily designed as a database and not an editor, but it does have a nice UI visualize and change things by hand.