Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is the best software to visualize a graph with a billion nodes?

news.ycombinator.com

Post reply on HN
Print view View original

101–110 of 117 posts

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#101

Post by bjourne »

throwaway425933 wrote: post #93
bane wrote: post #67

Visualizing large graphs is a natural desire for people with lots of connected data. But after a fairly small size, there's almost no utility in visualizing graphs. It's much more useful to compute various measures on the graph, and then query the graph using some combination of node/edge values and these computed values. You might subset out the nodes and edges of particular interest if you really want to see them -…

My use case is that I have a graph of flops, latches, buffers, AND, OR, NOT gates and I want to visualize how data is changing/getting corrupted as it goes through each of them.

I did that a few years go. It was a nice visualization up until restoring division. With more components than that the layout just becomes to cluttered to be meaningful. And it is very difficult to encode all heuristics one uses when drawing "pretty" circuit diagrams by hand into an algorithm.

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#102

Post by viraptor »

throwaway425933 wrote: post #94
viraptor wrote: post #16

It really feels like an under defined task. Do you actually need to see those nodes? At that scale, you never want to render 100B of them. Instead you would need some kind of density aggregation when zoomed out and moving to LoD style k-d tree partitioning when zoomed in. That's almost the area of rendering engines like Unreal's Nanite. You can create your own renderer for data like this, but game engines are likely…

My use case is that I have a graph of flops, latches, buffers, AND, OR, NOT gates and I want to visualize how data is changing/getting corrupted as it goes through each of them.

Ok, so you have nice natural boundaries between systems. If you're dealing with something processor-like, you have really good chokepoints where for example ALU / register / caches connect. The task may be way easier if you deal with one of them at a time. Maybe even abstract anything less interesting (memory/cache?) Would visualising things per-system work better for you, or maybe visualising separate systems getting affected instead of specific nodes?

Having the structure of the device available should also help with the layout - we know you can group the nodes logically into independent boxes instead of trying to auto-layout everything.

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#103

Post by samstave »

throwaway425933 wrote: post #95
snickerd00dle wrote: post #74

At that size what you're actually looking for is a game engine with a particle system.

Wow. I had not thought of that. While idea makes sense, even MVP for this is going to be daunting

Sent you email with some details on volumetric particle cloud in blender and pythons to snake a harness to that mvp using cuda/other gpu signal tools...

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#104

Post by seinecle »

_flux wrote: post #38

I'm also looking for a graph viewing tool, but my wishlist is different (not all of them are hard requirements): - Deal with 100k node graphs, preferably larger - Interactive filtering tools, e.g. filtering by node or edge data, transitive closures, highlighting paths matching a condition. Preferably filtering would result in minimally re-layouting the graph. - Does not need an very sophisticated layout algorithms, i…

Have you tried Gephi?

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#105

Post by seinecle »

oersted wrote: post #7

It is somewhat old-school, but Gephi is by far the best graph visualization tool I've used that stays robust and usable at such scales (at least ~10M, but possibly a lot more).

And work is underway to help Gephi handle larger graphs:

https://gephi.wordpress.com/2024/06/13/gephi-week-2024-peek-...

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#106

Post by withinboredom »

A couple of years ago, I had a similar issue. I don't have the code any more, but my output to a 3d model converter in a weekend and then threw it into unreal. Then I put on my VR goggles and walked around the graph. It was much easier to deal with in three dimensions instead of two.

From there I could write better visualizations. I got laid off before the project was completed, though.

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#107

Post by _flux »

seinecle wrote: post #104
_flux wrote: post #38

I'm also looking for a graph viewing tool, but my wishlist is different (not all of them are hard requirements): - Deal with 100k node graphs, preferably larger - Interactive filtering tools, e.g. filtering by node or edge data, transitive closures, highlighting paths matching a condition. Preferably filtering would result in minimally re-layouting the graph. - Does not need an very sophisticated layout algorithms, i…

Have you tried Gephi?

I have not!

So I gave it a try and it seems quite a capable tool. It doesn't check all my boxes and is more cumbersome to use than I'd wish, e.g. it is able to find shortest path between nodes, but activating that needs finding and entering the node ids manually.

However, this still seems the best tool I've ever seen for this purpose and it's also highly general. Thanks!

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#109

Post by egberts1 »

I was working on a node graph for nftables bison parser.

A blog that covers only failures of large SVG viewers having 10,000+ of nodes.

https://egbert.net/blog/articles/comparison-svg-viewers-larg...

More on https://egbert.net/blog/tags/graphviz.html

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#110

Post by trueismywork »

simonsarris wrote: post #47

As someone who's made graphing libraries for over a decade: Are you sure you want to visualize 1 billion nodes? What's the essential thing you're trying to see? Visualizations are great at helping humans parse data, but usually they work best at human scales. A billion nodes is at best looking at clouds, rather than nodes, which can be represented otherwise.

I have a slightly different use case. I have a dependency graph of tasks with each task having some attached info in form key value pairs. I want to be able to easily visualize the complete graph but then filter out stuff using conditions on attached info. The filtering should hide there nodes/tasks in graph and auto-resize display. What would be a good solution to this?
Post reply on HN