Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is the best software to visualize a graph with a billion nodes?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 117 posts

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#32

Post by heresie-dabord »

shoo wrote: post #5

what decision / downstream process is going to consume the 1B node graph render? is producing a render really necessary for that decision, or is rendering the graph waste? is there a way you can subsample or simplify or approximate the graph that'd be good enough? in some domains, certain problems that are defined on graphs can be simplified by pre-processing the graph, to reduce the problem to a simpler problem. e.g…

> is producing a render really necessary for that decision, or is rendering the graph waste?

Just to be clear, the OP already has a graph. There are nodes and relationships. The graph can be queried for understanding.

Rendering the graph is tractable for a small graph or a portion of the graph.

Trying to render all the nodes in an enormous graph is almost always an expensive quixotic adventure.

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#37

Post by michaelt »

You can visualise a graph with 9 billion nodes on https://www.openstreetmap.org :)

You could copy their design, if you know how you want to project your nodes into 2D. Essentially dividing the visualisation into a very large number of tiles, generated at 18 different zoom levels, then the 'slippy map' viewer loads the tiles corresponding to the chosen field of view.

Then a PostGIS database alongside, letting you run a query to get all the nodes in a given rectangle - such as if you want to find the ID number of a given node.

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#38

Post by _flux »

I'm also looking for a graph viewing tool, but my wishlist is different (not all of them are hard requirements):

- Deal with 100k node graphs, preferably larger

- Interactive filtering tools, e.g. filtering by node or edge data, transitive closures, highlighting paths matching a condition. Preferably filtering would result in minimally re-layouting the graph.

- Does not need an very sophisticated layout algorithms, if hiding or unranking nodes interactively is easy. E.g. centering on a node could layout other nodes using the selected node as the root.

- Ability to feed live data externally, add/remove nodes and edges programmatically

- Clusters (nodes would tell which clusters they belong in)

I'm actually thinking of writing that tool some day, but it would of course be nicer if it already existed ;). I'm thinking applications like studying TLA+ state traces, visualizing messaging graphs or debug data in real time, visualizing the dynamic state of a network.

Also if you have tips on applicable Rust crates to help creating that, those are appreciated!

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#39

Post by simpaticoder »

Hilbert curves (or similar) are often used for graphing billions of nodes[1]. However this will not by default show the relationships between nodes in a graph. Depending on your data you may be able to write a function to map from your edge list to a node index that hints at proximity.

Note that visualizations are limited by human perception to ~10000 elements, more usefully 1000 elements. You might try a force directed graph, perhaps a hierarchical variant wherein nodes can contain sub-graphs. Unless you have obvious root nodes, this variant would be interesting in that the user could start from an arbitrary set of nodes, giving different insights depending on their starting point.

1 - An excerpt from "Harder Drive", a rather silly implementation of a unix block device using ping latency with any host that will let him. He visualizes the full ipv4 address space in a hilbert curve at this offset: https://youtu.be/JcJSW7Rprio?si=0AlyMgaZjH7dmh5y&t=363

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#40

Post by mhx1138 »

_flux wrote: post #38

I'm also looking for a graph viewing tool, but my wishlist is different (not all of them are hard requirements): - Deal with 100k node graphs, preferably larger - Interactive filtering tools, e.g. filtering by node or edge data, transitive closures, highlighting paths matching a condition. Preferably filtering would result in minimally re-layouting the graph. - Does not need an very sophisticated layout algorithms, i…

Neo4j has an interactive graph browser built in.
Post reply on HN