Live data from Hacker News

Ask HN: What is the best software to visualize a graph with a billion nodes?

news.ycombinator.com

91–100 of 117 posts

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#91
post #67

Visualizing large graphs is a natural desire for people with lots of connected data. But after a fairly small size, there's almost no utility in visualizing graphs. It's much more useful to compute various measures on the graph, and then query the graph using some combination of node/edge values and these computed values. You might subset out the nodes and edges of particular interest if you really want to see them -…

Can you recommend any good literature on the subject?

Honestly, I've been away from the field for quite a long time so wouldn't be up to date. But, if you want kind of a good framing of the field, how it evolved and how it's different from other kinds of visualization (like scientific) maybe start here [0a][0b]

0 - https://www.cs.purdue.edu/homes/xmt/classes/slides/CS530/Inf...

- https://en.wikipedia.org/wiki/Data_and_information_visualiza...

There used to be a lively research field for information visualization that studied current visualization techniques and proposed new ones to solve specific challenges -- I remember when treemaps were first introduced for example [1]. Large networks were a pretty big area of research at the time with all kinds of centrality clustering, and edge minimization techniques.

1 - https://www.google.com/search?q=treemap+visualization&tbs=im...

A few teams even tried various kind of hyperbolic representations [2,3] so that areas under local inspection were magnified under your cursor, and the rest of the hairball was pushed off to the edges of the display. But with big graphs you run into quite a few big problems very quickly like local vs. global visibility, layout challenges, etc.

2 - https://graphics.stanford.edu/papers/webviz/webviz/node2.htm...

3 - https://www.caida.org/catalog/software/walrus/

Not specifically graph related, but the best critical thinker I know of in the space is probably Edward Tufte [4]. I have some problems with a few bits of his thinking, and other than sparklines his contributions are mostly in terms of critically challenging what should be represented, why, how, and methods of interaction, his critical analysis has stayed up there as some of the best. He has a book set that's a really great collection of his thoughts.

4 - https://www.edwardtufte.com/tufte/

If you approach this problem critically, you end up at the inevitable conclusion that trying to globally visualize a massive graph in general is basically useless. Sure there are specific topologies that can be abstracted into easier to display graphs, but the general case is not conducive. It's also somewhat surprising at how small a graph can be before visualizing it gets out of hand -- maybe a few dozen nodes and edges.

I remember the U.S. DoE did some really pioneering studies in the field and produced some underappreciated experts like Thomas, Cook and Risch [5,6]. I like Risch's concepts around visualizations as formal metaphors of data. I think he's successful in defining the rigorous atomic components of visualization that you can build up from. Considering OP's request in view of Tufte and Risch, I think that they really need to think about the potential for different metaphors at different levels of detail (since they specify zooming in and out). There may not exist a single metaphor that can visualize certain data at every conceivable scope and detail!

5 - https://ils.unc.edu/courses/2017_fall/inls641_001/books/RD_A...

6 - https://arxiv.org/pdf/0809.0884v1

One interesting artifact from all of this is that most of the research has long ago been captured and commoditized or made open source. There really isn't a market anymore for commercial visualization companies, or grant money for visualization research. D3.js [7] (and the derivatives) more or less took millions upon millions of dollars in R&D and commercial research and boiled it down into a free, open source, library that captured pretty much all of the major findings in one place. It's objectively better than anything that was on the market or in labs at the time I was in the space and it's free.

7 - https://d3js.org/

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#92
Thanks to everybody who replied. I will scale my ambitions for now. How can I visualize a one billion node graph. Lets say I want to visualize transitors in a modern AI chip (around 1B nodes). My original use case was to set color on various compinents on a transistor and visualize it. For example, all flops will have one color, all buffers another color, and then I wanted to visualize their distribution on the semiconductor die.

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#93
post #67

Visualizing large graphs is a natural desire for people with lots of connected data. But after a fairly small size, there's almost no utility in visualizing graphs. It's much more useful to compute various measures on the graph, and then query the graph using some combination of node/edge values and these computed values. You might subset out the nodes and edges of particular interest if you really want to see them -…

My use case is that I have a graph of flops, latches, buffers, AND, OR, NOT gates and I want to visualize how data is changing/getting corrupted as it goes through each of them.

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#94

It really feels like an under defined task. Do you actually need to see those nodes? At that scale, you never want to render 100B of them. Instead you would need some kind of density aggregation when zoomed out and moving to LoD style k-d tree partitioning when zoomed in. That's almost the area of rendering engines like Unreal's Nanite. You can create your own renderer for data like this, but game engines are likely…

My use case is that I have a graph of flops, latches, buffers, AND, OR, NOT gates and I want to visualize how data is changing/getting corrupted as it goes through each of them.

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#96

Tacking on a related question - what software should one use to interactively create/update/see a small graph? Thinking specifically about a graph of knowledge, so will be an iterative process. Just looking for anything more than a text editor really!

dot/vimdot/graphviz promises to do it but I was not able to get it to work yet

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#98
post #67

Visualizing large graphs is a natural desire for people with lots of connected data. But after a fairly small size, there's almost no utility in visualizing graphs. It's much more useful to compute various measures on the graph, and then query the graph using some combination of node/edge values and these computed values. You might subset out the nodes and edges of particular interest if you really want to see them -…

My use case is that I have a graph of flops, latches, buffers, AND, OR, NOT gates and I want to visualize how data is changing/getting corrupted as it goes through each of them.

It's likely that a better way to do this is not to "eat the elephant", but do it at some medium-scale or subcomponent level.

It sounds like perhaps what you are trying to do is something more like this?

http://visual6502.org/

check out the visual simulations of the

http://visual6502.org/sim/varm/armgl.html

http://visual6502.org/JSSim/index.html

http://visual6502.org/JSSim/expert-6800.html

I will say (and please forgive that digital circuits are not my field), there are almost certainly better techniques and approaches in the field to accomplish what you are trying to do. I would personally move away from what you are trying and seek insight in the domain that's able to produce multi-billion transistor microprocessors.

Perhaps there are tools for large-scale logic circuit simulation?

https://old.reddit.com/r/computerscience/comments/uhappo/bes...

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#99
post #50
post #44

Earlier quoted context omitted.

This is the way imo. Nobody is consuming 100b nodes in a chart.

It really depends on what the nodes represent, right? A 1080p monitor has: 1,920 × 1,080 = 2,073,600 pixels Each pixel can display 32-bit color, which equates to: 2^32 = 4,294,967,296 colors So, while each pixel can display one of 4.3 billion colors, the monitor can display combinations of those colors across its 2,073,600 pixels. The total number of possible color combinations on the screen is astronomical. The actu…

> Each pixel can display 32-bit color

It's only 24-bits of visible colours.

Post reply on HN