Live data from Hacker News

Ask HN: What is the best software to visualize a graph with a billion nodes?

news.ycombinator.com

81–90 of 117 posts

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#81
post #76

As many people already commented, no one actually visualizes graphs of that size at once. Context: I’m the CTO of a GraphViz company, I’ve been doing this for 10+ years. Here are my recommendations: - if you can generate a projection of your graph into millions of nodes, you might be able to get somewhere with Three.js, which is a JS library to generate WebGL graphics. The library is close enough to the metal to allo…

Just wanted to say, while I'll never use Ogma, it's really fun to play w/. Performant too.

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#82

It really feels like an under defined task. Do you actually need to see those nodes? At that scale, you never want to render 100B of them. Instead you would need some kind of density aggregation when zoomed out and moving to LoD style k-d tree partitioning when zoomed in. That's almost the area of rendering engines like Unreal's Nanite. You can create your own renderer for data like this, but game engines are likely…

there's 8 million pixels in 4k, so if you're trying to graph 8 million points, might as well just fill up the screen with a single color and call it a day. If you have 8 billion, well you can graph about 0.1% of that and fill up every single pixel of the screen, but then you're just looking at noise. To be able to show connections between nodes, you'd need maybe 9 pixels per node, so that's around 900k nodes you might be able to graph on a 4k screen, assuming a maximum number of connections between nodes is 8, and the connected nodes are adjacent. So now you're at about 0.01% can be graphed on yor display, and that's not even very usable and there'd not be a lot of information you could glean from that. You could go to 81 pixels per node and you'd be able to connect more nodes to a graph, and maybe you could make some sense of it that way, but you'd only be graphing 0.001% and at that point, what's your selection criteria? Your selection criteria for nodes would have more of an impact than how you choose to graph it.

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#83
post #63

Earlier quoted context omitted.

> Don't just blithely give them what they asked for. Depends on how much they're paying you for it.

Just in case you are not being sarcastic: there is a thing called ethics, which is the basis of human relations.

every man has a price though. whether that's a big cheque or a gun to your child's head.

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#84
post #67

Visualizing large graphs is a natural desire for people with lots of connected data. But after a fairly small size, there's almost no utility in visualizing graphs. It's much more useful to compute various measures on the graph, and then query the graph using some combination of node/edge values and these computed values. You might subset out the nodes and edges of particular interest if you really want to see them -…

> But after a fairly small size, there's almost no utility in visualizing graphs.

I want to stress this point and go a bit further. It can be worse as people have pareidolia[0], a tendency to see order in disorder. Like how you see familiar shapes in the clouds. There is a danger in that with large visualizations such as these that instead of conveying useful information, you counterproductively convince someone that something that isn't true is! Here's a relevant 3B1B video where this is kinda discussed. There is real meaning but the point is that it is also easy to be convinced of things that aren't true[1]. In fact, Grant's teaching style is so good you might even convince yourself of the thing he is disproving as he is revealing how we are tricked by the visualization. Remember what the original SO person latched onto.

I think it's important to recognize that visualization is a nontrivial exercise. Grant makes an important point at the end, restating how the visualization was an artifact and how if you dig deep enough into an arbitrary question, you _can_ find value. Because at the end of the day, there are rules to these things. The same is true about graphs. There will always be value in the graph, but the point of graphing is to highlight the concepts that we want to convey. In a way, many people interpret what graphs are doing and why we use them backwards. You don't create visualizations to then draw value from them, but rather your plots are a mathematical analysis that is in a more natural language for humans. This is subtle and might be confusing because people often are able to intuit what kind of graph should be used to convey data but are not thinking about what the process is doing. So what I'm saying is that you don't want to use arbitrary graphs, but there's the right graph for the job. You can find a lot of blogs on graph sins[2] and this point will become clearer.

At the heart, this is not so different than "lies, damned lies, and statistics." People often lie with data without stating anything that is untrue. With graphs, you can lie without stating a word, despite being worth a thousand. So the most important part of being a data scientist is not lying to yourself (which sounds harder than it is).

[0] https://en.wikipedia.org/wiki/Pareidolia

[1] https://www.youtube.com/watch?v=EK32jo7i5LQ

[2] Except this might be hard because if you Google this you'll have a hard time convincing google that you don't mean "sine". So instead search "graph deadly sins", "data visualization sins", "data is ugly", and so on. I'll pass you specifically the blog of "Dr. Moron" (Kennith Moreland) and one discussion of bad plots https://www.drmoron.org/posts/better-plots/ (Ken is a data visualization expert and both his blogs have a lot on vis). There's also vislies: https://www.vislies.org/2021/

(Source: started my PhD in viz and still have close friends in infoviz and sciviz who I get to hear their rants about their research, and occasionally I contribute)

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#85
post #67

Visualizing large graphs is a natural desire for people with lots of connected data. But after a fairly small size, there's almost no utility in visualizing graphs. It's much more useful to compute various measures on the graph, and then query the graph using some combination of node/edge values and these computed values. You might subset out the nodes and edges of particular interest if you really want to see them -…

I am pretty sour about it and will call out people who post "just another hairball" and act like they've done something special. I think there is a need for a tool that can extract and tell an interesting story based on a subgraph of a huge graph, but that takes thinking unlike hairball plotting, ai image generation and other seductive scourges. I went to an posthumous art show based on this guy https://www.amazon.co…

> Funny that might have something to do with his mysterous death... Maybe a tool that would help you do that is too dangerous for "them" to let you have!

This may be one of the most rediculous conspiracy theories i have ever heard. Big data (heh) had him killed... ok.

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#86
post #67

Visualizing large graphs is a natural desire for people with lots of connected data. But after a fairly small size, there's almost no utility in visualizing graphs. It's much more useful to compute various measures on the graph, and then query the graph using some combination of node/edge values and these computed values. You might subset out the nodes and edges of particular interest if you really want to see them -…

Can you recommend any good literature on the subject?

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#87
post #67

Visualizing large graphs is a natural desire for people with lots of connected data. But after a fairly small size, there's almost no utility in visualizing graphs. It's much more useful to compute various measures on the graph, and then query the graph using some combination of node/edge values and these computed values. You might subset out the nodes and edges of particular interest if you really want to see them -…

I am pretty sour about it and will call out people who post "just another hairball" and act like they've done something special. I think there is a need for a tool that can extract and tell an interesting story based on a subgraph of a huge graph, but that takes thinking unlike hairball plotting, ai image generation and other seductive scourges. I went to an posthumous art show based on this guy https://www.amazon.co…

I think you answer you unknowingly answered your own question. The reason no such tool exists is that this stuff is very hard. Worse, it is something that sounds and looks easy. Terrible graphs and misleading ones are not the result of maliciousness and cunning deception. Rather it is the opposite. Bad graphs happen because it is easy to visualize data, but hard to create good and meaningful visualizations[0]. It is because most people mindlessly apply a set of procedures to select the correct graph, not knowing the reasoning behind those procedures. It is in part due to the large quantity of people that have learned this and normalize/perpetuate the myth that visualization is easy. Because they do not distinguish the action from the end result. Just in the same way you might be able to perform all the manual tasks to assemble a house (use a screwdriver, hammer and nail, saw, fit pipes together, etc), it would be naive to assume that you could assemble a house. The reason there's so many terrible graphs is because it is easy to build a shanty and you rarely see an actual house to tell you what you're missing.

I doubt we'd see such a tool anytime soon. It takes expert experience and skill to make good visualizations and there are no well defined rules. If you see such a tool, I'd be wary of promises that are too big to be kept.

[0] Sometimes people complain about how something has a difficult/steep learning curve. It is important to note that while frustrating, this does not always make the learning curve a bad thing. Often a shallow learning curve can be bad because it convinces one that they have far greater ability than they actually do. We could argue that this is in part due to the improper way we visualize learning curves.

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#88

It really feels like an under defined task. Do you actually need to see those nodes? At that scale, you never want to render 100B of them. Instead you would need some kind of density aggregation when zoomed out and moving to LoD style k-d tree partitioning when zoomed in. That's almost the area of rendering engines like Unreal's Nanite. You can create your own renderer for data like this, but game engines are likely…

Truly, 100B nodes needs some sort of aggregation to have a chance at being useful. On a side project I've worked with normalizing >300GB semi-structured datasets that I could load up into dataframe libraries, I can't imagine working with a _graph_ of that size. I thought I was a genius when I figured out I could rent cloud computing resources with nearly a terabyte of RAM for less than federal minimum wage. At scale…

1) 100B? Try a thousand. Of course context matters, but I think it is common to overestimate the amount of information that can be visually conveyed at once. But it is also common to make errors in aggregation, or errors in how one interprets aggregation.

2) You may be interested in the large body of open source HPC visualization works. LLNL and ORNL are the two dominant labs in that space. Your issue might also be I/O since you can generate data faster than you can visualize it. One paradigm that HPC people utilize is "in situ" visualization. Where you visualize at runtime so that you do not hold back computation. At this scale, if you're not massively parallelizing your work, then it isn't the CPU that's the bottleneck, but the thing between the chair and keyboard. The downside of in situ is you have to hope you are visualizing the right data at the right time. But this paradigm includes pushing data to another machine that performs the processing/visualization or even storage (i.e. compute on the fast machine, push data to machine with lots of memory and that machine handles storage. Or more advanced, one stream to a visualization machine and another to storage). Checkout ADIOS2 for the I/O kind of stuff.

https://github.com/ornladios/ADIOS2

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#89

It really feels like an under defined task. Do you actually need to see those nodes? At that scale, you never want to render 100B of them. Instead you would need some kind of density aggregation when zoomed out and moving to LoD style k-d tree partitioning when zoomed in. That's almost the area of rendering engines like Unreal's Nanite. You can create your own renderer for data like this, but game engines are likely…

there's 8 million pixels in 4k, so if you're trying to graph 8 million points, might as well just fill up the screen with a single color and call it a day. If you have 8 billion, well you can graph about 0.1% of that and fill up every single pixel of the screen, but then you're just looking at noise. To be able to show connections between nodes, you'd need maybe 9 pixels per node, so that's around 900k nodes you migh…

It's unclear to me if you're making the same point I'm about to make. So I guess at best it's another point and at worst another framing?

I think the relationship to a 4k image is a great way to explain why you should never do this. Specifically because we can note how as resolution increases it gets difficult to distinguish the difference. Like the difference between 480p and 720p is quite large but 4k and 8k is... not. A big part of why the high res images even work is because the data being visualized is highly structured and neighboring data strongly relates. So maybe OP's graph contains highly structured graph cliques. But it is likely doubtful. Realistically, OP should be aiming for ways to convey their data with far less than 10k points. Maybe ask yourself a question: can you differentiate a picture of a thousand people from two thousand? Probably not.

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#90
post #67

Visualizing large graphs is a natural desire for people with lots of connected data. But after a fairly small size, there's almost no utility in visualizing graphs. It's much more useful to compute various measures on the graph, and then query the graph using some combination of node/edge values and these computed values. You might subset out the nodes and edges of particular interest if you really want to see them -…

Sometimes you can get farther with something like a summary statistics table of the different motifs that show up in a dataset.

Hairballs are not interesting, but the shapes that show up in a graph once you make a few cuts can be fascinating.

Post reply on HN