Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is the best software to visualize a graph with a billion nodes?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 117 posts

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#61

Post by arrowleaf »

viraptor wrote: post #16

It really feels like an under defined task. Do you actually need to see those nodes? At that scale, you never want to render 100B of them. Instead you would need some kind of density aggregation when zoomed out and moving to LoD style k-d tree partitioning when zoomed in. That's almost the area of rendering engines like Unreal's Nanite. You can create your own renderer for data like this, but game engines are likely…

Truly, 100B nodes needs some sort of aggregation to have a chance at being useful. On a side project I've worked with normalizing >300GB semi-structured datasets that I could load up into dataframe libraries, I can't imagine working with a _graph_ of that size. I thought I was a genius when I figured out I could rent cloud computing resources with nearly a terabyte of RAM for less than federal minimum wage. At scale you quickly realize that your approach to data analysis is really bound by CPU, not RAM. This is where you'd need to brush off your data structures and algorithms books. OP better be good at graph algorithms.

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#62

Post by bunderbunder »

Xcelerate wrote: post #51

I might be prematurely classifying your question as an instance of the XY problem, but I worked at a company that tried to create something similar — a graph visualization system that could handle 100B nodes as part of our core product and... well... I would caution you not to do so if your purpose is something along those lines. There's almost never a use case where a customer wants to see a gigantic graph. Or resea…

I'm reminded of the time back in the aughties when I was asked to help print a ~300,000 page PDF. That's about 30 boxes' worth of paper if you print double-sided. I spent an hour tracing the request back to its source and discovered that they really only wanted some specific pieces of information out of it. I extracted that information from the file and printed maybe 5 pages instead.

In moments like these your job is to not be the monkey's paw. Don't just blithely give them what they asked for. Ask more questions to find out what they're actually trying to accomplish, and help them compose a more specific request that's closer to what they actually want.

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#63

Post by exe34 »

bunderbunder wrote: post #62
Xcelerate wrote: post #51

I might be prematurely classifying your question as an instance of the XY problem, but I worked at a company that tried to create something similar — a graph visualization system that could handle 100B nodes as part of our core product and... well... I would caution you not to do so if your purpose is something along those lines. There's almost never a use case where a customer wants to see a gigantic graph. Or resea…

I'm reminded of the time back in the aughties when I was asked to help print a ~300,000 page PDF. That's about 30 boxes' worth of paper if you print double-sided. I spent an hour tracing the request back to its source and discovered that they really only wanted some specific pieces of information out of it. I extracted that information from the file and printed maybe 5 pages instead. In moments like these your job is…

> Don't just blithely give them what they asked for.

Depends on how much they're paying you for it.

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#64

Post by acomjean »

surrTurr wrote: post #52

Cytoscape JS[1] with canvas rendering. Probably won't be able to do a billion nodes, but the last time I compared graph rendering libraries it was the best one in terms of performance/customizability. If you need even more performance, there's VivaGraphJS[2], which uses webgl to render. If you want other resources, I also have a GitHub list of Graph-related libraries (visualizations etc.) on GitHub[3]. [1]: https://j…

We use cytoscape for some of our genetics tools. It works well.

It does tend to “hairball” (technical term) at about 500+ nodes. That’s not the tools fault, just large graphs tend to be difficult.

It’s just hard to imagine visualizing a million plus nodes without doing some clustering first.

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#65

Post by oasisbob »

bunderbunder wrote: post #62
Xcelerate wrote: post #51

I might be prematurely classifying your question as an instance of the XY problem, but I worked at a company that tried to create something similar — a graph visualization system that could handle 100B nodes as part of our core product and... well... I would caution you not to do so if your purpose is something along those lines. There's almost never a use case where a customer wants to see a gigantic graph. Or resea…

I'm reminded of the time back in the aughties when I was asked to help print a ~300,000 page PDF. That's about 30 boxes' worth of paper if you print double-sided. I spent an hour tracing the request back to its source and discovered that they really only wanted some specific pieces of information out of it. I extracted that information from the file and printed maybe 5 pages instead. In moments like these your job is…

Asked to print a 300,000 page PDF you say? Almost sounds like it was meant for this guy:

https://www.psihoyos.com/image/I0000jtF1ui2j79Q

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#66

Post by bunderbunder »

exe34 wrote: post #63
bunderbunder wrote: post #62

Earlier quoted context omitted.

I'm reminded of the time back in the aughties when I was asked to help print a ~300,000 page PDF. That's about 30 boxes' worth of paper if you print double-sided. I spent an hour tracing the request back to its source and discovered that they really only wanted some specific pieces of information out of it. I extracted that information from the file and printed maybe 5 pages instead. In moments like these your job is…

> Don't just blithely give them what they asked for. Depends on how much they're paying you for it.

Knowing how and when to be consultative is a key soft skill that helps get you access to the higher end of the pay scale. It's how you demonstrate that you're an independent thinker who doesn't need to be micro-managed.

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#67

Post by bane »

Visualizing large graphs is a natural desire for people with lots of connected data. But after a fairly small size, there's almost no utility in visualizing graphs. It's much more useful to compute various measures on the graph, and then query the graph using some combination of node/edge values and these computed values. You might subset out the nodes and edges of particular interest if you really want to see them -- or don't visualize at all and just inspect the graph nodes and edges very locally with some kind of tabular data viewer.

It used to be thought that visualizing super large graphs would reveal some kind of macro-scale structural insight, but it turns out that the visual structure ends up becoming dominated by the graph layout algorithm and the need to squash often inherently high-dimensional structures into 2 or 3 dimensions. You end up basically seeing patterns in the artifacts of the algorithm instead of any real structure.

There's a similar, but unrelated desire to overlay sequenced transaction data (like transportation logs) on a geographical map as a kind of visualization, which also almost never reveals any interesting insights. The better technique is almost always a different abstraction like a sequence diagram with the lanes being aggregated locations.

There's a bunch of these kinds of pitfalls in visualization that people who work in the space inevitably end up grinding against for a while before realizing it's pointless or there's a better abstraction.

(source: I used to run an infoviz startup for a few years that dealt with this exact topic)

Re: Ask HN: What is the best software to visualize a graph with a billion nodes?

#68

Post by zX41ZdbW »

simpaticoder wrote: post #39

Hilbert curves (or similar) are often used for graphing billions of nodes[1]. However this will not by default show the relationships between nodes in a graph. Depending on your data you may be able to write a function to map from your edge list to a node index that hints at proximity. Note that visualizations are limited by human perception to ~10000 elements, more usefully 1000 elements. You might try a force direc…

Thank you for the link! I've recently created a small service on this topic: https://reversedns.space/

It visualizes the IPv4 space based on reverse DNS responses.

Post reply on HN