Why I'm Lukewarm on Graph Neural Networks

I suspect we will not see a revolution in deep learning from GNN without a corresponding hardware(FPGA?) or optimization method(HSIC? HTMs?) advance.

GNNs trained by backprop are making many of the same mistakes that LSTMs did: solve one important problem(exploding/vanishing gradients), but introduce a bunch of hyperparameters that bring along their own set of problems. GRUs are successful in my opinion, because they remove some of those tunable parameters.

Of course as the post suggested, not being able to tune something, often loses out against some more tuned and curated solution. GNNs being the newest version, having tons of parameters to tweek. In the end do we get a better, ultimately more generalizable solution? Or do we just get more hyperparameters to tune, and spend more time and money for a modest, unrepeatable gain.

I don't know much about graph neural networks although it is a topic that I want to study in the next few months.

But what bothered me in your article is what you wrote about graph data structures.

NetworkX is indeed very slow, this is due to two facts: - NetworkX is a pure Python implementation and does not relay on some methods written in a faster language like C. - They use dictionaries to represent the graphs, which may have some advantages when mutating graphs, but of course have much worse locality than an adjacency list or a some sparse matrix format.

But even for python there are much faster libraries such as igraph. The data structure in igraph is an edge list.

A lot of single core graph libraries use an adjacency list internally, and while it is true that the index list for each vertex can be somewhere arbitrary in memory, they usually do not behave like a linked list, unless you graph is really sparse. One of the most used operations in graph algorithms is to iterate over the neighbors of a vertex, and for this, adjacency lists are very good.

They also have a small advantage over CSR matrices for adding or removing edges, and they might use slightly less memory, as their index type only needs to be able to index all vertices and not all edges, so they need half of the space, which is better for caches.

Google maps improved their worse-case residuals of traffic estimation using GNNs (messaging passing Nets, not graph embeddings).

Indeed, the best support for large graphs in Python is found inside scipy.sparse

NetworkX isn't a bad library, nor is it only suitable for "babies". Yeesh. Actual people work on this stuff, you know, and they may have different goals, requirements, etc than you. This whole post reeks of someone who's so in love with being a maverick speaking uncomfortable truths that they've lost sight of this human element.

If you're offended by (say) research papers that fail to break enough new ground to satisfy you, sorry about your luck, but again, not everyone's optimizing the same function as you. You'll probably do better bringing people around when you're not implying (or outright saying) their work is shit.