Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Gentle Introduction to Graph Neural Networks (2021)

distill.pub

Post reply on HN
Print view View original

21–30 of 36 posts

Re: A Gentle Introduction to Graph Neural Networks (2021)

#21

Post by algo_trader »

disattention wrote: post #18
magicalhippo wrote: post #13

Earlier quoted context omitted.

Naive question: Words in sentences kinda forms graphs, referencing other words or are leafs being referenced, both inside sentences and between sentences. Given the success of the attention mechanism in modern LLMs, how well would they do if you trained a LLM to process an actual graph? I guess you'd need some alternate tokenizer for optimal performance.

This is actually a good insight. It turns out that transformers are indeed a form of graph network, precisely because of the attention mechanism. Graph attention networks are actually a very popular GNN architecture. Generally, the issue with using an LLM style architecture for generic graphs is modeling the sparsity, but is possible by using the graph adjacency matrix to mask the attention matrix. There are a number…

Any canonical papers on GNN for code graphs?

Re: A Gentle Introduction to Graph Neural Networks (2021)

#22

Post by eperfa »

energy123 wrote: post #8
samsartor wrote: post #2

GNNs have been a bit of a disappointment to me. I've tried to apply them a couple times to my research but it has never worked out. For a long time GNNs were pitched as a generalization of CNNs. But CNNs are more powerful because the "adjacency weights" (so to speak) are more meaningful: they learn relative positional relationships. GNNs usually resort to pooling, like described here. And you can output an image with…

Google's GraphCast is a GNN: https://deepmind.google/discover/blog/graphcast-ai-model-for...

Google DeepMind's GenCast is based on diffusion: https://deepmind.google/discover/blog/gencast-predicts-weath...

(Partially) Google Research's/DeepMind's NeuralGCM is based on hybrid models using ODEs and learnt physics: https://www.nature.com/articles/s41586-024-07744-y

Microsoft Research's Aurora on vision transformers: https://www.microsoft.com/en-us/research/blog/introducing-au...

Huawei's Pangu Weather is also a 3D transformer I believe https://www.nature.com/articles/s41586-024-07744-y

I just wanted to highlight that there are multiple approaches in use for the same problem / in the same domain, and GNN does not seem to be the most widely used one.

Re: A Gentle Introduction to Graph Neural Networks (2021)

#23

Post by siver_john »

openrisk wrote: post #17

Very high quality work, its a pity that distill.pub did not find a sustainable way forward [1]. On GNN's, the lack of datasets [2] might be a reason they are not as talked about. This is something that has affected also the semantic web domain. [1] https://distill.pub/2021/distill-hiatus/ [2] https://huggingface.co/datasets?task_categories=task_categor...

Honestly, seeing this on the front page had my hopes up they had found out a way forward but alas.

Re: A Gentle Introduction to Graph Neural Networks (2021)

#24

Post by FjordWarden »

helltone wrote: post #5

It seems GNNs operate on a fixed topology. What if I want to approximate some transformation of the topology of the graph? For example learning how to layout a graph, or converting program abstract syntax trees to data flow graphs.

Maybe homology can help, it is a sort of calculus for discrete structures where you count how many N dimensional hole there are over time. Dunno about NN but that is what they can do with fMRI.

Re: A Gentle Introduction to Graph Neural Networks (2021)

#25

Post by biomcgary »

samsartor wrote: post #2

GNNs have been a bit of a disappointment to me. I've tried to apply them a couple times to my research but it has never worked out. For a long time GNNs were pitched as a generalization of CNNs. But CNNs are more powerful because the "adjacency weights" (so to speak) are more meaningful: they learn relative positional relationships. GNNs usually resort to pooling, like described here. And you can output an image with…

I've followed GNNs in biology and applied them in a couple domains, but have been disappointed by the results so far. I'm a bit surprised because I have successfully used other graph-based approaches in biology.

Re: A Gentle Introduction to Graph Neural Networks (2021)

#26

Post by whimsicalism »

magicalhippo wrote: post #13
cherryteastain wrote: post #12

There are a lot of papers using GNNs for physics simulations (e.g. computational fluid dynamics) because the unstructured meshes used to discretize the problem domain for such applications map very neatly to a graph structure. In practice, every such mesh/graph is used once to solve a particular problem. Hence it makes little sense to train a GNN for a specific graph. However, that's exactly what most papers did beca…

Naive question: Words in sentences kinda forms graphs, referencing other words or are leafs being referenced, both inside sentences and between sentences. Given the success of the attention mechanism in modern LLMs, how well would they do if you trained a LLM to process an actual graph? I guess you'd need some alternate tokenizer for optimal performance.

yep you're now pretty much at state-of-the-art

Re: A Gentle Introduction to Graph Neural Networks (2021)

#29

Post by Legend2440 »

eachro wrote: post #28

Is there consensus about whether gnn architectures are better than transformer based ones at this point? I am aware that transformers can be viewed as a gnn too.

GNNs let you build some of the structure of your problem into the network architecture. This can be useful if your problem has a well-understood structure, like physical simulations.

However in many cases we do not know the structure of our problem (that's why we want to use ML in the first place) and in these cases GNNs do not beat transformers.

Re: A Gentle Introduction to Graph Neural Networks (2021)

#30

Post by wenc »

openrisk wrote: post #17

Very high quality work, its a pity that distill.pub did not find a sustainable way forward [1]. On GNN's, the lack of datasets [2] might be a reason they are not as talked about. This is something that has affected also the semantic web domain. [1] https://distill.pub/2021/distill-hiatus/ [2] https://huggingface.co/datasets?task_categories=task_categor...

My personal hack is that before jumping into papers in unfamiliar areas like these, I go on YouTube and search for “ explained”.

The power of YouTube is that for any hot area, there are a bunch of people incentivized to make a short video that maximally engages. The quality can be quite high even at higher levels of math abstractions. The visuals are really helpful to get a feel for the abstractions (3 blue 1 brown proved this years ago).

There are some excellent videos of GNNS that in less than 10 mins gave me a launching point into the literature.

Post reply on HN