Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TensorFlow Graph Neural Networks

blog.tensorflow.org

Post reply on HN
Print view View original

11–20 of 23 posts

Re: TensorFlow Graph Neural Networks

#11

Post by atomflunder »

quibono wrote: post #9
lmeyerov wrote: post #8

Earlier quoted context omitted.

Think of it as an ensemble for blending your normal NN features (ex: RNN for time/clickstreams) with a model that can also leverage useful graph features (document citations, app logins, chemicals connecting, social graphs). We think a lot about security/fraud and digital journeys, where NN + xgboost are popular in general, and graph is used seperately (or upstream) for looking at broader structure. GNNs help blend t…

I remember reading a bit about GNNs circa 2019. At that time it seemed to have mostly to do with point clouds (for LIDAR data and for 3-D modelling mostly) but I imagine things have changed lots on this front. Are there any interesting papers/resources you could recommend for one to get back up to speed?

From what I can tell, the field is indeed evolving very rapidly, but I have only worked on a specific application (knowledge graph completion), so I can't give an overview over all the current day applications. I can, however, recommend William Hamilton's excellent text book, which is available online [1].

[1] https://www.cs.mcgill.ca/~wlh/grl_book/files/GRL_Book.pdf

Re: TensorFlow Graph Neural Networks

#12

Post by lmeyerov »

quibono wrote: post #9
lmeyerov wrote: post #8

Earlier quoted context omitted.

Think of it as an ensemble for blending your normal NN features (ex: RNN for time/clickstreams) with a model that can also leverage useful graph features (document citations, app logins, chemicals connecting, social graphs). We think a lot about security/fraud and digital journeys, where NN + xgboost are popular in general, and graph is used seperately (or upstream) for looking at broader structure. GNNs help blend t…

I remember reading a bit about GNNs circa 2019. At that time it seemed to have mostly to do with point clouds (for LIDAR data and for 3-D modelling mostly) but I imagine things have changed lots on this front. Are there any interesting papers/resources you could recommend for one to get back up to speed?

For enterprise relevance in our world, the exciting things have been handling heterogeneity via things like RGCNs, and handling bigger scales via DGL (GPU tricks, sampling tricks, ...). Imagine fraud, hacks, and entity resolution from everything you've recorded on a user interacting with a system.

There are important cases like maps and chemistry that take more specialized techniques, but we focus on events/logs/etc. So less to say on the niche stuff, even if those niches cover big use cases like "how google maps works" or "how google auto-designs their TPUs"

For the logs/events/transactions/clicks/devices/users/accounts cases, happy to chat, but maybe not as useful elsewhere :)

Re: TensorFlow Graph Neural Networks

#13

Post by H8crilA »

lmeyerov wrote: post #8
H8crilA wrote: post #7

Earlier quoted context omitted.

What's an example problem for which such networks work well?

Think of it as an ensemble for blending your normal NN features (ex: RNN for time/clickstreams) with a model that can also leverage useful graph features (document citations, app logins, chemicals connecting, social graphs). We think a lot about security/fraud and digital journeys, where NN + xgboost are popular in general, and graph is used seperately (or upstream) for looking at broader structure. GNNs help blend t…

So, if I understand you well, it would be something like this. Inputs:

1. {x0, x1, ...} - nodes in the graph, say users

2. Bunch of edges like {x_i, x_j}, say social connections

3. Some raw or processed features on nodes, say the text of posts, age of the account or some nlp-based scores for posts

4. Some raw or processed features on the edges (in particular maybe some coloring)

Before: people would train various classifiers/regressors directly on the nodes and/or edges, then maybe use the graph structure to propagate the scores.

After: But instead you could train whatever objective you have from raw features on the edges and nodes, with some extra message passing between nodes and edges. For example train (some of) that nlp-based classifier together with the graph part. And the benefit would be that, for example, you can extract some signals from the NLP part that would be more useful in determining the properties of neighbors, but not necessarily as useful in determining the properties of the current node/edge.

Question - what's the maximum range of such message passing? Sounds a bit like an RNN, where the unroll depth can be an issue. Though in practice most graphs have a low average path length, so maybe this is not a particularly big problem.

Although if you start unrolling graphs you'll very quickly load ~everything, so I guess the training must be completely reworked (flush data to distributed storage frequently then shuffle for the next step) or you cannot unroll further than maybe a few steps.

Re: TensorFlow Graph Neural Networks

#14

Post by lmeyerov »

H8crilA wrote: post #13
lmeyerov wrote: post #8

Earlier quoted context omitted.

Think of it as an ensemble for blending your normal NN features (ex: RNN for time/clickstreams) with a model that can also leverage useful graph features (document citations, app logins, chemicals connecting, social graphs). We think a lot about security/fraud and digital journeys, where NN + xgboost are popular in general, and graph is used seperately (or upstream) for looking at broader structure. GNNs help blend t…

So, if I understand you well, it would be something like this. Inputs: 1. {x0, x1, ...} - nodes in the graph, say users 2. Bunch of edges like {x_i, x_j}, say social connections 3. Some raw or processed features on nodes, say the text of posts, age of the account or some nlp-based scores for posts 4. Some raw or processed features on the edges (in particular maybe some coloring) Before: people would train various cla…

Yes I think you are seeing it

Before: People might precompute graph scores ("pagerank", ...) and use as features for tabular NNs. Or use simpler and slow GNNs like GraphSAGE bc the domain fit was great (ex: Pinterest social recs)

After: heterogeneity and scale for graphs that fit in CPU RAM (1TB) w decent GPUs

Re:unrolling, yeah a bunch of papers there :) sampling, artificial jump edges, and adversarial techniques have been helping with aspects of generalization (far data, unbalanced data, ...)

Re: TensorFlow Graph Neural Networks

#16

Post by elpakal »

H8crilA wrote: post #7
technologia wrote: post #3

I’m glad to see support for GNNs with tensorflow. Working with gnns for the past few years, personally for me it gets tiring to roll my own framework.

What's an example problem for which such networks work well?

They show a simple one in the post

> In the example below, we build a model using the TF-GNN Keras API to recommend movies to a user based on what they watched and genres that they liked.

> The code above works great, but sometimes we may want to use a more powerful custom model architecture for our GNNs. For example, in our previous use case, we might want to specify that certain movies or genres hold more weight when we give our recommendation.

Re: TensorFlow Graph Neural Networks

#17

Post by agentofoblivion »

claytonjy wrote: post #4
technologia wrote: post #3

I’m glad to see support for GNNs with tensorflow. Working with gnns for the past few years, personally for me it gets tiring to roll my own framework.

what's the state of GNN support elsewhere? does everyone else also roll their own, or are folks using Pytorch or something else?

Deep Graph Library (DGL) is the big one, which can use either PyTorch, MXNet or Tensorflow as the backend and is developed by AWS. You also have PyTorch Geometric and Jraph, which is built on top of JAX and used mostly by researchers at DeepMind as far as I can tell.

Re: TensorFlow Graph Neural Networks

#20

Post by thecleaner »

claytonjy wrote: post #4
technologia wrote: post #3

I’m glad to see support for GNNs with tensorflow. Working with gnns for the past few years, personally for me it gets tiring to roll my own framework.

what's the state of GNN support elsewhere? does everyone else also roll their own, or are folks using Pytorch or something else?

The bottleneck in GNN computations is that the aggregation ops cant be expressed as matrix operations and require writing custom kernels. This problem was solved in PyTorch with torch-scatter. The other bottleneck is subsampling (e.g k-hop) which also dont benefit from GPU support. Other than that the embedding aspects can just be written as nn ops.
Post reply on HN