Why I'm Lukewarm on Graph Neural Networks • Hacker News Board

Re: Why I'm Lukewarm on Graph Neural Networks

#41

Post by VHRanger » Tue, Jan 05, 2021, 3:03 AM UTC

jedbrown wrote: post #28

sischoel wrote: post #13

I don't know much about graph neural networks although it is a topic that I want to study in the next few months. But what bothered me in your article is what you wrote about graph data structures. NetworkX is indeed very slow, this is due to two facts: - NetworkX is a pure Python implementation and does not relay on some methods written in a faster language like C. - They use dictionaries to represent the graphs, wh…

The index type for CSR need only index nodes. It's the offsets (one per row) that need to index edges. The blog's portrayal of NVMe vs distributed memory is a bit off: > With modern NVMe drives random seeks aren’t slow anymore, much faster than distributed network calls like you do when scaling the linked list-based graph. Distributed memory machines has 1 microsecond off-node latency and 15-20 microsecond reductions…

Thanks for the correction. I was aware of the speed of distributed cluster reads, but I might be overestimating the speed of NVMe random seeks.

> The other issue is that NVMe is more expensive than DRAM

How are you calculating that?

When I upgraded my home server the other week, 16GB of DDR3-1600 cost nearly as much as 1TB of decently fast NVMe (HP SX920).

Would you still think an on-disk mmap'ed CSR graph library be useful for medium-scale usecases given modern SSDs? Or do you think such latencies kill any use

Re: Why I'm Lukewarm on Graph Neural Networks

#42

Post by VHRanger » Tue, Jan 05, 2021, 3:05 AM UTC

jim22k wrote: post #31

enriquto wrote: post #17

Earlier quoted context omitted.

Indeed, the best support for large graphs in Python is found inside scipy.sparse

Anyone who is serious about graph analysis in Python using linear algebra should check out GraphBLAS (graphblas.org). The inclusion of semirings makes things like SSSP and BFS extremely elegant on sparse adjacency matrices. The main implementation is SuiteSparse::GraphBLAS, a C library which has two Python bindings (search for grblas or pygraphblas). Disclosure: I'm the author of grblas. Both are available on conda-f…

Thanks!

What's the underlying data structure of the graph?

Re: Why I'm Lukewarm on Graph Neural Networks

#43

Post by VHRanger » Tue, Jan 05, 2021, 3:07 AM UTC

FL33TW00D wrote: post #30

Interesting post, I’m working on a whole graph classification problem at the moment. If GNNs aren’t the future, then where else should I be looking?

For whole graph classification, I think GNNs are still the future.

Re: Why I'm Lukewarm on Graph Neural Networks

#44

Post by jedbrown » Tue, Jan 05, 2021, 4:32 AM UTC

VHRanger wrote: post #41

jedbrown wrote: post #28

Earlier quoted context omitted.

The index type for CSR need only index nodes. It's the offsets (one per row) that need to index edges. The blog's portrayal of NVMe vs distributed memory is a bit off: > With modern NVMe drives random seeks aren’t slow anymore, much faster than distributed network calls like you do when scaling the linked list-based graph. Distributed memory machines has 1 microsecond off-node latency and 15-20 microsecond reductions…

Thanks for the correction. I was aware of the speed of distributed cluster reads, but I might be overestimating the speed of NVMe random seeks. > The other issue is that NVMe is more expensive than DRAM How are you calculating that? When I upgraded my home server the other week, 16GB of DDR3-1600 cost nearly as much as 1TB of decently fast NVMe (HP SX920). Would you still think an on-disk mmap'ed CSR graph library be…

Unless your data is at rest, you're paying for bandwidth, not capacity. SRAM > HBM > DRAM > NVMe when you're buying bandwidth. Seriously, when we have a low-latency network, the most cost-effective way to run some simulations and graph algorithms is out of L3 on EPYC (where mid-tier parts have over 1 TB/s at a fraction of the energy and cost of an A100/MI100).

Your NVMe quotes 3.2 GB/s, versus 25 GB/s (theoretical) for a stick of DDR4-3200 (which is in the same price range). A standard 2-socket EPYC server can give you 300 GB/s from DRAM, but you'd need several servers servers packed with NVMe to provide that bandwidth.

The rationale for persistent storage (like NVMe) as an algorithmic device is either (a) you need it to be persistent or (b) you have physical or algorithmic constraints that prevent you from using more parallelism and you're willing to pay 5-10x for the privilege of executing from NVMe.

Re: Why I'm Lukewarm on Graph Neural Networks

#45

Post by mdawg » Tue, Jan 05, 2021, 8:11 AM UTC

Nice article

Re: Why I'm Lukewarm on Graph Neural Networks

#46

Post by VHRanger » Tue, Jan 05, 2021, 1:21 PM UTC

jedbrown wrote: post #44

VHRanger wrote: post #41

Earlier quoted context omitted.

Thanks for the correction. I was aware of the speed of distributed cluster reads, but I might be overestimating the speed of NVMe random seeks. > The other issue is that NVMe is more expensive than DRAM How are you calculating that? When I upgraded my home server the other week, 16GB of DDR3-1600 cost nearly as much as 1TB of decently fast NVMe (HP SX920). Would you still think an on-disk mmap'ed CSR graph library be…

Unless your data is at rest, you're paying for bandwidth, not capacity. SRAM > HBM > DRAM > NVMe when you're buying bandwidth. Seriously, when we have a low-latency network, the most cost-effective way to run some simulations and graph algorithms is out of L3 on EPYC (where mid-tier parts have over 1 TB/s at a fraction of the energy and cost of an A100/MI100). Your NVMe quotes 3.2 GB/s, versus 25 GB/s (theoretical) f…

When you say "you're paying for XXX" you mean cloud prices (in terms of $ to complete a similar job)?

I was referring to running it on your own metal, but your argument might hold there.

Obviously from a cloud provider I'd just take a massive pile of RAM rather than NVMe drives.

Re: Why I'm Lukewarm on Graph Neural Networks

#47

Post by jedbrown » Tue, Jan 05, 2021, 2:47 PM UTC

VHRanger wrote: post #46

jedbrown wrote: post #44

Earlier quoted context omitted.

Unless your data is at rest, you're paying for bandwidth, not capacity. SRAM > HBM > DRAM > NVMe when you're buying bandwidth. Seriously, when we have a low-latency network, the most cost-effective way to run some simulations and graph algorithms is out of L3 on EPYC (where mid-tier parts have over 1 TB/s at a fraction of the energy and cost of an A100/MI100). Your NVMe quotes 3.2 GB/s, versus 25 GB/s (theoretical) f…

When you say "you're paying for XXX" you mean cloud prices (in terms of $ to complete a similar job)? I was referring to running it on your own metal, but your argument might hold there. Obviously from a cloud provider I'd just take a massive pile of RAM rather than NVMe drives.

Cloud pricing is competitive and thus a good proxy for relative on-prem costs if you keep the on-prem hardware busy. If you want to buy the capability to run the job on-prem, but the hardware sits idle most of the time, then NVMe might be optimal. However, you should be aware that these constraints have greatly increased the cost of the job itself.

Re: Why I'm Lukewarm on Graph Neural Networks

#48

Post by VHRanger » Tue, Jan 05, 2021, 5:46 PM UTC

beaupletga wrote: post #26

I need to handle (for work) a graph with 40 million nodes and more than 130 million edges. As expected, networkx couldn't handle more than a million nodes so I had to search for python libs which might handle that much data. This is why I've been using your lib ( https://github.com/VHRanger/nodevectors ) for at least 2 weeks now as well as these 2 other libs: https://github.com/louisabraham/fastnode2vec and https://g…

Cheers.

Please raise an issue on my library's github or drop me an email if you run into issues using my lib.

I'm always happy to help others on their projects.

Re: Why I'm Lukewarm on Graph Neural Networks

#49

Post by [deleted] » Tue, Jan 05, 2021, 10:18 PM UTC

SeeDave wrote: post #34

fedd wrote: post #33

I tried to implement a graph neural network back in 2006... https://github.com/fedd/weighscore

If you want to feel old: 2020 - 2006 = 14. 2006 - 14 = 1992. When the 'World Wide Web' first launched....

[deleted]

Re: Why I'm Lukewarm on Graph Neural Networks

#50

Post by Labo333 » Mon, Jan 18, 2021, 9:10 AM UTC

beaupletga wrote: post #26

I need to handle (for work) a graph with 40 million nodes and more than 130 million edges. As expected, networkx couldn't handle more than a million nodes so I had to search for python libs which might handle that much data. This is why I've been using your lib ( https://github.com/VHRanger/nodevectors ) for at least 2 weeks now as well as these 2 other libs: https://github.com/louisabraham/fastnode2vec and https://g…

Hi, I'm the author of [1]

nodevectors implements many more algorithms than fastnode2vec that focuses on node2vec. I'm pretty sure fastnode2vec is the fastest node2vec implementation because it uses CSR format, JIT compilation but also supports multiprocessing. The sampling algorithm has been improved compared to the paper (and to all other implementations) and allows truly linear memory consumption.

On the other hand, nodevectors has a lot of very cool methods like ProNE so you should definitely try it on your data. However, the original ProNE code is probably faster as it is written in C++ and uses multiple cores while nodevectors just uses the installed BLAS (you can probably get a massive speedup by installing OpenBLAS).

[1]: https://github.com/louisabraham/fastnode2vec