Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Microsoft Zero and DeepSpeed: Memory Efficient Large Neural Network Training

microsoft.com

Post reply on HN
Print view View original

11–20 of 21 posts

Re: Microsoft Zero and DeepSpeed: Memory Efficient Large Neural Network Training

#11

Post by andbberger »

skwb wrote: post #4

I work in deep learning for 3D imaging, and memory has constantly been the primary bottleneck for our group. U-net for example tends to be fairly "chonky", and isn't really super great in terms of parameter efficiency (but it is nice when you need an out of the box network that just "works"...). This has led medical imaging to use a lot of "patching" and other sliding window sort of techniques to help get over this b…

I love to hate on U-net. It works but it's just so inelegant. That is not a true convolution and only works for particular 'patch' sizes bothers me to no end.

I am not super up to date with the field, but has anyone caught on to using 'wavenet' like architectures yet? That is, dialated convolutions.

You have to be a little clever to get residual connections to work properly, but it's a true convolution that works for any patch size, is super-parameter efficient, and captures the same multi-scale features U-net was designed for.

Anecdotally, I used such an arch for some (unfortunately proprietary) 3D imaging work and achieved some nice results.

Re: Microsoft Zero and DeepSpeed: Memory Efficient Large Neural Network Training

#12

Post by coolswan »

jeffra wrote: post #5

I'm from the DeepSpeed team, we're happy to answer questions if people have them.

Will there be a whitepaper pertaining to T-NLG? Post is light on details.

Also, what's the timeline to release private preview? Outside of academia, who else will you likely be working with first?

Re: Microsoft Zero and DeepSpeed: Memory Efficient Large Neural Network Training

#13

Post by rckoepke »

skwb wrote: post #4

I work in deep learning for 3D imaging, and memory has constantly been the primary bottleneck for our group. U-net for example tends to be fairly "chonky", and isn't really super great in terms of parameter efficiency (but it is nice when you need an out of the box network that just "works"...). This has led medical imaging to use a lot of "patching" and other sliding window sort of techniques to help get over this b…

I think that silicon and data compression are well-optimized domains already.

However, Neural Nets are provably horrible w.r.t resource efficiency, as demonstrated by recent advances like EfficientNet, which managed to decrease the number of parameters drastically and improve top 1% performance as well.

Re: Microsoft Zero and DeepSpeed: Memory Efficient Large Neural Network Training

#14

Post by skwb »

andbberger wrote: post #11
skwb wrote: post #4

I work in deep learning for 3D imaging, and memory has constantly been the primary bottleneck for our group. U-net for example tends to be fairly "chonky", and isn't really super great in terms of parameter efficiency (but it is nice when you need an out of the box network that just "works"...). This has led medical imaging to use a lot of "patching" and other sliding window sort of techniques to help get over this b…

I love to hate on U-net. It works but it's just so inelegant. That is not a true convolution and only works for particular 'patch' sizes bothers me to no end. I am not super up to date with the field, but has anyone caught on to using 'wavenet' like architectures yet? That is, dialated convolutions. You have to be a little clever to get residual connections to work properly, but it's a true convolution that works for…

> "It works".

Well that's sorta the point. Personally I'm not a super huge fan of creating a super specific network architecture and resulting in 2-3% difference in performance. Certainly if you're doing something where a configuration makes sense (LSTM for time series for example), but I think there needs to be a rethinking of the Grand Theory of Deep Learning Architecture TM.

And frankly I think a unsaid reason why U-net is so popular is that it does generalize reasonably well with limited data, which in many fields is not as massive as COCO.

I realize it's sorta asking too much (I both want a NN that works both out of the box, super easily, and doesn't require a TON of data), but I think that's where the current pains are for really explosive growth in AI.

Re: Microsoft Zero and DeepSpeed: Memory Efficient Large Neural Network Training

#15

Post by saurkt »

coolswan wrote: post #12
jeffra wrote: post #5

I'm from the DeepSpeed team, we're happy to answer questions if people have them.

Will there be a whitepaper pertaining to T-NLG? Post is light on details. Also, what's the timeline to release private preview? Outside of academia, who else will you likely be working with first?

We don't have an exact date, but, we plan to share more details in a later submission. If you want access, please send an email to [turing_ AT _microsoft _DOT_ com]. Remove underscores and spaces.

Re: Microsoft Zero and DeepSpeed: Memory Efficient Large Neural Network Training

#16

Post by choppaface »

jeffra wrote: post #10
choppaface wrote: post #2

Even from the paper, it's hard to tell what this library actually does: Section 5 in https://arxiv.org/pdf/1910.02054.pdf The paper talks about parameter partitioning and overlapped communication, but doesn't actually give many details on how those things happen. The library appears to be an implementation of some common algos for solving the 'pebble game,' as explained decently here: https://medium.com/tensorflow/fi…

We tried to communicate the key ideas in the video released with the blog post. It shows how DeepSpeed and the ZeRO optimizer save memory, and shows exactly what happens during each iteration of training. It is quite different from standard data or model parallelism. The ZeRO optimizer helps scale large models regardless of the model topology. It works equally well for wide or deep models. Please let us know if you h…

Oh sorry I didn't make it to the video because the blog post intro made me bounce straight to the paper. I agree the video is a big help versus what's given in the paper.

It looks like your approach plays the 'pebble counting' game described in the OpenAI article I linked. Or maybe you'd like to explain what's different.

What would really help in the video (and paper) is a grounded example (like Resnet10 or AlexNet or just a 2-layer MLP) and drawing the connection between GPU buffers and layers. I feel the video covers details of the memory savings in way too much precision while the intuition behind the method (and how it translates to a graphical model of a NN) is essentially absent.

Re: Microsoft Zero and DeepSpeed: Memory Efficient Large Neural Network Training

#17

Post by davidwhite »

jeffra wrote: post #5

I'm from the DeepSpeed team, we're happy to answer questions if people have them.

Looks super cool. Does it remove the need for manual gradient checkpointing?

Also curious if there are expected to be memory / speed improvements if you're using it on a single GPU or if most gains come from improved parallelism across devices.

Re: Microsoft Zero and DeepSpeed: Memory Efficient Large Neural Network Training

#18

Post by andbberger »

skwb wrote: post #14
andbberger wrote: post #11

Earlier quoted context omitted.

I love to hate on U-net. It works but it's just so inelegant. That is not a true convolution and only works for particular 'patch' sizes bothers me to no end. I am not super up to date with the field, but has anyone caught on to using 'wavenet' like architectures yet? That is, dialated convolutions. You have to be a little clever to get residual connections to work properly, but it's a true convolution that works for…

> "It works". Well that's sorta the point. Personally I'm not a super huge fan of creating a super specific network architecture and resulting in 2-3% difference in performance. Certainly if you're doing something where a configuration makes sense (LSTM for time series for example), but I think there needs to be a rethinking of the Grand Theory of Deep Learning Architecture TM. And frankly I think a unsaid reason why…

> I think there needs to be a rethinking of the Grand Theory of Deep Learning Architecture TM.

strong agree. Although perhaps not so much a rethinking as a theory of all. Huge dearth of theory in the field. Daily practition involves regular use of black magic intuition for arch, problem posing and debugging. Weird times.

Re: Microsoft Zero and DeepSpeed: Memory Efficient Large Neural Network Training

#19

Post by tixocloud »

While on the surface, this looks interesting, can anyone help me understand who exactly needs to do and redo neural network training that will take advantage of these optimizations? I’m struggling to understand which companies/data scientists would use this.

Re: Microsoft Zero and DeepSpeed: Memory Efficient Large Neural Network Training

#20

Post by tixocloud »

jeffra wrote: post #8
Tenoke wrote: post #6

Earlier quoted context omitted.

This is great and looks very easy to use! I'd expect it to have a huge impact given how easy it makes for people to leverage a few or a few thousand GPUs. I do have a few questions, of course. Is it getting a lot of internal use already (beyond the example we just heard about)? Is it possible to do inference using a CPU and a lot of RAM using a model trained on multiple GPUs via DeepSpeed? Does it work with TPUs righ…

> Is it getting a lot of internal use already (beyond the example we just heard about)? We have hundreds of internal users of DeepSpeed using it to train production ready models, many of which have been already shipped. > Is it possible to do inference using a CPU and a lot of RAM using a model trained on multiple GPUs via DeepSpeed? It is definitely possible to do inference on CPU using a model trained on multiple G…

Are you able to share the use cases for production ready models?
Post reply on HN