I definitely enjoyed reading your article! Did you play around with any AI-specific accelerators (eg TPUs?) Looking at some basic cost analysis from a stranger on the Internet - https://medium.com/bigdatarepublic/cost-comparison-of-deep-l... - you can probably get a decent price reduction in training, especially using preemptive instances (and perhaps a better pricing contract with Google/AWS) It's kind of crazy how…
Last I've checked (a year or two ago) PyTorch support for TPU's were atrocious. Have they gotten any better?
How to train large deep learning models as a startup
41–50 of 84 posts
Re: How to train large deep learning models as a startup
#42I definitely enjoyed reading your article! Did you play around with any AI-specific accelerators (eg TPUs?) Looking at some basic cost analysis from a stranger on the Internet - https://medium.com/bigdatarepublic/cost-comparison-of-deep-l... - you can probably get a decent price reduction in training, especially using preemptive instances (and perhaps a better pricing contract with Google/AWS) It's kind of crazy how…
Last I've checked (a year or two ago) PyTorch support for TPU's were atrocious. Have they gotten any better?
PyTorch XLA is mature backend. In fact several other accelerators support PyTorch by lowering from XLA.
Re: How to train large deep learning models as a startup
#43This is an excellent article, which does a good job of detailing several factors involved here. But while it does suggest several ways to reduce the cost of training models, I'm left with a huge questions at the end.
How much does it ultimately cost to train a model at this size, and is it feasible to do without VS funding (and cloud credits)?
Re: How to train large deep learning models as a startup
#44> that still adds up to $2,451,526.58 to run 1,024 A100 GPUs for 34 days Salary costs are probably even higher than compute costs. Automatic Speech Recognition is an industrial scale application, it costs a lot to train, but so do many other projects in different fields. How expensive is a plane or a ship? How much can a single building cost? A rocket launch?
In what way are salary costs higher? This is on the order of 10 of their people’s annual salaries. This is for a single training run (meaning overall compute costs are higher), and it isn’t the only thing those ten or so people would have done that year (also meaning overall compute costs are higher).
yeah, but a cluster of the resulting model can transcribe thousands of hours of speech / second, 24/7 with a fixed accuracy, what can 10 humans do?
Re: How to train large deep learning models as a startup
#45Several hints here are severely outdated.
For instance, never train a model in end-to-end FP16. Use mixed precision, either via native TF/PyTorch or as a freebie when using TF32 on A100s. This’ll ensure that only suitable ops are run with lower precision; no need to fiddle with anything. Also, PyTorch DDP in multi-node regimes hasn’t been slower or less efficient than Horovod in ages.
Finally, buying a local cluster of TITAN Xs is an outright weird recommendation for massive models. VRAM limitations alone make this a losing proposition.
Re: How to train large deep learning models as a startup
#46tangent: i would dearly love to read a similar article focusing on practical advice on industrial application of statistical modelling, probabilistic programming & Bayesian inference
Re: How to train large deep learning models as a startup
#47Excellent and informative article--and a good bit of brand-building, I might say :-). One thing I'd love to see more writing about is prototyping and iterative development in these contexts--deep NNs are notoriously hard to get "right", and there seems to be a constant tension between model architecting, tuning hyperparameters, etc.--for example, you presumably don't want to have to wait a couple of weeks (and burn t…
This is an awesome blog post by Andrej Karpathy (the Director of AI at Tesla) about his recipe for training neural networks: https://karpathy.github.io/2019/04/25/recipe/
Re: How to train large deep learning models as a startup
#48> that still adds up to $2,451,526.58 to run 1,024 A100 GPUs for 34 days Salary costs are probably even higher than compute costs. Automatic Speech Recognition is an industrial scale application, it costs a lot to train, but so do many other projects in different fields. How expensive is a plane or a ship? How much can a single building cost? A rocket launch?
This is an entire seed round's worth of money on an operational expenditure.
Re: How to train large deep learning models as a startup
#49Earlier quoted context omitted.
> "create a NN and stack more layers until you have your MVP" I mean, that's a pretty good principled approach to a lot of ML problems.
I think you have a different definition of "principled" from most people.
I'm very curious as to what part of that process is not explained by the principles by which we understand neural networks to work.
I invite the possibility I've gone this long misunderstanding the definition of "principled" in this context.
Re: How to train large deep learning models as a startup
#50Check out Determined https://github.com/determined-ai/determined to help manage this kind of work at scale: Determined leverages Horovod under the hood, automatically manages cloud resources and can get you up on spot instances, T4's, etc. and will work on your local cluster as well. Gives you additional features like experiment management, scheduling, profiling, model registry, advanced hyperparameter tuning, etc. F…
Interesting. How do you guys manage spot interruptions when training on spot instances?
Users expose their model to our Trial API (https://docs.determined.ai/latest/topic-guides/model-definit...), the base class then implements a training loop (which can be enhanced with user-supplied callbacks, metrics, etc.) that has a whole bunch of bells and whistles. Easy distributed (multi-GPU and multi-node) training, automatic checkpointing, fault tolerance, etc.
Concretely, the system is regularly taking checkpoints (which include model weights and optimizer state) and so if the spots disappear (as they do), the system has enough information to resume from where things were last checkpointed when resources become available again.