Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to train large deep learning models as a startup

assemblyai.com

Post reply on HN
Print view View original

61–70 of 84 posts

Re: How to train large deep learning models as a startup

#61

Post by m_ke »

aledalgrande wrote: post #60
m_ke wrote: post #59

Earlier quoted context omitted.

As a bootstrapper I camped all night outside of bestbuy to get some 3090s. Other tips not mentioned in the article: 1. Tune your hyper parameters on a subset of the data. 2. Validate new methods with smaller models on public datasets. 3. Tune models instead of training from scratch (either public models or your previously trained ones).

Great hacks, although you have to be aware of the trade-offs: 1. if you choose the wrong subset, you'll find a non optimum local min 2. still risk dead ends when expanding the model and lengthen the time to finding that out 3. a lot of public models are made from inaccurate datasets, so beware Overall you have to start somewhere though, and your points are still valid.

1. The small subset is to test that your training pipeline works and converges near 0 loss.

2. Sure, but for most new hacks like mixup, randaugment and etc the results usually transfer over. Problem with deep learning is that most of the new results don't replicate so it's good to have a way to quickly validate things.

3. The lower level features are usually pretty data agnostic and transfer well to new tasks.

Re: How to train large deep learning models as a startup

#62

Post by dylanbfox »

Reubend wrote: post #43

This is an excellent article, which does a good job of detailing several factors involved here. But while it does suggest several ways to reduce the cost of training models, I'm left with a huge questions at the end. How much does it ultimately cost to train a model at this size, and is it feasible to do without VS funding (and cloud credits)?

Author here. Thanks for your comments!

In general - this is expensive stuff. Training big, accurate models just requires a lot of compute, and there is a "barrier to entry" wrt costs, even if you're able to get those costs down. I think it's similar to startups not really being able to get into the aerospace industry unless they raise lots of funding (ie, Boom Supersonic).

Practically speaking though, for startups without funding, or access to cloud credits, my advice would be to just train the best model you can, with the compute resources you have available. Try to close your first customer with an "MVP" model. Even if your model is not good enough for most customers - you can close one, get some incremental revenue, and keep iterating.

When we first started (2017), I trained models that were ~1/10 the size of our current models on a few K80s in AWS. These models were much worse compared to our models today, but they helped us make incremental progress to get to where we are now.

Re: How to train large deep learning models as a startup

#64

Post by dinvlad »

dylanbfox wrote: post #7
visarga wrote: post #4

> that still adds up to $2,451,526.58 to run 1,024 A100 GPUs for 34 days Salary costs are probably even higher than compute costs. Automatic Speech Recognition is an industrial scale application, it costs a lot to train, but so do many other projects in different fields. How expensive is a plane or a ship? How much can a single building cost? A rocket launch?

> Salary costs are probably even higher than compute costs. Yes exactly. Managing that much compute requires many humans!

I wouldn't be so sure :-)

Re: How to train large deep learning models as a startup

#66

Post by aabaker99 »

aledalgrande wrote: post #60
m_ke wrote: post #59

Earlier quoted context omitted.

As a bootstrapper I camped all night outside of bestbuy to get some 3090s. Other tips not mentioned in the article: 1. Tune your hyper parameters on a subset of the data. 2. Validate new methods with smaller models on public datasets. 3. Tune models instead of training from scratch (either public models or your previously trained ones).

Great hacks, although you have to be aware of the trade-offs: 1. if you choose the wrong subset, you'll find a non optimum local min 2. still risk dead ends when expanding the model and lengthen the time to finding that out 3. a lot of public models are made from inaccurate datasets, so beware Overall you have to start somewhere though, and your points are still valid.

1. Gradient descent almost always finds a non optimum local min (it is not guaranteed to find a global min).

Re: How to train large deep learning models as a startup

#67

Post by agnosticmantis »

aabaker99 wrote: post #66
aledalgrande wrote: post #60

Earlier quoted context omitted.

Great hacks, although you have to be aware of the trade-offs: 1. if you choose the wrong subset, you'll find a non optimum local min 2. still risk dead ends when expanding the model and lengthen the time to finding that out 3. a lot of public models are made from inaccurate datasets, so beware Overall you have to start somewhere though, and your points are still valid.

1. Gradient descent almost always finds a non optimum local min (it is not guaranteed to find a global min).

Isn’t the current best practice to train highly over-parametrized models to zero training error? That’d be a global optima, no?

Unless we’re talking about the optima of test error.

Re: How to train large deep learning models as a startup

#68

Post by cinntaile »

mkolodny wrote: post #47
elmomle wrote: post #38

Excellent and informative article--and a good bit of brand-building, I might say :-). One thing I'd love to see more writing about is prototyping and iterative development in these contexts--deep NNs are notoriously hard to get "right", and there seems to be a constant tension between model architecting, tuning hyperparameters, etc.--for example, you presumably don't want to have to wait a couple of weeks (and burn t…

This is an awesome blog post by Andrej Karpathy (the Director of AI at Tesla) about his recipe for training neural networks: https://karpathy.github.io/2019/04/25/recipe/

I would like to second this. Thanks for linking this. As someone starting out in deep learning and noticing that a lot of things are still more art than science this seems great for avoiding some footguns!

Re: How to train large deep learning models as a startup

#69

Post by bigbillheck »

etrain wrote: post #9

Check out Determined https://github.com/determined-ai/determined to help manage this kind of work at scale: Determined leverages Horovod under the hood, automatically manages cloud resources and can get you up on spot instances, T4's, etc. and will work on your local cluster as well. Gives you additional features like experiment management, scheduling, profiling, model registry, advanced hyperparameter tuning, etc. F…

Oh hey I interviewed with y'all a few years back, glad to see you're still around.

Re: How to train large deep learning models as a startup

#70

Post by 6gvONxR4sf7o »

cpill wrote: post #44
6gvONxR4sf7o wrote: post #21

Earlier quoted context omitted.

In what way are salary costs higher? This is on the order of 10 of their people’s annual salaries. This is for a single training run (meaning overall compute costs are higher), and it isn’t the only thing those ten or so people would have done that year (also meaning overall compute costs are higher).

yeah, but a cluster of the resulting model can transcribe thousands of hours of speech / second, 24/7 with a fixed accuracy, what can 10 humans do?

Huh. You and I have two very different readings on this. I'm talking about the ML researcher's time (what if they hired more people instead) and you're talking about human text processors (what if people did this work by hand instead). Kinda neat that we had such different readings.
Post reply on HN