71–80 of 84 posts
Earlier quoted context omitted.
1. Gradient descent almost always finds a non optimum local min (it is not guaranteed to find a global min).
Isn’t the current best practice to train highly over-parametrized models to zero training error? That’d be a global optima, no? Unless we’re talking about the optima of test error.
Several hints here are severely outdated. For instance, never train a model in end-to-end FP16. Use mixed precision, either via native TF/PyTorch or as a freebie when using TF32 on A100s. This’ll ensure that only suitable ops are run with lower precision; no need to fiddle with anything. Also, PyTorch DDP in multi-node regimes hasn’t been slower or less efficient than Horovod in ages. Finally, buying a local cluster…
Do you have an alternative recommendation?
Earlier quoted context omitted.
Great hacks, although you have to be aware of the trade-offs: 1. if you choose the wrong subset, you'll find a non optimum local min 2. still risk dead ends when expanding the model and lengthen the time to finding that out 3. a lot of public models are made from inaccurate datasets, so beware Overall you have to start somewhere though, and your points are still valid.
1. Gradient descent almost always finds a non optimum local min (it is not guaranteed to find a global min).
Several hints here are severely outdated. For instance, never train a model in end-to-end FP16. Use mixed precision, either via native TF/PyTorch or as a freebie when using TF32 on A100s. This’ll ensure that only suitable ops are run with lower precision; no need to fiddle with anything. Also, PyTorch DDP in multi-node regimes hasn’t been slower or less efficient than Horovod in ages. Finally, buying a local cluster…
> Finally, buying a local cluster of TITAN Xs is an outright weird recommendation for massive models. VRAM limitations alone make this a losing proposition. Do you have an alternative recommendation?
It provides some modern, real life, deep learning benchmarks using the mixed precision (TF32) that gp was referring to.
1. Purchase GPU machines from Lambda Labs. I went with machines with 256 GB of CPU RAM, 24-core AMD Threadrippers, 2 NVIDIA RTX 3090s, and 10gbps Ethernet. You might want to choose even more expensive GPUs.
2. Make sure your electrical circuits have sufficient capacity to run your GPU machines at peak power consumption. I gave each machine its own US residential electrical circuit. If you are storing your GPU servers in a data center, look into whether they can get you enough electrical power for Lambda Labs's 8-GPU machines. When talking with a data center's sales team, make sure they understand how much electrical power you need. They might charge you a lot of money if you ask for much more electrical power than they usually install in a cabinet. Try to negotiate with multiple data centers to see you can give you the best offer.
3. Purchase storage machines from 45Drives. I recommend buying their 30-drive machines and setting up a ZFS pool of 10 3-drive mirrors. Do not bother with raidz because your read and write speeds will be too slow, bottlenecking your ETL and training jobs.
4. Serve files from your storage machines to your GPU machines using NFS. I like to use MergerFS to merge mounts from different NFS servers. Alternatively, you might want to use Ceph, Min.io, or Lustre.
5. Buy Intel NUCs to run miscellaneous services--like monitoring--that you wouldn't want to colocate with your storage or GPU machines. They are small, cheap, and don't require a lot of electrical power. I bought a couple of NUCs with 64 GB of RAM and a 1 TB NVMe SSD each. Then I purchased external 10gbps Ethernet cards to plug into each NUC's 40gbps Thunderbolt 3 port.
6. Buy 10gbps network switches. MikroTik has affordable 4-port, 8-port, and 16-port 10gbps switches. These are SFP+ (optical) switches, so you may need to buy copper adapters. I really like MikroTik's balance of quality and affordability, so I also buy network routers and other equipment from MikroTik.
7. If possible, try to train models small enough that each model only needs one machine to train. For this reason, maybe you will want to buy one 10-GPU machine instead of 5 2-GPU machines. There are Amdahl's Law-style coordination costs to using multiple machines to train the same model. When I do large hyperparameter searches over many candidate models, I minimize these coordination costs and maximize throughput by limiting each model to only one machine. Of course, this is impossible if you are like AppliedAI and need 48 V100s to train a model.
8. If you do need to train a single model using multiple machines, I've heard good things about Horovod, but I'm also excited about Ray.io--which offers user-friendly distributed training wrappers around TensorFlow MultiWorkerMirroredStrategy, PyTorch's DistributedDataParallel, or Horovod (which itself can train TensorFlow, PyTorch, or MXNet).
Excellent and informative article--and a good bit of brand-building, I might say :-). One thing I'd love to see more writing about is prototyping and iterative development in these contexts--deep NNs are notoriously hard to get "right", and there seems to be a constant tension between model architecting, tuning hyperparameters, etc.--for example, you presumably don't want to have to wait a couple of weeks (and burn t…
This is an awesome blog post by Andrej Karpathy (the Director of AI at Tesla) about his recipe for training neural networks: https://karpathy.github.io/2019/04/25/recipe/
Earlier quoted context omitted.
1. Gradient descent almost always finds a non optimum local min (it is not guaranteed to find a global min).
Yeah but depending on the data you might have even worse results, selecting the right subset to be representative is really important.
> that still adds up to $2,451,526.58 to run 1,024 A100 GPUs for 34 days Salary costs are probably even higher than compute costs. Automatic Speech Recognition is an industrial scale application, it costs a lot to train, but so do many other projects in different fields. How expensive is a plane or a ship? How much can a single building cost? A rocket launch?
In what way are salary costs higher? This is on the order of 10 of their people’s annual salaries. This is for a single training run (meaning overall compute costs are higher), and it isn’t the only thing those ten or so people would have done that year (also meaning overall compute costs are higher).
in my experience it's often more like "just use linear regression and tell everyone you're using AI"
Thats for structured data, for non structured it's more like "create a NN and stack more layers until you have your MVP"
Several hints here are severely outdated. For instance, never train a model in end-to-end FP16. Use mixed precision, either via native TF/PyTorch or as a freebie when using TF32 on A100s. This’ll ensure that only suitable ops are run with lower precision; no need to fiddle with anything. Also, PyTorch DDP in multi-node regimes hasn’t been slower or less efficient than Horovod in ages. Finally, buying a local cluster…
This blog is more of an intro to a few high level concepts (multi-GPU and multi-node training, fp32 vs fp16, buying hardware and dedicated machines vs AWS/GCP, etc) for startups that are early into their deep learning journey, and that might need a nudge in the right direction.
If you're looking for a deep dive into the best GPUs to buy (cost/perf, etc), the link in the below comment gives a pretty good overview.
PS - I can send you some benchmarks we did that show (at least for us) Horovod is ~10% faster than DDP for multi-node training FWIW. Email is in my profile!