Ask HN: Worth it to buy 4x Nvidia Tesla K40 for AI?

I saw a post on a local market place that’s selling a complete system with 4 Tesla K40s 12 GBs VRAM w/ passive cooling for $400. The post description said that the system was intended to be used for training AI models, which is what I want to use it for… nothing too serious I am mostly still learning here. The cards themselves were released on 2013 and would have a combined cuda cores of 12,928 if I’m counting the 5th video card for a monitor (GTX 1660)

Here are the complete specs from the post description… from a dollar value of all these parts, I’m not really losing any money… I just don’t have good enough intuition to see if this system is worth it to learn practice modern day AI.

Specs:

Motherboard: MSI MAG Z390 Tomahawk gaming 9th generation with dual Ethernet ports for wiring with other servers, and max speed 4400 MHz in overclock mode. CPU: Intel Core i5-9400f @4.10 GHz x 6 cores (overclock mode). RAM: 64 GB (4x16) DDR4 max speed 3600 MHz. Storage: One m.2 NVMe SSD 256 GB (for operating system) + Two 3 TB Hard Disk Drive (for data storage) Gaming Display Support: 1 GTX 1660 Super graphic card with 6 GB memory and 1,408 cuda cores, supporting max 3 monitors at the same time. Bus max transfer speed 8.0 GB/s (gen3 mode). AI Deep Learning: 4 Tesla K40 AI accelerators each with 12 GB memory and 2,880 cuda cores, dedicating to machine or deep learning, Bus max transfer speed 8.0 GB/s (gen3 mode) each. Power supply safety: One 700 W PSU dedicated to the motherboard and the GTX 1660 monitor GPU. Another 1,000 W PSU dedicated to the Tesla K40 AI accelerators. CPU Cooling: Cooler Master liquid cooler with LED light control. AI Accelerator Cooling: 4 cooling fans at front and 3 cooling fans at back. Structure: Open frame of high strength Al alloy to safeguard your system in an intensive working environment. Power switch: Big button switch with 5 ft flexible extension cable, and LED indicator for hard drive.

This is essentially a GTX 970. Not worth it IMO.

It doesn't have any tensor cores, which means it will run a lot slower for many machine learning tasks, especially the deep learning stuff.

Multi-GPU training is a double-edged sword. If you are at the stage where you are running your code in a iPython notebook then you are almost certainly not going to benefit from the multiple GPUs, and I strongly suspect you'd be better with fewer and larger GPUs, even if training time is prolonged.

The reason I say that is, if we go with PyTorch, you basically have 2 options for multi-GPU training.

- DataParallel - where you clone your model over each GPU, but each one functions independently, and after each 'training step' they pool their data. This has downsides, in that you don't get to process intermediate layer outputs and synchronise your batch normalisation layers - so you can't use it to train 'big' models. It just makes your smaller models train more quickly. However, you can at least use these in a 'normal' training script.

- DistributedDataParallel - this is 'proper' multi-GPU training - you can now train big models and put a little bit of data on each GPU, and have then synchronise their results after each layer. However, this can be very annoying to use - each GPU runs in its own background process which is either spawned or forked (depending on Windows/Linux) and you therefore cannot run it in an iPython notebook, or an interactive Python console. It also makes tracking metrics etc. MUCH harder - because you need to reduce your metrics over each GPU process (because otherwise you get 4 accuracies, 4 mean squared errors etc. if you have 4 GPUs, and each process only sees one of them).

I personally prefer having 1 GPU with 24 Gb RAM over 3 GPU with 12 Gb RAM - because I can have a larger batch size on each GPU, which is VERY VERY advantageous in large models where you can only have small batch sizes, and batch normalisation starts falling down. I'd rather wait 2x as long for a 'better' model to train.

Those who do not recommend, do you have any suggestions for people just getting into the field?