What I've done recently is buying used servers and 10G/40g network switches from Ebay and rented a colo(colocation) rack, which can be had from $500-$1000 per month per rack. This often includes 100mbit++ internet, power, cooling and more (https://en.wikipedia.org/wiki/Colocation_centre)
This has been the most cost effective way for me to deploy for example Hadoop, Ceph, Elasticsearch, Huge Varnish cache solutions. While I understand this is not for everyone, it is absolutely something to consider if you have a strong devops team. I do all this myself as I've build my own automation tools over the years that simplify setup and monitoring.
On the other hand, I use AWS for GPU instances as I find it very cost effective because it is very easy to scale up and down by demand. And investing in this kind of hardware is expensive/risky. The power efficiency / performance is still following Moores law for each year for GPU's, and I expect new hardware that is better optimized for neural networks / machine learning is just around the corner.