White screen without JS
Show HN: GPU-Accelerated Inference Hosting
21–30 of 36 posts
Re: Show HN: GPU-Accelerated Inference Hosting
#22So is this mainly focused on deployment for applications with high-speed inference requirements? I didn't dive into product in detail. I run my own deep-learning based web-app and inference speed optimization is pretty non-trivial. As far as I know production level speed requirements require use of tensorrt which is definitely not hot-start and requires more than a few minutes to load (i'm not too sure what's going o…
I suspect the audience is more about the GPU hosting aspect, effectively making GPU-based applications “serverless”.
To me, adding GPUs into the devops mix typically increases the complexity significantly, and I would definitely pay money to someone who can just take my model, host it, and let them deal with the complexities around it.
Re: Show HN: GPU-Accelerated Inference Hosting
#23What are the specs of an Nvidia m80?
I’ve never heard of that type before and I wasn’t able to find anything with google.
Furthermore more, the lack of company information (address, company registration nr etc) and the fact that it’s not clear where the servers are located geographically makes me a bit hesitant.
Re: Show HN: GPU-Accelerated Inference Hosting
#24Looking at the examples in the landing page.. so I don't need any kind of authentication to do inference? Anyone can run the models I upload?
Re: Show HN: GPU-Accelerated Inference Hosting
#25Re: Show HN: GPU-Accelerated Inference Hosting
#26Or, do your inference using an AVX-512 CPU:
https://NN-512.com (open source, free software, no dependencies)
With batch size 1, NN-512 is easily 2x faster than TensorFlow and does 27 ResNet50 inferences per second on a c5.xlarge instance. For more unusual networks, like DenseNet or ResNeXt, the performance gap is wider.
Even if you allow TensorFlow to use a larger ResNet50 batch size, NN-512 is easily 1.3x faster.
If you need a few dozen inferences per second per server, this is the cheapest way. And you're not depending on a proprietary solution whose parent company could go out of business in a year.
If you need Transformers instead of convolutions, Fabrice Bellard's LibNC is a good solution: https://bellard.org/libnc/
Re: Show HN: GPU-Accelerated Inference Hosting
#27Earlier quoted context omitted.
Thank you! We don't have any cold start delay! In our custom environment, you can do exactly what you are describing (running both CPU and GPU code). We provide you with access to the GPU and the CUDA libraries installed. It's basically lambda (minus the cold start) with GPU access. We can scale a lot very quickly depending on how much you need.
That's impressive! Are you willing to talk a bit about how this all works? I assume you host the hardware yourself somewhere, which in the days of AWS et al must be pretty tough to pull off, especially with these specs. Where do you get the hardware from these days with the crypto craze?
Yes! A more in-depth blog post is coming soon. We do host the hardware ourselves, for complete control over the GPUs. We found a great infrastructure provider that is also experiencing shortages.
Re: Show HN: GPU-Accelerated Inference Hosting
#28What are the specs of an Nvidia m80? I’ve never heard of that type before and I wasn’t able to find anything with google. Furthermore more, the lack of company information (address, company registration nr etc) and the fact that it’s not clear where the servers are located geographically makes me a bit hesitant.
Sorry that's a typo, they are K80s: https://www.nvidia.com/en-gb/data-center/tesla-k80/
Re: Show HN: GPU-Accelerated Inference Hosting
#29Or, do your inference using an AVX-512 CPU: https://NN-512.com (open source, free software, no dependencies) With batch size 1, NN-512 is easily 2x faster than TensorFlow and does 27 ResNet50 inferences per second on a c5.xlarge instance. For more unusual networks, like DenseNet or ResNeXt, the performance gap is wider. Even if you allow TensorFlow to use a larger ResNet50 batch size, NN-512 is easily 1.3x faster. If…
Oh that's very interesting, how ready for production is it? It only works for TF right?
> If you need a few dozen inferences per second per server, this is the cheapest way. And you're not depending on a proprietary solution whose parent company could go out of business in a year.
Definitely the cheapest way.
We've been in business for more than a year already actually :)
Re: Show HN: GPU-Accelerated Inference Hosting
#30So is this mainly focused on deployment for applications with high-speed inference requirements? I didn't dive into product in detail. I run my own deep-learning based web-app and inference speed optimization is pretty non-trivial. As far as I know production level speed requirements require use of tensorrt which is definitely not hot-start and requires more than a few minutes to load (i'm not too sure what's going o…
We don't use TensorRT at the moment, but it is something that we are exploring.