Show HN: GPU-Accelerated Inference Hosting
inferrd.com
Show HN: GPU-Accelerated Inference Hosting
1–10 of 36 posts
Re: Show HN: GPU-Accelerated Inference Hosting
#2[deleted]
Re: Show HN: GPU-Accelerated Inference Hosting
#3Does it need to reinitialize for each request or is there a warm start / cold start model like lambda? I don't really understand how you can charge per request.
Re: Show HN: GPU-Accelerated Inference Hosting
#4Does it need to reinitialize for each request or is there a warm start / cold start model like lambda? I don't really understand how you can charge per request.
I guess GPU loading is quick? Like ~10 seconds?
Re: Show HN: GPU-Accelerated Inference Hosting
#5could I run spleeter on it?
Re: Show HN: GPU-Accelerated Inference Hosting
#6Does it need to reinitialize for each request or is there a warm start / cold start model like lambda? I don't really understand how you can charge per request.
There is no cold start! We keep your service hot all the time.
Re: Show HN: GPU-Accelerated Inference Hosting
#7could I run spleeter on it?
Yes absolutely, you can run almost anything in our custom environment.
Re: Show HN: GPU-Accelerated Inference Hosting
#8This is nice, and I've wanted this kind of thing repeatedly over the last 5 years! I think you often want to run little bits of CPU-based code in addition to your deep learning graph. So I think a better deployment model might be basically Lambda but with CUDA access... or something like that.
The factors that I think would make this service most valuable are low cost (think, lower than GPU's on AWS or similar, even at scale), high burst capability from cold start (1000QPS is a good target), and of course low cold start delays (This led me down a rabbit hole in years past and the technical solution seems to be generally, the ability to swap models in and out of GPU ram very quickly. Possibly using NVIDIA's unified memory subsystem.
Re: Show HN: GPU-Accelerated Inference Hosting
#9Does it need to reinitialize for each request or is there a warm start / cold start model like lambda? I don't really understand how you can charge per request.
The pricing appears to be static per model with a ceiling on the monthly request count, not charged per request.
Edit: Actually, I didn't spot the free tier of 1000 requests. I wonder how you avoid the problem of a lot of users leaving defunct/disused models running while still keeping them hot - presumably some kind of limit to the model count?
Re: Show HN: GPU-Accelerated Inference Hosting
#10This is nice, and I've wanted this kind of thing repeatedly over the last 5 years! I think you often want to run little bits of CPU-based code in addition to your deep learning graph. So I think a better deployment model might be basically Lambda but with CUDA access... or something like that. The factors that I think would make this service most valuable are low cost (think, lower than GPU's on AWS or similar, even…
Thank you!
We don't have any cold start delay! In our custom environment, you can do exactly what you are describing (running both CPU and GPU code). We provide you with access to the GPU and the CUDA libraries installed. It's basically lambda (minus the cold start) with GPU access.
We can scale a lot very quickly depending on how much you need.