Does anyone know of any solutions for saving GPU compute during server downtime? Is there a managed solution to turn off a pod and turn it back on when I need it? I'm currently doing model inference and most of the time I'm just paying for compute without serving any user requests.
Ask HN: Cheaper way to do model inference?
1–4 of 4 posts
Re: Ask HN: Cheaper way to do model inference?
#2You are running inference on something like an EC2 instance?
Re: Ask HN: Cheaper way to do model inference?
#3You are running inference on something like an EC2 instance?
Yea exactly that's what I'm doing right now. I've looked into solutions like RunPod serverless, anyone have any experience with it?
Re: Ask HN: Cheaper way to do model inference?
#4Huggingface Inference Endpoints can autoscale to 0 and cost nothing when not being used.