Do any of the "serverless"/saas model hosting services perform optimizations such as quantization or input micro-batching?
Ask HN: What do you use for ML Hosting?
41–50 of 69 posts
Re: Ask HN: What do you use for ML Hosting?
#42Vast.ai Nobody has better prices.
Another vote for vast.ai, has been around quite a while and I've been using them for shell access to bare metal machines stuffed with GPUs, always had a decent experience.
Re: Ask HN: What do you use for ML Hosting?
#43Wow, looks like there's a ton of choices here I haven't looked at. For iterate.world we use replicate but just added kandinsky from runpod. Thinking about switching everything to runpod because it's 5-10x cheaper and we only use models that they have anyway.
There's one I won't share that's is now defunct but you could use any diffuser's compatible project on Hugging Face, which was such a cool feature. I wish someone (cheap) would implement this!
edit: just looked at banana.dev in this thread, their templates look closest to the HuggingFace integration though I don't think they have webhooks.
Re: Ask HN: What do you use for ML Hosting?
#44I’m using a docker container on Ubuntu, which is on my home lab that’s an esxi 6.5 hypervisor.
Going to be building a new machine with a few hundred GB of ram and then, at some point in the next 6 months, looking at getting a good GPU with a bunch of vRAM.
Wrapped the thing in a flask app so I can expose APIs I build out.
Re: Ask HN: What do you use for ML Hosting?
#45We use Sagemaker at work because AWS. I don't really like their style of APIs but it works.
Re: Ask HN: What do you use for ML Hosting?
#46Wow, looks like there's a ton of choices here I haven't looked at. For iterate.world we use replicate but just added kandinsky from runpod. Thinking about switching everything to runpod because it's 5-10x cheaper and we only use models that they have anyway. There's one I won't share that's is now defunct but you could use any diffuser's compatible project on Hugging Face, which was such a cool feature. I wish someon…
Founder of Replicate here. Also YC founder (W20). :)
It's also worth noting that we bill by the second for how long your prediction is running, and we don't bill for any idle time, so in practice Replicate works out cheaper for many workloads. We can give discounts if you're putting through a decent amount of traffic. We should be able to match Runpod's pricing.
Drop me an email: ben@replicate.com
Re: Ask HN: What do you use for ML Hosting?
#47I have had good experience with Replicate and Runpod. Replicate seems to be nicer but has very bad cold boot issue. Runpod is great once you have an app set up! I use mix of both for my side project: https://trainengine.ai
Founder of Replicate here. Looks like you're using DreamBooth for TrainEngine. We have a beta version of really fast cold boots for DreamBooth trainings. I'll drop you an email to get you set up with it.
We've got some big cold improvements rolling out across everything soon. We can also just keep models switched on to avoid cold boots entirely.
Re: Ask HN: What do you use for ML Hosting?
#48Founder of https://replicate.com/ here, which has been mentioned a few times. Happy to help you get set up. :) ben@replicate.com
Re: Ask HN: What do you use for ML Hosting?
#49Hey! Would love to have you try https://banana.dev (bias: I'm one of the founders). We run A100s for you and scale 0->1->n->0 on demand, so you only pay for what you use. I'm at erik@banana.dev if you want any help with it :)
Looks great. It appears aimed at the inference use case rather than training, yes?
Re: Ask HN: What do you use for ML Hosting?
#50Hey! Would love to have you try https://banana.dev (bias: I'm one of the founders). We run A100s for you and scale 0->1->n->0 on demand, so you only pay for what you use. I'm at erik@banana.dev if you want any help with it :)
Looks great. It appears aimed at the inference use case rather than training, yes?
yeah, we're optimizing the infra for realtime inference, though people definitely still do run training on us, with a weights upload implemented at the end of your handler.