I'm trying to setup server to run ML inferences. I need to provision a somewhat beefy gpu with a decent amount of RAM (8-16 GB). Does anyone here have personal experience and recommendations about the various companies operating in this space?
Ask HN: What do you use for ML Hosting?
1–10 of 69 posts
Re: Ask HN: What do you use for ML Hosting?
#2For serverless: check the list I posted here https://news.ycombinator.com/item?id=34742087 (I ended up using Banana, it was fine)
For non-serverless, some to check out are these (though likely all overkill if you just need a single GPU)
vast.ai
Lambda labs
Re: Ask HN: What do you use for ML Hosting?
#3I'm currently running a Discord bot with a 7B model off a free Oracle Ampere instance with their Pytorch Accelerated[0] image. It's not terribly fast, but totally usable for group chats that want to interrogate an AI. If you're doing some sort of offline processing or non-time-imperative operation, something like this might be worth looking into.
[0] https://cloudmarketplace.oracle.com/marketplace/en_US/adf.ta...
Re: Ask HN: What do you use for ML Hosting?
#4Here are some candidates:
- HuggingFace Inference Endpoints: https://huggingface.co/inference-endpoints
- Amazon SageMaker: https://aws.amazon.com/sagemaker/
- Replicate: https://replicate.com/
The first two are more customizable than the last. SageMaker is the cheapest.
Re: Ask HN: What do you use for ML Hosting?
#5I'm currently running a Discord bot with a 7B model off a free Oracle Ampere instance with their Pytorch Accelerated[0] image. It's not terribly fast, but totally usable for group chats that want to interrogate an AI. If you're doing some sort of offline processing or non-time-imperative operation, something like this might be worth looking into. [0] https://cloudmarketplace.oracle.com/marketplace/en_US/adf.ta...
Does that use all 4 OCPUs / 24GB memory?
Re: Ask HN: What do you use for ML Hosting?
#6I'm currently running a Discord bot with a 7B model off a free Oracle Ampere instance with their Pytorch Accelerated[0] image. It's not terribly fast, but totally usable for group chats that want to interrogate an AI. If you're doing some sort of offline processing or non-time-imperative operation, something like this might be worth looking into. [0] https://cloudmarketplace.oracle.com/marketplace/en_US/adf.ta...
Does that use all 4 OCPUs / 24GB memory?
It can! I'm using 2 cores per request though, and I've got memory to spare.
Re: Ask HN: What do you use for ML Hosting?
#7Vultr GPU: https://www.vultr.com/products/cloud-gpu/
Re: Ask HN: What do you use for ML Hosting?
#8I'm currently running a Discord bot with a 7B model off a free Oracle Ampere instance with their Pytorch Accelerated[0] image. It's not terribly fast, but totally usable for group chats that want to interrogate an AI. If you're doing some sort of offline processing or non-time-imperative operation, something like this might be worth looking into. [0] https://cloudmarketplace.oracle.com/marketplace/en_US/adf.ta...
what discord bot? :)
Re: Ask HN: What do you use for ML Hosting?
#9Vast.ai
Nobody has better prices.
Re: Ask HN: What do you use for ML Hosting?
#10Check out JetML.com (I'm the founder). Happy to help get you started with a demo if you want to reach out nick@jetml.com.