Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What do you use for ML Hosting?

news.ycombinator.com

Post reply on HN
Print view View original

61–69 of 69 posts

Re: Ask HN: What do you use for ML Hosting?

#61

Post by Rastonbury »

bfirsh wrote: post #47
asadm wrote: post #23

I have had good experience with Replicate and Runpod. Replicate seems to be nicer but has very bad cold boot issue. Runpod is great once you have an app set up! I use mix of both for my side project: https://trainengine.ai

Founder of Replicate here. Looks like you're using DreamBooth for TrainEngine. We have a beta version of really fast cold boots for DreamBooth trainings. I'll drop you an email to get you set up with it. We've got some big cold improvements rolling out across everything soon. We can also just keep models switched on to avoid cold boots entirely.

Seconding the cold boot issue, hoping it's fixed I've had to look elsewhere because I can't have users waiting 3 min for a 5-15 second inference unexpectedly, I must say I do like the site and product!

Re: Ask HN: What do you use for ML Hosting?

#63

Post by quickthrower2 »

thundergolfer wrote: post #36
version_five wrote: post #30

My preference is not to have to change my code to use some special framework, and just get access to a gpu machine I can run my stuff on. I'm assuming you know what you need for a GPU. If you're unsure, consider trying to run inferences on a CPU and see how long it takes and if it could work. And then just look at price and reliability for a gpu machine with the different cloud providers. Ovh is cheap but the only th…

> are worth avoiding so you don't get stuck with somebody else's framework. Modal eng here. Modal is not setup as a framework. Think of more as Python-defined serverless infrastructure that has native support for the Python runtime. This is in some places called "Infrastructure from code", as opposed to "Infrastructure as code" which means just source-controlling K8s YAML and Cloudformation. A major benefit of this a…

Glad to see this concept being used. I was thinking the other day how odd it is to have IaC then keep the code siloed from the app code. Why doesn’t the app make it’s own infra? I think you have articulated how that would work.

Re: Ask HN: What do you use for ML Hosting?

#64

Post by silksowed »

aaronharnly wrote: post #49
edunteman wrote: post #12

Hey! Would love to have you try https://banana.dev (bias: I'm one of the founders). We run A100s for you and scale 0->1->n->0 on demand, so you only pay for what you use. I'm at erik@banana.dev if you want any help with it :)

Looks great. It appears aimed at the inference use case rather than training, yes?

how were you able to tell this? still trying to understand what infra is better used for inference (say realtime image category matching) vs training (feeding a chatbot huge sums of data)

Re: Ask HN: What do you use for ML Hosting?

#66

Post by GC_tris »

Genesis Cloud (https://www.genesiscloud.com/pricing).

Disclaimer: I am the CTO ;)

Why use us?

Competitive prices (billing by the minute, only pay when you actually run an instance). High reliability (professional DCs, customized hardware to suit requirements). Good connectivity (traffic is also free, no in-/egress fees). High security level (full VMs with dedicated GPUs with proper separation of customers instead of shared hosts with docker). Free storage. A great support team. Green energy (no greenwashing by carbon offsetting, we use energy sources that are renewable and carbon free at the source (geothermal/hydro)).

I could go on... Would love it if you just try our services, after sign up there are free credits available for risk free testing.

Re: Ask HN: What do you use for ML Hosting?

#67

Post by ronyfadel »

thundergolfer wrote: post #18

On Modal.com these 34 lines of code is all you need to serverlessly run BERT text generation inference on an A10G (which has 24GB of GPU memory). No Dockerfile, no YAML, no Terraform or AWS Cloudformation. Just these 34 lines. import modal def download_model(): from transformers import pipeline pipeline("fill-mask", model="bert-base-uncased") CACHE_PATH = "/root/model_cache" # model location in image ENV = modal.Secr…

How does Modal perform, latency-wise? Since it has to spin up an instance/GPU, does it take long to return results?

Re: Ask HN: What do you use for ML Hosting?

#68

Post by Monika1012 »

chiragjn wrote: post #58

Disclaimer: I work at Truefoundry You can give us a shot at https://truefoundry.com We are a general purpose ML Deployments platform which works on top of your existing Kubernetes clusters (AWS EKS, GCP GKE or Azure AKS) abstracting away the complexity of dealing with cloud providers and Kubernetes. We support Services for ML web apps, APIs, Jobs for ML training jobs, Model Registry for storing models, Model Servers…

Love TrueFoundry! We use it for Infra provisioning on our own cloud and deploying the ML Models behind a choice of a specific model server. Pricing model is also good for early start-ups :)

Re: Ask HN: What do you use for ML Hosting?

#69

Post by chpatrick »

thundergolfer wrote: post #18

On Modal.com these 34 lines of code is all you need to serverlessly run BERT text generation inference on an A10G (which has 24GB of GPU memory). No Dockerfile, no YAML, no Terraform or AWS Cloudformation. Just these 34 lines. import modal def download_model(): from transformers import pipeline pipeline("fill-mask", model="bert-base-uncased") CACHE_PATH = "/root/model_cache" # model location in image ENV = modal.Secr…

Modal's usability is amazing, I'm just a bit wary because they use AWS but somehow their prices are lower than that of AWS's GPU machines.
Post reply on HN