Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What do you use for ML Hosting?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 69 posts

Re: Ask HN: What do you use for ML Hosting?

#51

Post by edunteman »

rgbrgb wrote: post #43

Wow, looks like there's a ton of choices here I haven't looked at. For iterate.world we use replicate but just added kandinsky from runpod. Thinking about switching everything to runpod because it's 5-10x cheaper and we only use models that they have anyway. There's one I won't share that's is now defunct but you could use any diffuser's compatible project on Hugging Face, which was such a cool feature. I wish someon…

Hey! Banana founder here. Explicit webhook support coming out soon, though one could always add an http POST request to their webhook endpoint at the end of their handler to send the data that way rather than awaiting the results from the client. It'd take some customization, but you're into our templates, you can click the github icon in the UI to see the source repo, fork it, add the HTTP POST call at the end of the handler, and then deploy that to Banana as a custom repo.

Re: Ask HN: What do you use for ML Hosting?

#53

Post by tasn »

edunteman wrote: post #51
rgbrgb wrote: post #43

Wow, looks like there's a ton of choices here I haven't looked at. For iterate.world we use replicate but just added kandinsky from runpod. Thinking about switching everything to runpod because it's 5-10x cheaper and we only use models that they have anyway. There's one I won't share that's is now defunct but you could use any diffuser's compatible project on Hugging Face, which was such a cool feature. I wish someon…

Hey! Banana founder here. Explicit webhook support coming out soon, though one could always add an http POST request to their webhook endpoint at the end of their handler to send the data that way rather than awaiting the results from the client. It'd take some customization, but you're into our templates, you can click the github icon in the UI to see the source repo, fork it, add the HTTP POST call at the end of th…

Let us know if you need help with webhooks, we make it super easy...

https://www.svix.com/

Re: Ask HN: What do you use for ML Hosting?

#54

Post by ArtWomb »

thundergolfer wrote: post #18

On Modal.com these 34 lines of code is all you need to serverlessly run BERT text generation inference on an A10G (which has 24GB of GPU memory). No Dockerfile, no YAML, no Terraform or AWS Cloudformation. Just these 34 lines. import modal def download_model(): from transformers import pipeline pipeline("fill-mask", model="bert-base-uncased") CACHE_PATH = "/root/model_cache" # model location in image ENV = modal.Secr…

Perfect sell. I'm signing-up just from this comment alone ;)

Re: Ask HN: What do you use for ML Hosting?

#55

Post by samstave »

alpaca128 wrote: post #31
samstave wrote: post #28

Earlier quoted context omitted.

Kind of funny how cellphones went the opposite way - we all hated "paying for usage (minutes/txts))" and now we want just the Plan.

I still use that because I rarely use the phone without wifi and don't make a lot of calls. And because it's 2023 I can change to a paid plan for a month at any time in the app, so it's the best of both worlds.

I have the cheapest plan, 5GB, and an unlimited data Android tablet for a total of $30+$15=$45 for both...

I pay my cell bill a year in advance - then I never have to worry about a bill ever.

Re: Ask HN: What do you use for ML Hosting?

#56

Post by RGDub »

I've been very happy with Genesis Cloud (www.genesiscloud.com) - they have worked with me on getting additional GPU capacity and have very reasonable prices. 0.7 USD/HR for a Nvidia GeForce RTX 3090. They give you $15 in credits for starting an account but you can get $50 with this referral code: https://gnsiscld.co/x5tpz .

Re: Ask HN: What do you use for ML Hosting?

#58

Post by chiragjn »

Disclaimer: I work at Truefoundry

You can give us a shot at https://truefoundry.com We are a general purpose ML Deployments platform which works on top of your existing Kubernetes clusters (AWS EKS, GCP GKE or Azure AKS) abstracting away the complexity of dealing with cloud providers and Kubernetes. We support Services for ML web apps, APIs, Jobs for ML training jobs, Model Registry for storing models, Model Servers for no code model deployments. (Our platform can be partially or completely self hosted for privacy and compliance)

Adding one or more GPUs (V100, T4, A10, A100, etc) is simply one extra line https://docs.truefoundry.com/docs/gpus#adding-gpu-to-service...

Examples:

- Stable Diffusion with Gradio: https://github.com/truefoundry/truefoundry-examples/tree/mai...

- GPT-J 6B fp16 with FastAPI: https://github.com/truefoundry/truefoundry-examples/tree/mai...

Re: Ask HN: What do you use for ML Hosting?

#59

Post by mathewvp »

bootsmann wrote: post #11
smoldesu wrote: post #3

I'm currently running a Discord bot with a 7B model off a free Oracle Ampere instance with their Pytorch Accelerated[0] image. It's not terribly fast, but totally usable for group chats that want to interrogate an AI. If you're doing some sort of offline processing or non-time-imperative operation, something like this might be worth looking into. [0] https://cloudmarketplace.oracle.com/marketplace/en_US/adf.ta...

Oracles free forever tier is so underrated. They just throw half a startup at you at no cost.

They also shutdown your instances after a while and ask you to upgrade to paid tier.

Re: Ask HN: What do you use for ML Hosting?

#60

Post by agutgutia1991 »

chiragjn wrote: post #58

Disclaimer: I work at Truefoundry You can give us a shot at https://truefoundry.com We are a general purpose ML Deployments platform which works on top of your existing Kubernetes clusters (AWS EKS, GCP GKE or Azure AKS) abstracting away the complexity of dealing with cloud providers and Kubernetes. We support Services for ML web apps, APIs, Jobs for ML training jobs, Model Registry for storing models, Model Servers…

PS: (I am one of the founders) - you can write to us at founders@truefoundry.com We can help understand your use case and try to suggest whatever is best from whatever is available in the ML Serving ecosystem.
Post reply on HN