Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What do you use for ML Hosting?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 69 posts

Re: Ask HN: What do you use for ML Hosting?

#21

Post by codeptualize »

thundergolfer wrote: post #18

On Modal.com these 34 lines of code is all you need to serverlessly run BERT text generation inference on an A10G (which has 24GB of GPU memory). No Dockerfile, no YAML, no Terraform or AWS Cloudformation. Just these 34 lines. import modal def download_model(): from transformers import pipeline pipeline("fill-mask", model="bert-base-uncased") CACHE_PATH = "/root/model_cache" # model location in image ENV = modal.Secr…

Love Modal. We use it for data processing, queues, apis, and all sorts of random things. Such a great product!

Re: Ask HN: What do you use for ML Hosting?

#22

Post by edunteman »

senko wrote: post #19
edunteman wrote: post #12

Hey! Would love to have you try https://banana.dev (bias: I'm one of the founders). We run A100s for you and scale 0->1->n->0 on demand, so you only pay for what you use. I'm at erik@banana.dev if you want any help with it :)

Looking at your pricing, is that per seconds of GPU usage or per total seconds the app is running? Eg. might have only a few minutes of usage in an hour and the rest of the time is spent waiting for requests. How's that billed?

https://docs.banana.dev/banana-docs/core-concepts/billing You're only billed for active replica time. Call comes in, we start a replica, it handles the request, it waits around for a 10s (configurable) idle timeout to handle any additional calls, and shuts down if no other calls to serve. The idle timeout is to prevent cold boots when not necessary, but is billed, so you can get closer to pure pay-per-call pricing by reducing idle timeout.

Re: Ask HN: What do you use for ML Hosting?

#24

Post by jaflo »

edunteman wrote: post #12

Hey! Would love to have you try https://banana.dev (bias: I'm one of the founders). We run A100s for you and scale 0->1->n->0 on demand, so you only pay for what you use. I'm at erik@banana.dev if you want any help with it :)

+1 on banana.dev, I used it for a side project and deployed some custom code and it was a good experience! I liked the pricing model (lack of minimums and pay for usage instead of a "plan") and how you can package up whatever code you want.

Re: Ask HN: What do you use for ML Hosting?

#25

Post by therealmarv »

thundergolfer wrote: post #18

On Modal.com these 34 lines of code is all you need to serverlessly run BERT text generation inference on an A10G (which has 24GB of GPU memory). No Dockerfile, no YAML, no Terraform or AWS Cloudformation. Just these 34 lines. import modal def download_model(): from transformers import pipeline pipeline("fill-mask", model="bert-base-uncased") CACHE_PATH = "/root/model_cache" # model location in image ENV = modal.Secr…

btw. HN supports very simple code formatting, just indent by two or more spaces https://news.ycombinator.com/formatdoc

Re: Ask HN: What do you use for ML Hosting?

#26

Post by thundergolfer »

therealmarv wrote: post #25
thundergolfer wrote: post #18

On Modal.com these 34 lines of code is all you need to serverlessly run BERT text generation inference on an A10G (which has 24GB of GPU memory). No Dockerfile, no YAML, no Terraform or AWS Cloudformation. Just these 34 lines. import modal def download_model(): from transformers import pipeline pipeline("fill-mask", model="bert-base-uncased") CACHE_PATH = "/root/model_cache" # model location in image ENV = modal.Secr…

btw. HN supports very simple code formatting, just indent by two or more spaces https://news.ycombinator.com/formatdoc

Ah nice. Thank you. I was using backticks

Re: Ask HN: What do you use for ML Hosting?

#28

Post by samstave »

jaflo wrote: post #24
edunteman wrote: post #12

Hey! Would love to have you try https://banana.dev (bias: I'm one of the founders). We run A100s for you and scale 0->1->n->0 on demand, so you only pay for what you use. I'm at erik@banana.dev if you want any help with it :)

+1 on banana.dev, I used it for a side project and deployed some custom code and it was a good experience! I liked the pricing model (lack of minimums and pay for usage instead of a "plan") and how you can package up whatever code you want.

Kind of funny how cellphones went the opposite way - we all hated "paying for usage (minutes/txts))" and now we want just the Plan.

Re: Ask HN: What do you use for ML Hosting?

#29

Post by edunteman »

jaflo wrote: post #24
edunteman wrote: post #12

Hey! Would love to have you try https://banana.dev (bias: I'm one of the founders). We run A100s for you and scale 0->1->n->0 on demand, so you only pay for what you use. I'm at erik@banana.dev if you want any help with it :)

+1 on banana.dev, I used it for a side project and deployed some custom code and it was a good experience! I liked the pricing model (lack of minimums and pay for usage instead of a "plan") and how you can package up whatever code you want.

Thanks for the +1!

Small note here: our billing is changing within the next month, to up-front payments that apply as a credit balance to your account. It still won't have minimums and you'll have the option to set up auto-refill on your balance, so it will functionally remain pay-as-you-go, but just wanted to add flavor to your comment on the pricing model.

Thanks for using us btw, you rock

Re: Ask HN: What do you use for ML Hosting?

#30

Post by version_five »

My preference is not to have to change my code to use some special framework, and just get access to a gpu machine I can run my stuff on.

I'm assuming you know what you need for a GPU. If you're unsure, consider trying to run inferences on a CPU and see how long it takes and if it could work.

And then just look at price and reliability for a gpu machine with the different cloud providers. Ovh is cheap but the only thing worse than their reliability is their customer service. Various niche players offering V100s used to pop up that were pretty cheap. AWS is more expensive, more reliable, they may still have availability problems. Paperspace looks pretty good. Etc.

Post reply on HN