Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What do you use for ML Hosting?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 69 posts

Re: Ask HN: What do you use for ML Hosting?

#31

Post by alpaca128 »

samstave wrote: post #28
jaflo wrote: post #24

Earlier quoted context omitted.

+1 on banana.dev, I used it for a side project and deployed some custom code and it was a good experience! I liked the pricing model (lack of minimums and pay for usage instead of a "plan") and how you can package up whatever code you want.

Kind of funny how cellphones went the opposite way - we all hated "paying for usage (minutes/txts))" and now we want just the Plan.

I still use that because I rarely use the phone without wifi and don't make a lot of calls. And because it's 2023 I can change to a paid plan for a month at any time in the app, so it's the best of both worlds.

Re: Ask HN: What do you use for ML Hosting?

#36

Post by thundergolfer »

version_five wrote: post #30

My preference is not to have to change my code to use some special framework, and just get access to a gpu machine I can run my stuff on. I'm assuming you know what you need for a GPU. If you're unsure, consider trying to run inferences on a CPU and see how long it takes and if it could work. And then just look at price and reliability for a gpu machine with the different cloud providers. Ovh is cheap but the only th…

> are worth avoiding so you don't get stuck with somebody else's framework.

Modal eng here. Modal is not setup as a framework. Think of more as Python-defined serverless infrastructure that has native support for the Python runtime. This is in some places called "Infrastructure from code", as opposed to "Infrastructure as code" which means just source-controlling K8s YAML and Cloudformation.

A major benefit of this approach is that the cloud becomes part of your dev loop, as opposed to doing `docker build`, `docker push`, `kubectl`, etc just to ship a change to a GPU.

In the script I posted Modal APIs are mixed in with standard Python code for brevity, but many customers just keep their code in their own modules and have a `modal_infra.py` module that defines the serverless infrastructure.

Re: Ask HN: What do you use for ML Hosting?

#37

Post by version_five »

thundergolfer wrote: post #36
version_five wrote: post #30

My preference is not to have to change my code to use some special framework, and just get access to a gpu machine I can run my stuff on. I'm assuming you know what you need for a GPU. If you're unsure, consider trying to run inferences on a CPU and see how long it takes and if it could work. And then just look at price and reliability for a gpu machine with the different cloud providers. Ovh is cheap but the only th…

> are worth avoiding so you don't get stuck with somebody else's framework. Modal eng here. Modal is not setup as a framework. Think of more as Python-defined serverless infrastructure that has native support for the Python runtime. This is in some places called "Infrastructure from code", as opposed to "Infrastructure as code" which means just source-controlling K8s YAML and Cloudformation. A major benefit of this a…

Understood, thanks for clarifying. I'll edit my post.

Re: Ask HN: What do you use for ML Hosting?

#39

Post by naderkhalil »

version_five wrote: post #30

My preference is not to have to change my code to use some special framework, and just get access to a gpu machine I can run my stuff on. I'm assuming you know what you need for a GPU. If you're unsure, consider trying to run inferences on a CPU and see how long it takes and if it could work. And then just look at price and reliability for a gpu machine with the different cloud providers. Ovh is cheap but the only th…

That makes sense, Brev.dev is a really simple way to run your code on a configured GPU without having to change your code. It'll also optimize your GPU to save money when possible.

Re: Ask HN: What do you use for ML Hosting?

#40

Post by zitterbewegung »

Have you tried self hosting? All you need is business internet with a static IP which is quite inexpensive and doing inference can be done on CPU depending what you want to perform inferfence. Also, wherever you are hosting a good rule of thumb is to have at least 1.5 times the amount of regular ram with your VRAM.
Post reply on HN