Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

github.com

Post reply on HN
Print view View original

61–70 of 72 posts

Re: Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

#61

Post by Unheard3610 »

johnthecto wrote: post #60
rgthelen wrote: post #21

Earlier quoted context omitted.

Yeah, that’s a fair framing — it is kind of an “application layer” for AI orchestration, but focused on ownership and portability instead of just convenience. Yeah, the beachead will be our biggest issue - where to find first hard-core users. I was thinking legal (they have a need for AI, but data cannot leave their servers), healthcare (same as legal, but more regualtions), and government (not right now, but normall…

Like your ideas around legal and Healthcare. Both sectors where you've got interest and money. Another way might be to look at partnering with a service org who does transformation/modernization as a way to accelerate delivery. Maybe MSPs? They're always trying to figure out how to lean out. An idea might be to try and get a vertical sooner rather than later. The only thing better than an interested lawyer would be a…

Yeah! Or maybe the GPU rental and lab market? (At least those that aren’t already fully hyperscaler captured?) like we were chatting about John?

Re: Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

#62

Post by Unheard3610 »

johnthecto wrote: post #60
rgthelen wrote: post #21

Earlier quoted context omitted.

Yeah, that’s a fair framing — it is kind of an “application layer” for AI orchestration, but focused on ownership and portability instead of just convenience. Yeah, the beachead will be our biggest issue - where to find first hard-core users. I was thinking legal (they have a need for AI, but data cannot leave their servers), healthcare (same as legal, but more regualtions), and government (not right now, but normall…

Like your ideas around legal and Healthcare. Both sectors where you've got interest and money. Another way might be to look at partnering with a service org who does transformation/modernization as a way to accelerate delivery. Maybe MSPs? They're always trying to figure out how to lean out. An idea might be to try and get a vertical sooner rather than later. The only thing better than an interested lawyer would be a…

Follow on, great question (separately from John) can it run on Vulkan? He was saying he has a hard time finding stuff that doesn’t only run on CUDA or ROCm and sees this as a huge opportunity

Re: Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

#64

Post by rgthelen »

johnthecto wrote: post #63

Heya - one blue sea question. Where are you on Vulkan support? Hard to find good stacks to use with all this great intel and non-rocm amd hardware. Might be a good angle too rather than chasing the usual Nvidia money train.

Funny you bring it up. We shipped Vulkan support TODAY through a tight integration Lemonade (https://lemonade-server.ai).

We now support AMD, Intel, CPU, and Cuda/Nvidia.

Hit me up if you want a walk through - this is in dev right now (you have to pull down the repo to run it), but we'll ship it as a part of our next release.

https://docs.llamafarm.dev/docs/models#lemonade-runtime

Re: Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

#65

Post by rgthelen »

gus_22 wrote: post #59

Would like to connect. I've got a YC colleague I have asked to setup an intro. Agree with John's point below about the chicken and egg scenario. Our community flagged your post. Fundamental problems like who cares, who pays reinforce why this hasn't already been done (well and at scale). And why this time might be the right time!

Appreciate that — and totally agree. The “who cares / who pays” question is exactly why this hasn’t scaled before.

Our bet is that the timing’s finally right: local inference, smaller and more powerful open models (Qwen, Granite, Deepseek), and enterprise appetite for control have all converged. We’re working with large enterprises (especially in regulated industries) where innovation teams need to build and run AI systems internally, across mixed or disconnected environments.

That’s the wedge — not another SaaS, but a reproducible, ownable AI layer that can actually move between cloud, edge, and air-gapped. Just reach out, no intro needed - robert @ llamafarm.dev

Re: Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

#66

Post by rgthelen »

Unheard3610 wrote: post #62
johnthecto wrote: post #60

Earlier quoted context omitted.

Like your ideas around legal and Healthcare. Both sectors where you've got interest and money. Another way might be to look at partnering with a service org who does transformation/modernization as a way to accelerate delivery. Maybe MSPs? They're always trying to figure out how to lean out. An idea might be to try and get a vertical sooner rather than later. The only thing better than an interested lawyer would be a…

Follow on, great question (separately from John) can it run on Vulkan? He was saying he has a hard time finding stuff that doesn’t only run on CUDA or ROCm and sees this as a huge opportunity

Yes! Vulcan support is in dev (will be in the next versioned release, probably tomorrow).

You can pull down the repo and run a few easy commands to get inference up and running.

https://docs.llamafarm.dev/docs/models#lemonade-runtime

Re: Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

#67

Post by rgthelen »

johnthecto wrote: post #60
rgthelen wrote: post #21

Earlier quoted context omitted.

Yeah, that’s a fair framing — it is kind of an “application layer” for AI orchestration, but focused on ownership and portability instead of just convenience. Yeah, the beachead will be our biggest issue - where to find first hard-core users. I was thinking legal (they have a need for AI, but data cannot leave their servers), healthcare (same as legal, but more regualtions), and government (not right now, but normall…

Like your ideas around legal and Healthcare. Both sectors where you've got interest and money. Another way might be to look at partnering with a service org who does transformation/modernization as a way to accelerate delivery. Maybe MSPs? They're always trying to figure out how to lean out. An idea might be to try and get a vertical sooner rather than later. The only thing better than an interested lawyer would be a…

Yeah, totally fair. The “horizontal orchestration” story only goes so far — at some point you need vertical depth.

We’re starting with regulated enterprises (defense, healthcare, legal, fintech) where control and compliance actually matter. The same YAML-defined system can run in AWS, in a hospital, or fully air-gapped — no lock-in, no data leaving.

We’re building a few sample recipes to show it in action:

Legal: doc analysis + precedent search with local vector DBs

Healthcare: privacy-preserving RAG over clinical notes

Industrial/Defense: offline sensor or alert agents that sync later

Partnering with MSPs and modernization firms seems like the obvious path — they already have the relationships and budgets, and LlamaFarm gives them something repeatable to deploy.

Still figuring that out though — what’s the best way to actually get into the MSP funnel?

Re: Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

#68

Post by rgthelen »

Unheard3610 wrote: post #61
johnthecto wrote: post #60

Earlier quoted context omitted.

Like your ideas around legal and Healthcare. Both sectors where you've got interest and money. Another way might be to look at partnering with a service org who does transformation/modernization as a way to accelerate delivery. Maybe MSPs? They're always trying to figure out how to lean out. An idea might be to try and get a vertical sooner rather than later. The only thing better than an interested lawyer would be a…

Yeah! Or maybe the GPU rental and lab market? (At least those that aren’t already fully hyperscaler captured?) like we were chatting about John?

We are adding continuous model fine-tuning soon, and being able to bring extra horsepower into training is an opportunity. There is also a genuine opportunity to do the same thing with shared resources inside an internal server or VPC; timing and utilizing resources, such as GPUs, during off-hours to train, improve, etc., is a never-recovered opportunity that many enterprises leave on the table.

Any GPU that is not being used at 80% capacity needs to be put to work; we have a lot of work that can be done. (A lot of industries cannot lease their GPUs to the public due to regulatory issues).

Re: Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

#69

Post by rgthelen »

serjester wrote: post #25

Congrats on the launch. YC 2022? I'm assuming this was a pivot - what lead to it and how do you guys plan on making money long term?

Oh man, yeah, big pivot.

We still are Rownd (https://rownd.com); but we see the writing on the wall. SaaS Software that helps with "hard code" problems is going the way of the dodo.

What used to take a few weeks and was hard to maintain can be down with Codex in the background. We are still bringing in decent revenue and have no plans to sunset, we are just not investing in it.

We all have IBM backgrounds - not sexy, but we are good at running complex software in customer datacenters and in their clouds. AI is going to have to run locally to extract full value from regulated industries.

We are using a services + support model, likely going vertical (legal, healthcare, and we had some good momentum in the US Gov until 1 October :)).

Re: Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

#70

Post by rgthelen »

swyx wrote: post #56
jbstack wrote: post #49

Earlier quoted context omitted.

This whole comment is predicated on the idea that something is only worthwhile if it will lead to a "venture scale startup" and that to do anything else is to "waste the best years of your life". I, for one, am glad that not everyone shares your ethos. Conversely, I'm also glad that there are people out there "building agents". Diversity is a good thing. Encouraging everyone to only do one thing is a bad thing.

bros identify as YC W22, this is a reasonable expectation

Yeah, we are in it to win it.
Post reply on HN