Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

github.com

Post reply on HN
Print view View original

21–30 of 72 posts

Re: Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

#21

Post by rgthelen »

johnthecto wrote: post #17

So this sounds like an application layer approach, maybe just shy of a replit or base44, with the twist that you can own the pipeline. While there's something to that, I think there are some further questions around differentiation that need to be answered. I think the biggest challenge is going to be the beachead: what client demographic has the cash to want to own the pipeline and not use SaaS, but doesn't have the…

Yeah, that’s a fair framing — it is kind of an “application layer” for AI orchestration, but focused on ownership and portability instead of just convenience.

Yeah, the beachead will be our biggest issue - where to find first hard-core users. I was thinking legal (they have a need for AI, but data cannot leave their servers), healthcare (same as legal, but more regualtions), and government (not right now, but normally have deep pockets).

What do you think is a good starting place?

Re: Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

#22

Post by rgthelen »

outfinity wrote: post #18

Hope you are right and we decentralize AI properly...

The hardest part, honestly, is the runtime. How do we make it super easy actually to deploy this. We are still working on that. Where do you see a few good places to focus at first? I was thinking AWS and Google, since both have good GPU pricing models, but I am probably missing a few good ones!

Re: Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

#23

Post by mhamann »

johnthecto wrote: post #17

So this sounds like an application layer approach, maybe just shy of a replit or base44, with the twist that you can own the pipeline. While there's something to that, I think there are some further questions around differentiation that need to be answered. I think the biggest challenge is going to be the beachead: what client demographic has the cash to want to own the pipeline and not use SaaS, but doesn't have the…

I think that enterprises and small businesses alike need stuff like this, regardless of whether they're software companies or some other vertical like healthcare or legal. I worked at IBM for over a decade and it was always preferable to start with an open source framework if it fit your problem space, especially for internal stuff. We shipped products with components built on Elastic, Drupal, Express, etc.

You could make the same argument for Kubernetes. If you have the cash and the team, why not build it yourself? Most don't have the expertise or the time to find/train the people who do.

People want AI that works out of the box on day one. Not day 100.

Re: Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

#26

Post by rachelradulo »

serjester wrote: post #25

Congrats on the launch. YC 2022? I'm assuming this was a pivot - what lead to it and how do you guys plan on making money long term?

Yep! We were working on an authentication startup (https://news.ycombinator.com/item?id=30615352) and built it to $1.5M in ARR, but then we saw even a bigger pain point; local AI is hard. When we tried building a corporate knowledge base with RAG and local models, we hit the same wall: a painful gap between prototype and production.

Production-ready enterprise AI requires solving model management, RAG pipelines, model fine-tuning, prompt engineering, failover, cost optimization, and deployment orchestration. You can’t just be good at one or two of these, you have to be great at all of them or your project won't succeed. And so Llamafarm was born!

Monetization-wise - We’re open source and free forever, with revenue coming from enterprise support, managed deployments, and compliance packages—basically, companies pay for confidence, not code.

Re: Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

#27

Post by rachelradulo »

bityard wrote: post #24

Open source but backed by venture capital, so what is your monetization strategy?

Fair question. The core will always stay open source and free. We’ll monetize around it with things like managed hosting, enterprise support, and compliance options (HIPAA, SOC2, etc). Basically, we make money when teams want someone to stand behind it in production, not for using the software itself. But let us know if you have other ideas! We're still new to open source

Re: Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

#28

Post by rgthelen »

jochalek wrote: post #14

Very cool to see a serious local first effort. Looking back at how far local models have come I definitely believe their usefulness combined with RAG or in domain specific contexts is soon to be (or already is) on par with general purpose gpt5-like massive parameter cloud models. The ability to generate quality responses without having to relinquish private data to the cloud used to be a pipedream. It's exciting to s…

What are a few use-cases you want to see this used for?

Re: Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

#29

Post by singlepaynews »

Very cool. I jumped in here thinking it was gonna be something else though: a packaged service for distributing on-prem model running across multiple GPUs.

I'm basically imagining a vast.ai type deployment of an on-prem GPT; assuming that most infra is consumer GPUs on consumer devices, the idea of running the "company cluster" as combined compute of the company's machines

Re: Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

#30

Post by mhamann »

singlepaynews wrote: post #29

Very cool. I jumped in here thinking it was gonna be something else though: a packaged service for distributing on-prem model running across multiple GPUs. I'm basically imagining a vast.ai type deployment of an on-prem GPT; assuming that most infra is consumer GPUs on consumer devices, the idea of running the "company cluster" as combined compute of the company's machines

Great point. I can see how you'd land there. Also a great idea! xD

Maybe a better descriptor is "self-sovereign AI?" "Self-hosted AI?"

Post reply on HN