Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

github.com

Post reply on HN
Print view View original

41–50 of 72 posts

Re: Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

#41

Post by rgthelen »

olokobayusuf wrote: post #34
singlepaynews wrote: post #29

Very cool. I jumped in here thinking it was gonna be something else though: a packaged service for distributing on-prem model running across multiple GPUs. I'm basically imagining a vast.ai type deployment of an on-prem GPT; assuming that most infra is consumer GPUs on consumer devices, the idea of running the "company cluster" as combined compute of the company's machines

We're building something closer to this at Muna: https://docs.muna.ai . Check us out and let me know what you think!

Let me know when you open source it; I think there is a place for this and I think we could integrate it as a plug in pretty easily into the LlamaFarm framework :)

Re: Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

#42

Post by mhamann »

rgthelen wrote: post #40
jochalek wrote: post #39

Earlier quoted context omitted.

Sounds like something that could be implemented with llm-d, though I've not experimented with it. https://llm-d.ai/blog/intelligent-inference-scheduling-with-...

Yeah, I don't see why we could not integrate that. I think that is the next step as we move our workloads to production.

`lf deploy` here we come!

Re: Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

#43

Post by ivape »

We built a bunch of AI demos but they were impossible to get to production. It would work perfectly on our laptop, but when we deployed it, something broke, and RAG would degrade.

How did RAG degrade when it went to prod? Do you mean your prod server had throughput issues?

Re: Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

#44

Post by Eisenstein »

How do you deal with the space continually evolving? Like, MCP changed major ways over the course of a few months, new models are released with significant capability upgrades every month, inference engines like llamacpp get updated multiple times a day. But organizations want to setup their frameworks and then maintain them. Will this let them do that?

Re: Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

#45

Post by mhamann »

Eisenstein wrote: post #44

How do you deal with the space continually evolving? Like, MCP changed major ways over the course of a few months, new models are released with significant capability upgrades every month, inference engines like llamacpp get updated multiple times a day. But organizations want to setup their frameworks and then maintain them. Will this let them do that?

Yes, our goal is to provide a stable, open source platform on top of the cutting-edge AI tools. We can systematically update dependencies as needed and ensure that outputs meet quality requirements.

We also have plans for eval features in the product so that users can measure the quality of changes over time, whether to their own project configs or actual LlamaFarm updates.

Yes, all that's a bit hand-wavy, I know. :-) But we do recognize the problem and have real ideas on solutions. But execution is everything. ;-)

Re: Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

#46

Post by mhamann »

ivape wrote: post #43

We built a bunch of AI demos but they were impossible to get to production. It would work perfectly on our laptop, but when we deployed it, something broke, and RAG would degrade. How did RAG degrade when it went to prod? Do you mean your prod server had throughput issues?

Multiple areas of degradation. Typically, you don't ship a dataset to prod and then never change it. You want the system to continue to learn and improve as new data is available. This can create performance issues as the dataset grows in size. But also, your model's performance in terms of quality can degrade over time if you're not constantly evaluating its responses. This can occur because of new info within RAG, a model swap/upgrade, or changes to prompts. Keeping all of those knives in the air is tricky. We're hoping we can solve a bunch of pain points around this so that reliable AI systems are accessible to anyone.

Re: Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

#47

Post by swyx »

yeah guys look i wish you well and respect for launching and all but this is just not going to ever be a venture scale startup and you should calibrate your expectations. you could be wasting the best years of your life being a wrapper of a wrapper of a wrapper and competing on developer experience in open source for no money, or you could be building agents.

build agents. please.

Re: Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

#48

Post by swyx »

swyx wrote: post #47

yeah guys look i wish you well and respect for launching and all but this is just not going to ever be a venture scale startup and you should calibrate your expectations. you could be wasting the best years of your life being a wrapper of a wrapper of a wrapper and competing on developer experience in open source for no money, or you could be building agents. build agents. please.

more context - https://x.com/swyx/status/1904256213661192405 i really mean it in as non mean spirited a way as possible

Re: Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

#49

Post by jbstack »

swyx wrote: post #47

yeah guys look i wish you well and respect for launching and all but this is just not going to ever be a venture scale startup and you should calibrate your expectations. you could be wasting the best years of your life being a wrapper of a wrapper of a wrapper and competing on developer experience in open source for no money, or you could be building agents. build agents. please.

This whole comment is predicated on the idea that something is only worthwhile if it will lead to a "venture scale startup" and that to do anything else is to "waste the best years of your life".

I, for one, am glad that not everyone shares your ethos. Conversely, I'm also glad that there are people out there "building agents". Diversity is a good thing. Encouraging everyone to only do one thing is a bad thing.

Re: Launch HN: LlamaFarm (YC W22) – Open-source framework for distributed AI

#50

Post by jbstack »

rachelradulo wrote: post #27
bityard wrote: post #24

Open source but backed by venture capital, so what is your monetization strategy?

Fair question. The core will always stay open source and free. We’ll monetize around it with things like managed hosting, enterprise support, and compliance options (HIPAA, SOC2, etc). Basically, we make money when teams want someone to stand behind it in production, not for using the software itself. But let us know if you have other ideas! We're still new to open source

So to clarify, does this mean you don't plan to go down the route of having a "community edition" vs a "enterprise edition" with missing features in the former?
Post reply on HN