Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open source AI must win

opensourceaimustwin.com

Post reply on HN
Print view View original

491–500 of 538 posts

Re: Open source AI must win

#491

Post by yogthos »

ThejaCH wrote: post #485
yogthos wrote: post #327

Earlier quoted context omitted.

There are two parts to this too. One is the raw model capability and the other is how well the harness guides the model and meets its expectations. I really think for stuff like agentic coding, this has to be treated as a package. This is my favorite example of how much difference a harness can make even for a tiny model https://github.com/itigges22/ATLAS And you're bang on with the storage comparison, we're basicall…

The repo is so cool!

It's really neat project, gives me a lot of hope that we'll be able to run local models to do most of the work in the near future.

Re: Open source AI must win

#492

Post by salter2 »

killerstorm wrote: post #425
pizzao wrote: post #420

Earlier quoted context omitted.

I wonder if there is way local small LLMs can complement each other in away that the sum-total yields a much more performant LLM

Perhaps some radical MoE where you download _exactly_ the components you need as you need them. Currently MoE is switched usually on per-token per-layer basis, so you need all weights locally. But e.g. Apple made one which pre-selects all experts based on prompt embedding. That might be further scaled up - e.g. predict exactly what's needed

Perhaps something similar to speculative decoding.

Speculating Experts Accelerates Inference for Mixture-of-Experts: https://arxiv.org/abs/2603.19289

Re: Open source AI must win

#493

Post by rurban »

bombcar wrote: post #466
rurban wrote: post #336

Earlier quoted context omitted.

You got it wrong. Inference can use crap GPU's. Training needs the 100x more expensive big guns. Our training machine is 100x more expensive than our inference machine.

How is the result of training stored? How big is that? It seems reasonable to assume we’ll eventually plateau and all we’ll need is relatively infrequent training.

Not so often. The GPU's are running 100% for 3 weeks for a training run. We do images only, but it's the same process. And then we can use the costly GPU's for inference, local model coding agents. Training is about 4x a year. But it depends what ideas the PM or the costumers have. If they has more, more training tasks. Eg. more viruses to detect.

Re: Open source AI must win

#494

Post by iugtmkbdfil834 »

boutell wrote: post #481
iugtmkbdfil834 wrote: post #477

Earlier quoted context omitted.

Dunno, in a sense, torrents came among similar restrictions. Everything at consumer level was just plain awful and at dial up level, mebbe ISDN if you were very lucky, with fiber only available to ridiculously rich people and corps. But with restrictions, came approaches on how to mitigate them.

If weights can't be looked at almost instantly in bulk, it just doesn't work. It's a different problem from distributing file downloads.

I used it as an example. I understand the problem is hard. My larger point was that this is exactly how actual progress tends to take place. Well, that and porn.

Re: Open source AI must win

#495

Post by AtlasBarfed »

palisade wrote: post #44

I've been contemplating a decentralized model training system for some time using volunteer machines that we all contribute. But, it is astronomically difficult. The communication speeds are untenable. And, there is the issue of data poisoning from untrusted nodes. I've almost cracked that last issue with a self-healing checkpointed rollback system that doesn't have to throw out anything that follows the corrupt datu…

I just read something or saw something about in document recalculation being a completely wasted step in every single training run. Is that true

Re: Open source AI must win

#496

Post by iugtmkbdfil834 »

aspenmartin wrote: post #480
iugtmkbdfil834 wrote: post #477

Earlier quoted context omitted.

Dunno, in a sense, torrents came among similar restrictions. Everything at consumer level was just plain awful and at dial up level, mebbe ISDN if you were very lucky, with fiber only available to ridiculously rich people and corps. But with restrictions, came approaches on how to mitigate them.

Yes but not violations of the laws of physics. You need extremely fast communications, memory bandwidth, etc; you cannot get that with distributed training. You're up against the speed of light and the interconnect that powers the internet. You will always have horrifically slow latency compared to if you pack the servers together in the same place with specialized networking.

Agree about the physics; disagree about the larger point.

I am not questioning that servers packed together may achieve an optimal result in how we are currently doing things, but, and this is my point, what if we didn't.

This is entirely the wrong question to ask. The question to ask is: how it could be adapted to distributed training.

Re: Open source AI must win

#497

Post by taylorhou »

xtracto wrote: post #391

This, and distributed LLM inference. We are at a point where no single person can setup a rig to run a SOTA model, it is just too expensive. So we must build and adopt frameworks that allow individuals to share resources to run SOTA models in a distributed manner. That way they will also be non-censorable by governments. Also The only way to prevent that one entity weaponizes it, is by giving EVERYONE access to it.

I built Teale.com and opensourced it. My domain contribution to society. It powers fully distributed inference on Mac, windows, Linux, android, iOS, hell even harmonyOS.

Opensource/weight models will get better and better and eventually we will have mythos level running on smartphone/eyeglass hardware.

It is stupidly tedious currently to match supply with demand though because physical hardware like a 16gb ram MacBook doesn't mean there's truly 16gb available let alone matching models and all of their settings (kvcache, context limit, temperature, etc) to demand.

Would appreciate any help cus we need ai inference by the people for the people.

Re: Open source AI must win

#498

Post by taylorhou »

palisade wrote: post #44

I've been contemplating a decentralized model training system for some time using volunteer machines that we all contribute. But, it is astronomically difficult. The communication speeds are untenable. And, there is the issue of data poisoning from untrusted nodes. I've almost cracked that last issue with a self-healing checkpointed rollback system that doesn't have to throw out anything that follows the corrupt datu…

Let's collab. I'm one guy too but I built distributed inference network (teale.com) banging away for about a month with opus/gpt

Re: Open source AI must win

#499

Post by WASDx »

xtracto wrote: post #391

This, and distributed LLM inference. We are at a point where no single person can setup a rig to run a SOTA model, it is just too expensive. So we must build and adopt frameworks that allow individuals to share resources to run SOTA models in a distributed manner. That way they will also be non-censorable by governments. Also The only way to prevent that one entity weaponizes it, is by giving EVERYONE access to it.

> distributed LLM inference

This seems extremely inefficient considering data transfer between model layers if the model is distributed. I found this project called Petals that claim up to 4 tok/s for a 180B model although its repository hasn't been updated in two years.

https://petals.dev/

Re: Open source AI must win

#500

Post by stymaar »

WASDx wrote: post #499
xtracto wrote: post #391

This, and distributed LLM inference. We are at a point where no single person can setup a rig to run a SOTA model, it is just too expensive. So we must build and adopt frameworks that allow individuals to share resources to run SOTA models in a distributed manner. That way they will also be non-censorable by governments. Also The only way to prevent that one entity weaponizes it, is by giving EVERYONE access to it.

> distributed LLM inference This seems extremely inefficient considering data transfer between model layers if the model is distributed. I found this project called Petals that claim up to 4 tok/s for a 180B model although its repository hasn't been updated in two years. https://petals.dev/

For token generation, yes: because current-gen LLMs are autoregressive you need to add the inter-node latency for every since token.

For prompt processing it would work though, and it could for diffusion LLMs as well.

Post reply on HN