351–360 of 538 posts
Earlier quoted context omitted.
We can run open weight models on our own machines.
yes, but a model that runs on my own machine will never have the capacity of a model that runs in a datacenter. as i said, it can't compete with that.
I've been contemplating a decentralized model training system for some time using volunteer machines that we all contribute. But, it is astronomically difficult. The communication speeds are untenable. And, there is the issue of data poisoning from untrusted nodes. I've almost cracked that last issue with a self-healing checkpointed rollback system that doesn't have to throw out anything that follows the corrupt datu…
>But when people think of decentralized training, they don’t first think of gigantic datacenters, owned by the same company, training models across large distances. Instead, they imagine thousands of small datacenters, or individual consumers, pooling their spare compute over the internet to orchestrate a training run larger than any single actor could manage alone. Many companies are pursuing this vision: Pluralis R…
Models have limited shelf live while things are improving rapidly, and decentralized training is just more wasteful.
However, things might change if we get to what Karpathy calls "cognitive core" - a stable model backbone which can be extended via skills/adapters/etc. Development of extensions to the core can be a lot more decentralized.
But for now these decentralized training attempts function largely as a deterrent to anti-open-source collusion
I would be totally willing to pay $50 per month to support an open source AI lab, rather to get open source models as byproducts of corporations.
I think OpenAI ruined the trust. How do you know this 'open source AI lab' won't somehow spin off a for-profit?
Open-source AI can, by definition, never "win". AI is just hillclimbing today, and closed labs can always absorb everything the open world does and build upon it. It doesn't really matter for most use cases, because the way AI is working is capability saturation. https://www.delanceyukschoolschesschallenge.com/the-rising-t... The only exception to this is fields that are inherently adversarial (to nature or others) a…
Absorbing all the good ideas or data from openly available systems doesn't seem to be the only determiner
Earlier quoted context omitted.
> Open source 'winning' just means that there exists at least one open source alternative to closed models which is as good as, say, GPT 4... I mean, we're essentially there already with Google Gemma models. Is this really true? We just don't know what the maximum capability of AI is. If it turns out AI can be as intelligent and capable as something like Data from Star Trek, no one is going to be thinking GPT 4 is go…
>>We just don't know what the maximum capability of AI is For all theory purposes there is no limit. Thats what the latest loop engineering trend is about, you are asking AI to find solutions to a problem going by listing steps, and if solution not found in those steps, to treat each step as a separate problem and repeat the process until the master solution to the master problem is found. Once a solution is found, o…
Agency seems to correlate with the ability to make good decisions. It's kind of surprising how much agency is required to make good technical decisions. It's not even about business domain knowledge; a lot of agency is needed even in a pure tech context.
I've been contemplating a decentralized model training system for some time using volunteer machines that we all contribute. But, it is astronomically difficult. The communication speeds are untenable. And, there is the issue of data poisoning from untrusted nodes. I've almost cracked that last issue with a self-healing checkpointed rollback system that doesn't have to throw out anything that follows the corrupt datu…
>The communication speeds are untenable. Can it be parallelized or not? If you take a model, make two copies, and fine-tune each one on different data, what happens when you merge them? Does it work if you freeze different layers? I think this works if the steps are small enough. And the transfer should become tenable if the steps are big enough. Where's the cutoff?
Earlier quoted context omitted.
It's too late. You can one-shot a port of Linux to Rust and stop contributing to open source. The value of software is going to tend towards zero. The value of the software developer the same. Anthropic is now a kingmaker. It gets to decide which businesses get the expensive private model that can generate entire business functions at the drop of a hat. If you can't afford the price tag, then competition in the marke…
> You can one-shot a port of Linux to Rust and stop contributing to open source. Touch grass brother. Seriously.
what is Open Source AI even? to me Open Source, like Free Software, is something i can run on my own computer. any AI system that runs on a computer that i do not control is by my definition not Open Source. so how then can Open Source AI win? it can't even compete. even if we collect enough money and create a dedicated Open Source organization to build and run a community owned AI datacenter, how does that help? so…
Qwen models are actually very competitive with frontier models, and you can run them on your local computer. Gotta have a decent graphics card and by that time the current cost of the rig may not justify it over paying $100/month for cloud model but it’s all out there.