Why doesn't someone just start a gofundme/kickstarter with the goal of funding the training of an open-source ChatGPT-capable model?
Could you train a ChatGPT-beating model for $85k and run it in a browser?
141–150 of 175 posts
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#142Earlier quoted context omitted.
Unfortunately training is not emberassingly parallelisable [0] problem. It would require new architecture. Current models diverge too fast. By the time you'd download and/or calculate your contribution the model would descend somewhere else and your delta would not be applicable - based off wrong initial state. It would be great if merge-ability would exist. It would also likely apply to efficient/optimal shrinking f…
hmmm... seems like you're reinventing distributed learning. merge-ability does exist and you can average the results.
Distributed learning sucks for this type of models, averaging the results helps if you can do that often which requires very high bandwidth - i.e. the Infiniband interconnects between Nvidia pods which go up to 200 Gbps.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#143Why doesn't someone just start a gofundme/kickstarter with the goal of funding the training of an open-source ChatGPT-capable model?
Create a clone of OpenAI that pledges to remains open and remains not for profit.
That could do really well via crowd funding with the right spin/marketing behind it.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#144Earlier quoted context omitted.
That’d be cool but I don’t think most idle consumer GPUs (6-8GB) would have large enough memory for a single iteration (batch size 1) of modern LLMs. But I’d love to see more federated/distributed learning platforms.
Is it possible to break the model apart? Or does the entire thing need to be architected from the get-go such that an individual GPU can own a portion end to end?
It's possible to break the model apart (I mean, for the larger models it's not that a 8Gb card isn't enough but even a single 80Gb card isn't enough) but that needs a high-speed interconnect (Nvidia pods provide hundreds of Gbps, and use all of that) as you need to exchange those parameters quite often, so you're just as limited by your compute as you are by the interconnect speed.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#145Earlier quoted context omitted.
> I want the best LLMs to be open source too How do you do this without being incredibly wealthy?
Pooling resources a la SETI@home would be an interesting option I would love to see.
This is the type of task where if you'd want to pool resources, then it would be more efficient to pool dollars and buy compute power rather than pool compute power - I'd assume that if treat the decentralized hardware as free, just the the extra electricity cost of using it is more expensive than just renting a centralized server which can do it efficiently.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#146Earlier quoted context omitted.
For a server farm, sure, for one machine, I don't know. Assuming it plugs into a normal 15A circuit, and you have a we-work or something where you don't pay for power, is the operational cost of one machine really material?
it's hard to tell from what you're saying: you're planning on putting an ML infrastructure training server on a regular 15A circuit, not in a data center or machine room? And power is paid for by somebody else? My thinking about pricing doesn't include that option because I wouldn't just hook a server like that up to a regular outlet in an office and use it for production work. If that works for you- you can happily…
Why couldn't a 75k machine live unattended in an office? If the same office has just a hundred employee workstations, those in total are worth much more than that, heck, Apple offered a Mac Pro configuration that was $50k for a single workstation.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#147Can someone at the EU, the only player in this thing with no strategy yet just pool together enough resources so the open-source people can train models. We don't ask much, just give compute power
Yes, there are a bunch of government-funded supercomputers or clusters which can be obtained for public research needs (based on an evaluation of which projects are likely to bring the most benefit), and are used, among other things, to train large language models. E.g. some interesting Swedish models got trained on https://www.nsc.liu.se/systems/berzelius/ .
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#148Why doesn't someone just start a gofundme/kickstarter with the goal of funding the training of an open-source ChatGPT-capable model?
Create a clone of OpenAI that pledges to remains open and remains not for profit. That could do really well via crowd funding with the right spin/marketing behind it.
And when everyone buys in, you go private everything and reap the benefits. Brilliant!
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#149We need a DAWNBench* benchmark for training ChatGPT the fastest and cheapest.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#150Earlier quoted context omitted.
Create a clone of OpenAI that pledges to remains open and remains not for profit. That could do really well via crowd funding with the right spin/marketing behind it.
And when everyone buys in, you go private everything and reap the benefits. Brilliant!
Except, don’t go private this time. Appoint an external board not controlled by the CEO and write it into your bylaws.