WebGPU is going to be a major component in this. Modern GPU's prevalent in mobile devices, desktops and laptops, is more than enough to do all of this client side.
Could you train a ChatGPT-beating model for $85k and run it in a browser?
101–110 of 175 posts
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#102Are there any training/ownership models like Folding@Home? People could donate idle GPU resources in exchange for access to the data, and perhaps ownership. Then instead of someone needing to pony up $85k to train a model, a thousand people can train a fraction of the model on their consumer GPU and pool the results, reap the collective rewards.
How long until somebody creates a crypto project on that?
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#103Are there any training/ownership models like Folding@Home? People could donate idle GPU resources in exchange for access to the data, and perhaps ownership. Then instead of someone needing to pony up $85k to train a model, a thousand people can train a fraction of the model on their consumer GPU and pool the results, reap the collective rewards.
A few people have built frameworks to do this. There is still a very large open problem in how to federate large numbers of loosely coupled computers to speed up training "interesting" models. I've worked in both domains (protein folding via Folding@Home/protein folding using supercomputers, and ML training on single nodes/ML training on supercomputers) and at least so far, ML hasn't really been a good match for emba…
And you can rule out most of the monte carlo stuff too. Which rules out parallelization modern statistical frameworks like STAN used for explainable models; things like Finance modeling of risk which is a sampling of posteriors using MCMC also can't be parallelized.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#104Earlier quoted context omitted.
Yeah, you're right. I wrote this a couple of weeks ago at the height of LLaMA hype, but with further experience I don't think the GPT-3 comparisons hold weight. My biggest problem: I haven't managed to get a great summarization out of a LLaMA derivative that runs on my laptop yet. Maybe I haven't tried the right model or the right prompt yet though, but that feels essential to me for a bunch of different applications…
Have you tried bigger models? Llama-65B can indeed compete with GPT-3 according to various benchmarks. The next thing would be to get the fine-tuning as good as OpenAI's.
I wonder how accurate those benchmarks are in terms of actual problem solving capability. I think there's a major line at which point LLM becomes actually useful and it actually feels like you are speaking to something intelligent and that can be useful for you in terms of productivity etc.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#105A wonderful thing about software development is that there is so much reserved space for creativity that we have huge gaps between costs and value. Whether the average person could do this for 85k I'm uncertain of - but there is a very significant slice of people that could do it for well under 85k now that the ground work has been done. This leads to the hilarious paradox where a software based business worth millio…
> This leads to the hilarious paradox where a software based business worth millions could be built on top of code valued around 60k to write.
Or the fact that software based businesses just took a massive hit in value overnight and cannot possibly defend such high valuations anymore.
The value of companies is quickly going to shift from tech moats to brands.
Think CocaCola - anyone can create a drink that tastes as good or better than coke, but it's incredibly hard to compete with the CocaCola brand.
Now think what would have happened if CocaCola had been super expensive to make, and all of a sudden, in a matter of weeks, it became incredibly cheap.
This is what happened to the saltpeter industry in 1909 when synthetic saltpeter was invented. The whole industry was extinct in a few years.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#106It seems the quality goes up & cost goes down significantly with Colossal AI's recent push: https://medium.com/@yangyou_berkeley/colossalchat-an-open-so... Their writeup makes it sounds like, net, 2X+ over Alpaca, and that's an early run The browser side is interesting too. Browser JS VMs have a memory cap of 1GB, so that may ultimately be the bottleneck here...
I thought the memory limit (in V8 at least) was 2GB due to the GC not wanting to pass 64 bit pointers around, and using the high bit of a 32-bit offset for .. something I now forget ..?
Do you have a source showing a JS runtime with a 1GB limit?
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#107It seems the quality goes up & cost goes down significantly with Colossal AI's recent push: https://medium.com/@yangyou_berkeley/colossalchat-an-open-so... Their writeup makes it sounds like, net, 2X+ over Alpaca, and that's an early run The browser side is interesting too. Browser JS VMs have a memory cap of 1GB, so that may ultimately be the bottleneck here...
does the 1gb limit extend to wasm?
WASM is specified to have 32-bit pointers, which is 4GB. AFAIK browser implementations respect that (when I did some nominal testing a couple years ago)
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#108Keep in mind that image transformer models like stable diffusion are generally smaller than language models, so they are easier to fit in wasm space. Also. you can finetune llama-7b on a 3090 for about $3 using LoRA.
Generative image models don't use transformers, they're diffusion models. LLMs are transformers.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#109The big problem with AI R&D is that nobody can keep up with the big bux companies. It makes this kind of project a bit pointless. Even if you can run a GPT3-equivalent on a web browser, how many people are going to bother (except as a stunt) when GPT4 is available?
The ones that can't use the GPT4 for whatever reason. Maybe you are a company and you don't want to send OpenAI your prompts. Or a person who has very private prompts and feel sketchy about sending them over. Or maybe you are an individual who has a use case that's too edgy for OpenAI or a silicon valley corporate image. When Replika shut down people trying to have virtual boyfriend/girlfriends on their platform, the…
Those are seriously niche use cases. They exist but can they fund gpt5 level development?
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#110It seems the quality goes up & cost goes down significantly with Colossal AI's recent push: https://medium.com/@yangyou_berkeley/colossalchat-an-open-so... Their writeup makes it sounds like, net, 2X+ over Alpaca, and that's an early run The browser side is interesting too. Browser JS VMs have a memory cap of 1GB, so that may ultimately be the bottleneck here...
I thought the memory limit (in V8 at least) was 2GB due to the GC not wanting to pass 64 bit pointers around, and using the high bit of a 32-bit offset for .. something I now forget ..? Do you have a source showing a JS runtime with a 1GB limit?
UPDATE: After a nominal amount of googling around it appears valid sizes have increased on 64-bit systems to a maximum of 8GB, and stayed at 2GB on 32-bit systems, for FF at least. I guess it's probably 'implementation defined'
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...