What we need is a RETRO style model where basically after the input you go through a small net that just fetches a desired set of weights from a server (serving data without compute is dirt cheap) and is then executed locally. We’ll get there eventually
Could you train a ChatGPT-beating model for $85k and run it in a browser?
131–140 of 175 posts
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#132Earlier quoted context omitted.
A few people have built frameworks to do this. There is still a very large open problem in how to federate large numbers of loosely coupled computers to speed up training "interesting" models. I've worked in both domains (protein folding via Folding@Home/protein folding using supercomputers, and ML training on single nodes/ML training on supercomputers) and at least so far, ML hasn't really been a good match for emba…
And you can rule out most of the monte carlo stuff too. Which rules out parallelization modern statistical frameworks like STAN used for explainable models; things like Finance modeling of risk which is a sampling of posteriors using MCMC also can't be parallelized.
Assuming the chains can reach an equilibrium point (i.e. burn in) quickly, M samples from an MCMC can be parallelized by running N chains in parallel each for M/N iterations. You still end up with M total samples from your target distribution.
You’re only out of luck if each iteration is too compute intense to fit on one worker node, even if each iteration might be embarrassingly parallelizable, since the overhead of having to aggregate computations across workers at every iteration would be too high.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#133What we need is a RETRO style model where basically after the input you go through a small net that just fetches a desired set of weights from a server (serving data without compute is dirt cheap) and is then executed locally. We’ll get there eventually
Can anyone explain or link some resource on why these big GPT models all don't incorporate any RETRO style? I'm only very superficially following ML developments and I was so hyped by RETRO and then none of the modern world changing models apply it.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#134I wonder why anyone would want to run it in a browser, other than to show it could be done? It's not like the extra latency would matter, since these things are slow. Running it on a server you control makes more sense. You can pick appropriate hardware for running the AI. Then access it from any browser you like, including from your phone, and switch devices whenever you like. It won't use up all the CPU/GPU on a po…
The browser thing is definitely more for show than anything else - I used it to help demonstrate quite how surprisingly lightweight these models can be.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#135What we need is a RETRO style model where basically after the input you go through a small net that just fetches a desired set of weights from a server (serving data without compute is dirt cheap) and is then executed locally. We’ll get there eventually
Can anyone explain or link some resource on why these big GPT models all don't incorporate any RETRO style? I'm only very superficially following ML developments and I was so hyped by RETRO and then none of the modern world changing models apply it.
Openai might very well be using that internally who knows how they implement things. Also emad retweeted a RETRO related thing a bit back so they might very well be using that for their awaited LM, here’s hoping
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#136If you have ~100k to spend, aren't there options to buy a gpu rather than just blow it all on cloud? How much is an 8xA100 machine? 4xA100 is 75k, 8 is 140k https://shop.lambdalabs.com/deep-learning/servers/hyperplane...
If you bought an 8xA100 machine for $140k you would have to run it continuously for over 10,000 hours (about 14 months) to train the 7B model. By that time the value of the A100s you bought would have depreciated substantially; especially because cloud companies will be renting/selling A100s at a discount as they bring H100s online. It might still be worth it, but it's not a home run.
If 8-bit training methods take off, I think the calculus is going to change rapidly, with newer cards that have decent amounts of memory and 8-bit acceleration starting to become dramatically more cost and time effective than the venerable A100s.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#137The big problem with AI R&D is that nobody can keep up with the big bux companies. It makes this kind of project a bit pointless. Even if you can run a GPT3-equivalent on a web browser, how many people are going to bother (except as a stunt) when GPT4 is available?
The ones that can't use the GPT4 for whatever reason. Maybe you are a company and you don't want to send OpenAI your prompts. Or a person who has very private prompts and feel sketchy about sending them over. Or maybe you are an individual who has a use case that's too edgy for OpenAI or a silicon valley corporate image. When Replika shut down people trying to have virtual boyfriend/girlfriends on their platform, the…
Or maybe you're in Italy and OpenAI had just been banned from the country for not adhering to GDPR. I suspect the rest of the EU may follow soon.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#138Keep in mind that image transformer models like stable diffusion are generally smaller than language models, so they are easier to fit in wasm space. Also. you can finetune llama-7b on a 3090 for about $3 using LoRA.
Generative image models don't use transformers, they're diffusion models. LLMs are transformers.
Ah yes that's right. Well they technically do use a visual transformer for CLIP text encoder as I understand.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#139[deleted]
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#140Are there any training/ownership models like Folding@Home? People could donate idle GPU resources in exchange for access to the data, and perhaps ownership. Then instead of someone needing to pony up $85k to train a model, a thousand people can train a fraction of the model on their consumer GPU and pool the results, reap the collective rewards.
Every parameter needs to reach every other parameter. Ideally enough core memory for that. But their tiling algorithms.