Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Could you train a ChatGPT-beating model for $85k and run it in a browser?

simonwillison.net

Post reply on HN
Print view View original

61–70 of 175 posts

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#61

Post by rspoerri »

pmoriarty wrote: post #33
rspoerri wrote: post #12

So cool it runs on a browser /sarcasm/ i might not even need a computer. Or internet when we are at it. It either runs locally or it runs on the cloud. Data could come from both locations as well. So it's mostly technically irrelevant if it's displaying in a browser or not. Except when it comes to usability. I don't get it why people love software running in a browser. I often close important tools i have not saved w…

There are a bunch of reasons people/companies like web apps: 1 - Everyone already has a web browser, so there's no software to download (or the software is automatically downloaded, installed and run, if you want to look at it that way... either way, the experience is a lot easier and more seamless for the user) 2 - The website owner has control of the software, so they can update it and manage user access as they li…

Most of these aspects make it better for the company or developer, only in some cases it makes it easier for the user in my opinion. Some arguments against it are:

1 - Not everyone has or wants fast access to the internet all the time.

2 - I try to prevent access of most of the apps to the internet. I don't want companies to access my data or even metadata of my usage.

3 - sure, but it doesn't make it better for the user.

4 - Also supporting different screen sizes and interaction types (touch or mouse) can be a big part of the work.

The most important part for a user is if he/she is only using the app rarely or once. Not having to install it will make the difference between using it or not. However with the app stores most OS's feature today this can change pretty soon and be equally simple.

I might be old school on this, but i resent subscription based apps. For applications that do not need to change, deliver no additional service or aren't absolutely vital for me i will never subscribe. And browser based app's are at the core of this unfortunate development. But that's gone very far from the original topic :-)

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#62

Post by pessimizer »

sroussey wrote: post #38
ushakov wrote: post #17

Now imagine loading 3.9 GB each time you want to interact with a webpage

10yrs from now models will be in the OS. Maybe even in silicon. No downloads required.

Not in mine. I don't even want redhat's bullshit in there. I'm not installing some black box into my OS that was programmed with motives that can't be extracted from the model at rest.

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#63

Post by mirekrusin »

whalesalad wrote: post #49

Are there any training/ownership models like Folding@Home? People could donate idle GPU resources in exchange for access to the data, and perhaps ownership. Then instead of someone needing to pony up $85k to train a model, a thousand people can train a fraction of the model on their consumer GPU and pool the results, reap the collective rewards.

Unfortunately training is not emberassingly parallelisable [0] problem. It would require new architecture. Current models diverge too fast. By the time you'd download and/or calculate your contribution the model would descend somewhere else and your delta would not be applicable - based off wrong initial state.

It would be great if merge-ability would exist. It would also likely apply to efficient/optimal shrinking for models.

Maybe you could dispatch tasks to train on many variations of similar tasks and take average of results? It could probably help in some way, but you'd still have large serialized pipeline to munch through and you'd likely require some serious hardware ie. dual gtx 4090 on client side.

[0] https://en.wikipedia.org/wiki/Embarrassingly_parallel

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#64

Post by KeplerBoy »

dekhn wrote: post #43
version_five wrote: post #29

Earlier quoted context omitted.

For a server farm, sure, for one machine, I don't know. Assuming it plugs into a normal 15A circuit, and you have a we-work or something where you don't pay for power, is the operational cost of one machine really material?

it's hard to tell from what you're saying: you're planning on putting an ML infrastructure training server on a regular 15A circuit, not in a data center or machine room? And power is paid for by somebody else? My thinking about pricing doesn't include that option because I wouldn't just hook a server like that up to a regular outlet in an office and use it for production work. If that works for you- you can happily…

Nvidia happily sells what you're describing. They call it "DGX Station A100", it has 4 80GB A100 and retails for 80k. Not sure i believe their claimed noise level of Of course that's still a very small system when talking LLM training, the only reason why i would not put that in a regular office is it's extreme price. Do you really want something worth 80k in a form factor that could be casually carried through the door?

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#65

Post by _trampeltier »

whalesalad wrote: post #49

Are there any training/ownership models like Folding@Home? People could donate idle GPU resources in exchange for access to the data, and perhaps ownership. Then instead of someone needing to pony up $85k to train a model, a thousand people can train a fraction of the model on their consumer GPU and pool the results, reap the collective rewards.

Start a Boinc project.

https://boinc.berkeley.edu/projects.php

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#66

Post by riedel »

ftxbro wrote: post #52
whalesalad wrote: post #49

Are there any training/ownership models like Folding@Home? People could donate idle GPU resources in exchange for access to the data, and perhaps ownership. Then instead of someone needing to pony up $85k to train a model, a thousand people can train a fraction of the model on their consumer GPU and pool the results, reap the collective rewards.

Yes there is petals/bloom https://github.com/bigscience-workshop/petals but it's not so great. Maybe it will improve or a better one will come.

I read that it is only scoring the model collaboratively but it allows some fine-tuning I guess.

Getting the actual gradient descent to parallelize is more difficult because one needs to average the gradient when using data/batch parallelism. It becomes more a network speed than GPU speed problem. Or are LLMs somehow different?

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#67

Post by girthbrooks »

version_five wrote: post #16

If you have ~100k to spend, aren't there options to buy a gpu rather than just blow it all on cloud? How much is an 8xA100 machine? 4xA100 is 75k, 8 is 140k https://shop.lambdalabs.com/deep-learning/servers/hyperplane...

These are wild pieces of hardware, thanks for linking. I wonder how loud they get.

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#68

Post by simonw »

avereveard wrote: post #47
simonw wrote: post #34

Earlier quoted context omitted.

Yeah, you're right. I wrote this a couple of weeks ago at the height of LLaMA hype, but with further experience I don't think the GPT-3 comparisons hold weight. My biggest problem: I haven't managed to get a great summarization out of a LLaMA derivative that runs on my laptop yet. Maybe I haven't tried the right model or the right prompt yet though, but that feels essential to me for a bunch of different applications…

my biggest problem with these models is that they cannot reliably produce structured data. even davinci can be used as part of a chain, because you can direct it to structure and unstructure data, and then extract the single component and build them into tasks. cohere, llama et al are currently struggling to consistently produce these result reliably, even if you can chat with them and frankly it's not about the chat…

This feels solvable to me. I wonder if you could use fine tuning against LLaMA to teach it to do this better?

GPT-3 etc can only do this because they had a LOT of code included in their training sets.

The LLaMA paper says Github was 4.5% of the training corpus, so maybe it does have that stuff baked in and just needs extra tuning or different prompts to tap into that knowledge.

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#69

Post by dekhn »

digitallyfree wrote: post #39
dekhn wrote: post #20

Earlier quoted context omitted.

you're comparing the capital cost of acquiring a GPU machine with the operational cost of renting one in the cloud. Ignoring the operational costs of on-prem hardware is pretty common, but those costs are significant and can greatly change the calculation.

For a single unit one could have it in their home or office, rather than a datacenter or colo. If the user sets up and manages the machine themselves there is no additional IT cost. The greatest operating expense would be the power cost.

"If the user sets up and manages the machine themselves there is no additional IT cost" In my experience, physical hardware has a management overhead over cloud resources. Backups, large disk storage for big models, etc.

Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?

#70

Post by lmeyerov »

It seems the quality goes up & cost goes down significantly with Colossal AI's recent push: https://medium.com/@yangyou_berkeley/colossalchat-an-open-so...

Their writeup makes it sounds like, net, 2X+ over Alpaca, and that's an early run

The browser side is interesting too. Browser JS VMs have a memory cap of 1GB, so that may ultimately be the bottleneck here...

Post reply on HN