The big problem with AI R&D is that nobody can keep up with the big bux companies. It makes this kind of project a bit pointless. Even if you can run a GPT3-equivalent on a web browser, how many people are going to bother (except as a stunt) when GPT4 is available?
Could you train a ChatGPT-beating model for $85k and run it in a browser?
91–100 of 175 posts
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#92Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#93Earlier quoted context omitted.
my biggest problem with these models is that they cannot reliably produce structured data. even davinci can be used as part of a chain, because you can direct it to structure and unstructure data, and then extract the single component and build them into tasks. cohere, llama et al are currently struggling to consistently produce these result reliably, even if you can chat with them and frankly it's not about the chat…
This feels solvable to me. I wonder if you could use fine tuning against LLaMA to teach it to do this better? GPT-3 etc can only do this because they had a LOT of code included in their training sets. The LLaMA paper says Github was 4.5% of the training corpus, so maybe it does have that stuff baked in and just needs extra tuning or different prompts to tap into that knowledge.
I have done it trough stages, so first stages emits in natural language in the format of "context: ... and question: ...." and then the second stage collect it as json, but then wait time doubles.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#94Earlier quoted context omitted.
> I want the best LLMs to be open source too How do you do this without being incredibly wealthy?
Yes a rugged individual would have to be incredibly wealthy to do it! But maybe the governments will make one and maintain it with taxes as an infrastructure service, like roads, giving everyone expanded powers of cognition, memory, and expertise, and raising the consciousnesses of humanity to new heights. Probably in USA it wouldn't happen if we judge ourselves only in zero sum relation to others - helping everyone…
Some governments probably alread do and use it against so-called terrorists or enemies of the people…
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#95The big problem with AI R&D is that nobody can keep up with the big bux companies. It makes this kind of project a bit pointless. Even if you can run a GPT3-equivalent on a web browser, how many people are going to bother (except as a stunt) when GPT4 is available?
An increasingly common complaint I'm hearing about GPT3/4/etc is people who don't want to pass any of their private data to another company.
Running models locally is by far the most promising solution for that concern.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#96The big problem with AI R&D is that nobody can keep up with the big bux companies. It makes this kind of project a bit pointless. Even if you can run a GPT3-equivalent on a web browser, how many people are going to bother (except as a stunt) when GPT4 is available?
The ones that can't use the GPT4 for whatever reason. Maybe you are a company and you don't want to send OpenAI your prompts. Or a person who has very private prompts and feel sketchy about sending them over.
Or maybe you are an individual who has a use case that's too edgy for OpenAI or a silicon valley corporate image. When Replika shut down people trying to have virtual boyfriend/girlfriends on their platform, their reddit filled up with people who mourned like they just lost a partner.
I think it's important that alternative non-big bux company options exist, even if most people don't want to or need to use them.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#97Earlier quoted context omitted.
> I want the best LLMs to be open source too How do you do this without being incredibly wealthy?
You (1) are a company who (2) understands the business domain and has an appropriate business plan. Sadly the reality of funding today makes it unlikely that these two will both be simultaneously satisfied. The problem is that history will look back on the necessary business plan and deem it a failure even if it generates a company that does a billion dollars plus in annual revenue. This is actually not unique to lar…
History is littered with great software that died because no-one used it because the business model was terrible. Capturing $1B of value is better than 0, and everyone understands this. And who cares what history thinks anyway?
OpenAI has spent a lot of money to get their result. It's safe to assume it will take a lot of money to get a similar result, and then to share it (although I assume bit torrent will be good enough). Once people are running their models, they can innovate to their hearts content. It's not clear how or why they'd give money back to the enabling technology. So how does money flow back to the innovators in proportion to the value produced, if not a SaaS?
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#98Just want to say SimonW has become one of my favorite writers covering the AI revolution. Always fun thought experiments with linked code and very constructive for people thinking about how to make this stuff more accessible to the masses.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#99With the explosion of LLMs and people figuring out ways to train/use them relatively cheaply, unique data sets will become that much more valuable, and will be the key differentiator between LLMs.
Interestingly, it seems like companies that run chat programs where they can read the chats are best suited to building "human conversation" LLMs, but someone who manages large text datasets for others are in the perfect place to "win" the LLM battle.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#100> as opposed to OpenAI’s continuing practice of not revealing the sources of their training data.
Looks like that choice makes it more difficult to adopt, trust, or collaborate on the new tech.
What are the benefits? Is there more to that than competitive advantage? If not, ClosedAI sounds more accurate.