Forget ChatGPT: why researchers now run small AIs on their laptops • Hacker News Board

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#11

Post by brap » Sat, Sep 21, 2024, 2:18 PM UTC

McBainiel wrote: post #9

> Microsoft used LLMs to write millions of short stories and textbooks in which one thing builds on another. The result of training on this text, Bubeck says, is a model that fits on a mobile phone but has the power of the initial 2022 version of ChatGPT. I thought training LLMs on content created by LLMs was ill-advised but this would suggest otherwise

I think it can be a tradeoff to get to smaller models. Use larger models trained on the whole internet to produce output that would train the smaller model.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#12

Post by wslh » Sat, Sep 21, 2024, 2:19 PM UTC

What's the current cost of building a DIY bare-bones machine setup to run the top LLaMA 3.1 models? I understand that two nodes are typically required for this. Has anyone built something similar recently, and what hardware specs would you recommend for optimal performance? Also, do you suggest waiting for any upcoming hardware releases before making a purchase?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#13

Post by gugagore » Sat, Sep 21, 2024, 2:20 PM UTC

McBainiel wrote: post #9

> Microsoft used LLMs to write millions of short stories and textbooks in which one thing builds on another. The result of training on this text, Bubeck says, is a model that fits on a mobile phone but has the power of the initial 2022 version of ChatGPT. I thought training LLMs on content created by LLMs was ill-advised but this would suggest otherwise

Generally (not just for LLMs) this is called student-teacher training and/or knowledge distillation.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#14

Post by brap » Sat, Sep 21, 2024, 2:20 PM UTC

Some companies (OpenAI, Anthropic…) base their whole business on hosted closed source models. What’s going to happen when all of this inevitably gets commoditized?

This is why I’m putting my money on Google in the long run. They have the reach to make it useful and the monetization behemoth to make it profitable.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#15

Post by andai » Sat, Sep 21, 2024, 2:21 PM UTC

What local models is everyone using?

The last one I used was Llama 3.1 8B which was pretty good (I have an old laptop).

Has there been any major development since then?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#16

Post by mrbungie » Sat, Sep 21, 2024, 2:23 PM UTC

McBainiel wrote: post #9

> Microsoft used LLMs to write millions of short stories and textbooks in which one thing builds on another. The result of training on this text, Bubeck says, is a model that fits on a mobile phone but has the power of the initial 2022 version of ChatGPT. I thought training LLMs on content created by LLMs was ill-advised but this would suggest otherwise

I would guess correctly aligned and/or finely filtered synthetic data coming from LLMs may be good.

Mode colapse theories (and simplified models used as proof of existence of said problem) assume affected LLMs are going to be trained with poor quality LLM-generated batches of text from the internet (i.e. reddit or other social networks).

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#17

Post by andai » Sat, Sep 21, 2024, 2:23 PM UTC

McBainiel wrote: post #9

> Microsoft used LLMs to write millions of short stories and textbooks in which one thing builds on another. The result of training on this text, Bubeck says, is a model that fits on a mobile phone but has the power of the initial 2022 version of ChatGPT. I thought training LLMs on content created by LLMs was ill-advised but this would suggest otherwise

Look into Microsoft's Phi papers. The whole idea here is that if you train models on higher quality data (i.e. textbooks instead of blogspam) you get higher quality results.

The exact training is proprietary but they seem to use a lot of GPT-4 generated training data.

On that note... I've often wondered if broad memorization of trivia is really a sensible use of precious neurons. It seems like a system trained on a narrower range of high quality inputs would be much more useful (to me) than one that memorized billions of things I have no interest in.

At least at the small model scale, the general knowledge aspect seems to be very unreliable anyways -- so why not throw it out entirely?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#18

Post by kkielhofner » Sat, Sep 21, 2024, 2:27 PM UTC

McBainiel wrote: post #9

> Microsoft used LLMs to write millions of short stories and textbooks in which one thing builds on another. The result of training on this text, Bubeck says, is a model that fits on a mobile phone but has the power of the initial 2022 version of ChatGPT. I thought training LLMs on content created by LLMs was ill-advised but this would suggest otherwise

Synthetic data (data from some kind of generative AI) has been used in some form or another for quite some time[0]. The license for LLaMA 3.1 has been updated to specifically allow its use for generation of synthetic training data. Famously, there is a ToS clause from OpenAI in terms of using them for data generation for other models but it's not enforced ATM. It's pretty common/typical to look through a model card, paper, etc and see the use of an LLM or other generative AI for some form of synthetic data generation in the development process - various stages of data prep, training, evaluation, etc.

Phi is another really good example but that's already covered from the article.

[0] - https://www.latent.space/i/146879553/synthetic-data-is-all-y...

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#19

Post by alexander2002 » Sat, Sep 21, 2024, 2:28 PM UTC

diggan wrote: post #4

Summary: It's cheaper, safer for handling sensitive data, easier to reproduce results (only way to be 100% sure it's reproduce even, as "external" models can change anytime), higher degree of customization, no internet connectivity requirements, more efficient, more flexible.

An AI chip on laptop devices would be amazing!

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#20

Post by ta988 » Sat, Sep 21, 2024, 2:29 PM UTC

alexander2002 wrote: post #19

diggan wrote: post #4

Summary: It's cheaper, safer for handling sensitive data, easier to reproduce results (only way to be 100% sure it's reproduce even, as "external" models can change anytime), higher degree of customization, no internet connectivity requirements, more efficient, more flexible.

An AI chip on laptop devices would be amazing!

They already exist. Nvidia GPUs on laptops, M series CPUs from Apple, NPUs...