Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forget ChatGPT: why researchers now run small AIs on their laptops

nature.com

Post reply on HN
Print view View original

81–90 of 385 posts

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#81

Post by whimsicalism »

brap wrote: post #14

Some companies (OpenAI, Anthropic…) base their whole business on hosted closed source models. What’s going to happen when all of this inevitably gets commoditized? This is why I’m putting my money on Google in the long run. They have the reach to make it useful and the monetization behemoth to make it profitable.

Their hope is to reach AGI and effective post-scarcity for most things that we currently view as scarce.

I know it sounds crazy but that is what they actually believe and is a regular theme of conversations in SF. They also think it is a flywheel and whoever wins the race in the next few years will be so far ahead in terms of iteration capability/synthetic data that they will be the runaway winner.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#82

Post by IshKebab »

pella wrote: post #69

Llama 3.1 405B "2 MacBooks is all you need. Llama 3.1 405B running distributed across 2 MacBooks using @exolabs_ home AI cluster" https://x.com/AIatMeta/status/1834633042339741961

"All you need is £10k of Apple laptops..."

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#83

Post by Anunayj »

I recently experimented with running llama-3.1-8b-instruct locally on my Consumer hardware, aka my Nvidia RTX 4060 with 8GB VRAM, as I wanted to experiment with prompting pdfs with a large context which is extremely expensive with how LLMs are priced.

I was able to fit the model with decent speeds (30 tokens/seconds) and a 20k token context completely on the GPU.

For summarization, the performance of these models are decent enough. However unfortunately in my use case I felt using Gemini's Free Tier with it's multimodal capabilities and much better quality output made running local LLMs not really worth it as of right now, atleast for consumers.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#85

Post by calf »

gugagore wrote: post #13
McBainiel wrote: post #9

> Microsoft used LLMs to write millions of short stories and textbooks in which one thing builds on another. The result of training on this text, Bubeck says, is a model that fits on a mobile phone but has the power of the initial 2022 version of ChatGPT. I thought training LLMs on content created by LLMs was ill-advised but this would suggest otherwise

Generally (not just for LLMs) this is called student-teacher training and/or knowledge distillation.

It reminds me of when I take notes from a textbook then intensively review my own notes

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#86

Post by zdragnar »

throwaway314155 wrote: post #52
brap wrote: post #14

Some companies (OpenAI, Anthropic…) base their whole business on hosted closed source models. What’s going to happen when all of this inevitably gets commoditized? This is why I’m putting my money on Google in the long run. They have the reach to make it useful and the monetization behemoth to make it profitable.

I don't have a horse in the race but wouldn't Meta be more likely to commoditize things given that they sort of already are?

Search

Gmail

Docs

Android

Chrome (browser and Chromebooks)

I don't use any Meta properties at all, but at least a dozen alphabet ones. My wife uses Facebook, but that's about it. I can see it being handy for insta filters.

YMMV of course, but I suspect alphabet has much deeper reach, even if the actual overall number of people is similar.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#87

Post by 101008 »

sparky_ wrote: post #71
csmpltn wrote: post #21

Earlier quoted context omitted.

There's plenty of competition in this space already, and it'll only get accelerated with time. There's not enough "moat" in building proprietary LLMs - you can tell by how the leading companies in this space are basically down to fighting over patents and regulatory capture (ie. mounting legal and technical barriers to scraping, procuring hardware, locking down datasets, releasing less information to the public about…

This is the best take on the generative AI fad I've yet seen. I wish I could upvote this twice.

I had the same impression. I have been suffering a lot lately about the future for engineers (not having work, etc), even habing anxiety when I read news about AI, but these comments make me feel better and relaxed.

I even considered blocking HN.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#88

Post by dchuk »

HexDecOctBin wrote: post #50

May as well ask here: what is the best way to use something like an LLM as a personal knowledge base? I have a few thousand book, papers and articles collected over the last decade. And while I have meticulously categorised them for fast lookup, it's getting harder and harder to search for the desired info, especially in categories which I might not have explored recently. I do have a 4070 (12 GB VRAM), so I thought…

Look into this: https://www.anthropic.com/news/contextual-retrieval

And this: https://microsoft.github.io/graphrag/

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#89

Post by visarga »

jannyfer wrote: post #79
vessenes wrote: post #76

All this will be an interesting side note in the history of language models in the next eight months when roughly 1.5 billion iPhone users will get a local language model tied seamlessly to a mid-tier cloud based language model native in their OS. What I think will be interesting is seeing which of the open models stick around and for how long when we have super easy ‘good enough’ models that provide quality integrat…

> in the next eight months when roughly 1.5 billion iPhone users will get a local language model tied seamlessly to a mid-tier cloud based language model native in their OS. Only iPhone 15 Pro or later will get Apple Intelligence, so the number will be wayyy smaller.

Not in EU they won't.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#90

Post by HPsquared »

iJohnDoe wrote: post #37
McBainiel wrote: post #9

> Microsoft used LLMs to write millions of short stories and textbooks in which one thing builds on another. The result of training on this text, Bubeck says, is a model that fits on a mobile phone but has the power of the initial 2022 version of ChatGPT. I thought training LLMs on content created by LLMs was ill-advised but this would suggest otherwise

> Microsoft used LLMs to write millions of short stories and textbooks Millions? Where are they? Where are they used?

Model developers don't usually release training data like that.
Post reply on HN