Forget ChatGPT: why researchers now run small AIs on their laptops • Hacker News Board

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#71

Post by sparky_ » Sat, Sep 21, 2024, 3:23 PM UTC

csmpltn wrote: post #21

brap wrote: post #14

Some companies (OpenAI, Anthropic…) base their whole business on hosted closed source models. What’s going to happen when all of this inevitably gets commoditized? This is why I’m putting my money on Google in the long run. They have the reach to make it useful and the monetization behemoth to make it profitable.

There's plenty of competition in this space already, and it'll only get accelerated with time. There's not enough "moat" in building proprietary LLMs - you can tell by how the leading companies in this space are basically down to fighting over patents and regulatory capture (ie. mounting legal and technical barriers to scraping, procuring hardware, locking down datasets, releasing less information to the public about…

This is the best take on the generative AI fad I've yet seen. I wish I could upvote this twice.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#72

Post by sandwichmonger » Sat, Sep 21, 2024, 3:24 PM UTC

McBainiel wrote: post #9

> Microsoft used LLMs to write millions of short stories and textbooks in which one thing builds on another. The result of training on this text, Bubeck says, is a model that fits on a mobile phone but has the power of the initial 2022 version of ChatGPT. I thought training LLMs on content created by LLMs was ill-advised but this would suggest otherwise

That's the number one way of getting mad LLM disease. Feeding LLMs to LLMs.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#73

Post by smcleod » Sat, Sep 21, 2024, 3:24 PM UTC

exe34 wrote: post #30

tetrisgm wrote: post #7

I like self hosting random stuff on docker. Ollama has been a great addition. I know it's not, but it feels on par with ChatGPT. It works perfectly on my 4090, but I've also seen it work perfectly on my friend's M3 laptop. It feels like an excellent alternative for when you don't need the heavy weights, but want something bespoke and private. I've integrated it with my Obsidian notes for 1) note generation 2) fuzzy s…

which model are you using? what size/quant/etc? thanks!

Come join us on Reddit’s /r/localllama. Great community for local LLMs.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#74

Post by meiraleal » Sat, Sep 21, 2024, 3:24 PM UTC

albertgoeswoof wrote: post #62

I have a three year old M1 Max, 32gb RAM. Llama 8bn runs at 25 tokens/sec, that’s fast enough, and covers 80% of what I need. On my ryzen 5600h machine, I get about 10 tokens/second, which is slow enough to be annoying. If I get stuck on a problem, switch to chat gpt or phind.com and see what that gives. Sometimes, it’s not the LLM that helps, but changing the context and rewriting the question. However I cannot use…

We need browser and OS level API (mobile) integration to the local LLM.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#75

Post by meiraleal » Sat, Sep 21, 2024, 3:27 PM UTC

Dalewyn wrote: post #40

aurareturn wrote: post #36

Earlier quoted context omitted.

It will have around 250GB/s of bandwidth which makes it nearly unusable for 70b models. So the high amount of RAM doesn’t help with large models.

Fast. Large. Cheap. You may only pick two.

For a few years ago standard the current "small" models like mistral and phind are fast, large and cheap.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#76

Post by vessenes » Sat, Sep 21, 2024, 3:29 PM UTC

All this will be an interesting side note in the history of language models in the next eight months when roughly 1.5 billion iPhone users will get a local language model tied seamlessly to a mid-tier cloud based language model native in their OS.

What I think will be interesting is seeing which of the open models stick around and for how long when we have super easy ‘good enough’ models that provide quality integration. My bet is not many, sadly. I’m sure Llama will continue to be developed, and perhaps Mistral will get additional European government support, and we’ll have at least one offering from China like Qwen, and Bytedance and Tencent will continue to compete a-la Google and co. But, I don’t know if there’s a market for ten separately trained open foundation models long term.

I’d like to make sure there’s some diversity in research and implementation of these in the open access space. It’s a critical tool for humans, and it seems possible to me that leaders will be able to keep extending the gap for a while; when you’re using that gap not just to build faster AI, but do other things, the future feels pretty high volatility right now. Which is interesting! But, I’d prefer we come out of it with people all over the world having access to these.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#77

Post by meiraleal » Sat, Sep 21, 2024, 3:34 PM UTC

csmpltn wrote: post #21

brap wrote: post #14

Some companies (OpenAI, Anthropic…) base their whole business on hosted closed source models. What’s going to happen when all of this inevitably gets commoditized? This is why I’m putting my money on Google in the long run. They have the reach to make it useful and the monetization behemoth to make it profitable.

There's plenty of competition in this space already, and it'll only get accelerated with time. There's not enough "moat" in building proprietary LLMs - you can tell by how the leading companies in this space are basically down to fighting over patents and regulatory capture (ie. mounting legal and technical barriers to scraping, procuring hardware, locking down datasets, releasing less information to the public about…

And then the successful chatgpt wrappers with traction will become valuable than the companies creating propietary LLMs. I bet openai will start buying many AI apps to find profitable niches.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#78

Post by snovv_crash » Sat, Sep 21, 2024, 3:35 PM UTC

andai wrote: post #17

McBainiel wrote: post #9

> Microsoft used LLMs to write millions of short stories and textbooks in which one thing builds on another. The result of training on this text, Bubeck says, is a model that fits on a mobile phone but has the power of the initial 2022 version of ChatGPT. I thought training LLMs on content created by LLMs was ill-advised but this would suggest otherwise

Look into Microsoft's Phi papers. The whole idea here is that if you train models on higher quality data (i.e. textbooks instead of blogspam) you get higher quality results. The exact training is proprietary but they seem to use a lot of GPT-4 generated training data. On that note... I've often wondered if broad memorization of trivia is really a sensible use of precious neurons. It seems like a system trained on a n…

From what I've seen Phi does well in benchmarks but poorly in real world scenarios. They also made some odd decisions regarding the network structure which means that the memory requirements for larger context is really high.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#79

Post by jannyfer » Sat, Sep 21, 2024, 3:37 PM UTC

vessenes wrote: post #76

All this will be an interesting side note in the history of language models in the next eight months when roughly 1.5 billion iPhone users will get a local language model tied seamlessly to a mid-tier cloud based language model native in their OS. What I think will be interesting is seeing which of the open models stick around and for how long when we have super easy ‘good enough’ models that provide quality integrat…

> in the next eight months when roughly 1.5 billion iPhone users will get a local language model tied seamlessly to a mid-tier cloud based language model native in their OS.

Only iPhone 15 Pro or later will get Apple Intelligence, so the number will be wayyy smaller.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#80

Post by darby_nine » Sat, Sep 21, 2024, 3:37 PM UTC

vessenes wrote: post #76

All this will be an interesting side note in the history of language models in the next eight months when roughly 1.5 billion iPhone users will get a local language model tied seamlessly to a mid-tier cloud based language model native in their OS. What I think will be interesting is seeing which of the open models stick around and for how long when we have super easy ‘good enough’ models that provide quality integrat…

I expect people will just ship with their own model where the built-in one isn't sufficient.

When people describe it as a "critical tool" i feel like I'm missing basic information about how people use computers and interact with the world. In what way is it critical for anything? It's still just a toy at this point.