Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forget ChatGPT: why researchers now run small AIs on their laptops

nature.com

Post reply on HN
Print view View original

201–210 of 385 posts

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#201

Post by evilduck »

meiraleal wrote: post #74
albertgoeswoof wrote: post #62

I have a three year old M1 Max, 32gb RAM. Llama 8bn runs at 25 tokens/sec, that’s fast enough, and covers 80% of what I need. On my ryzen 5600h machine, I get about 10 tokens/second, which is slow enough to be annoying. If I get stuck on a problem, switch to chat gpt or phind.com and see what that gives. Sometimes, it’s not the LLM that helps, but changing the context and rewriting the question. However I cannot use…

We need browser and OS level API (mobile) integration to the local LLM.

Both are in their early stages:

https://developer.chrome.com/docs/ai

https://developer.apple.com/documentation/AppIntents/Integra...

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#202

Post by AlienRobot »

kaoD wrote: post #194
noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

Well this was my experience... User: Hey, how are you? Llama: [object Object] It's funny but I don't think I did anything wrong?

2000: Javascript is webpages.

2010: Javascript is webservers.

2020: Javascript is desktop applications.

2024: Javascript is AI.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#203

Post by privacyis1mp »

twh270 wrote: post #198
privacyis1mp wrote: post #161

Earlier quoted context omitted.

I built Fluid app exactly with that in mind. You can run local AI on mac without really knowing what an LLM/ollama is. Plug&Play. Sorry for the blatant ad, though I do hope it's useful for some ppl reading this thread: https://getfluid.app

I'm interested, but I can't find any documentation for it. Can I give it local content (documents, spreadsheets, code, etc.) and ask questions?

> Can I give it local content (documents, spreadsheets, code, etc.) It's coming roughly in December (may be sooner).

Roadmap is following:

- October - private remote AI (when you need smarter AI than your machine can handle, but don't want your data to be logged or stored anywhere)

- November - Web search capabilities (so the AI will be capable of doing websearch out of the box)

- December - PDF, docs, code embedding. 2025 - tighter MacOS integration with context awareness.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#204

Post by MrDrMcCoy »

talldayo wrote: post #139
Larrikin wrote: post #116

Earlier quoted context omitted.

Why would anyone buy a Raspberry Pi when they can get a fully decked out Mac Pro? There are different use cases and computers are already pretty powerful. Maybe your local model won't be able to produce tests that check all the corner cases of the class you just wrote for work in your massive code base. But the small model is perfectly capable of summarizing the weather from an API call and maybe tack on a joke that…

> Why would anyone buy a Raspberry Pi when they can get a fully decked out Mac Pro? They want compliant Linux drivers?

Since when did Broadcom provide those?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#205

Post by jxcl »

toddmorey wrote: post #102

I narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company. [1] It fee…

This has inspired me.

I do a lot of stargazing and have experimented with voice memos for recording my observations. The problem of course is later going back and listening to the voice memo and getting organized information out of what essentially turns into me rambling to myself.

I'm going to try to use whisper + AI to transcribe my voice memos into structured notes.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#206

Post by sparkybrap »

create-username wrote: post #159

there's no small AI that I know of and masters ancient Greek, Latin, English, German and French and that I can run on my 18 GB macbook pro. Please correct me if I'm wrong. It would make my life slightly more comfortable

I agree. Even bi-lingual (English+1) small models would be very useful to process localized data, for ex english-french, english-german, etc.

Right now the small models (llama 8B) can't handle this type of task, although they could if they were trained for bi lingual data.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#207

Post by evbogue »

AlienRobot wrote: post #202
kaoD wrote: post #194

Earlier quoted context omitted.

Well this was my experience... User: Hey, how are you? Llama: [object Object] It's funny but I don't think I did anything wrong?

2000: Javascript is webpages. 2010: Javascript is webservers. 2020: Javascript is desktop applications. 2024: Javascript is AI.

From this data we must conclude that within our lifetimes all matter in the universe will eventually be reprogrammed in JavaScript.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#208

Post by pieix »

sandspar wrote: post #165

What advantages do local models have over exterior models? Why would I run one locally if ChatGPT works well?

1) Offline connectivity — pretty cool to be able to debug technical problems while flying (or otherwise off grid) with a local LLM, and current 8B models are usually good enough for the first line of questions that you otherwise would have googled.

2) Privacy

3) Removing safety filters — there are some great “abliterated” models out there that have had their refusal behavior removed. Running these locally and never having your request refused due to corporate risk aversion is a very different experience to calling a safety-neutered API.

Depending on your use case some, all, or none of these will be relevant, but they are undeniable benefits that are very much within reach using a laptop and the current crop of models.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#209

Post by the_gorilla »

wkat4242 wrote: post #181
noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

Yeah I set up a local server with a strong GPU but even without that it's ok, just a lot slower. The biggest benefits for me are the uncensored models. I'm pretty kinky so the regular models tend to shut me out way too much, they all enforce this prudish victorian mentality that seems to be prevalent in the US but not where I live. Censored models are just unusable to me which includes all the hosted models. It's jus…

[flagged]

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#210

Post by earslap »

IshKebab wrote: post #82
pella wrote: post #69

Llama 3.1 405B "2 MacBooks is all you need. Llama 3.1 405B running distributed across 2 MacBooks using @exolabs_ home AI cluster" https://x.com/AIatMeta/status/1834633042339741961

"All you need is £10k of Apple laptops..."

yes but still, a local model, a lightning in a bottle that is between GPT3.5 and GPT4 (closer to 4), yours forever, for about that price is pretty good deal today. probably won't be a good deal in a couple years but for the value, it is not that unsettling. When ChatGPT first launched 2 years ago we all wondered what it would take to have something close to that locally with no strings attached, and turns out it is "a couple years and about $10k" (all due to open weights provided by some companies, training such a model still costs millions) which is neat. It will never be more expensive.
Post reply on HN