Forget ChatGPT: why researchers now run small AIs on their laptops • Hacker News Board

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#101

Post by staticman2 » Sat, Sep 21, 2024, 4:08 PM UTC

McBainiel wrote: post #9

> Microsoft used LLMs to write millions of short stories and textbooks in which one thing builds on another. The result of training on this text, Bubeck says, is a model that fits on a mobile phone but has the power of the initial 2022 version of ChatGPT. I thought training LLMs on content created by LLMs was ill-advised but this would suggest otherwise

There's been efforts to train small LLM's on bigger LLM's. Ever since Llama came out the community was creating custom fine tunes this way using ChatGPT.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#102

Post by toddmorey » Sat, Sep 21, 2024, 4:08 PM UTC

I narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company.

[1] It feels super strange to talk to yourself, but luckily I'm out early enough that I'm often alone. Worst case, I pretend I'm talking to someone on the phone.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#103

Post by racked » Sat, Sep 21, 2024, 4:14 PM UTC

toddmorey wrote: post #102

I narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company. [1] It fee…

What software did you use to set all this up? Kindof interested in giving this a shot myself.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#104

Post by azeirah » Sat, Sep 21, 2024, 4:16 PM UTC

racked wrote: post #103

toddmorey wrote: post #102

I narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company. [1] It fee…

What software did you use to set all this up? Kindof interested in giving this a shot myself.

You can use llama.cpp, it runs on almost all hardware. Whisper.cpp is similar, but unless you have a mid or high end nvidia card it will be a bit slower.

Still very reasonable on modern hardware.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#105

Post by alyandon » Sat, Sep 21, 2024, 4:16 PM UTC

toddmorey wrote: post #102

I narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company. [1] It fee…

I would be greatly interested in knowing how you set all that up if you felt like sharing the specifics.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#106

Post by vincvinc » Sat, Sep 21, 2024, 4:18 PM UTC

toddmorey wrote: post #102

I narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company. [1] It fee…

I was thinking about making this the other day. Would you mind sharing what you used?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#107

Post by noman-land » Sat, Sep 21, 2024, 4:18 PM UTC

dockerd wrote: post #53

What spec people recommend here to run small models like Llama3.1 or mistral-nemo etc. Also is it sensible to wait for newer mac, amd, nvidia hardware releasing soon?

You basically need as much RAM as the size of the model.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#108

Post by wazdra » Sat, Sep 21, 2024, 4:19 PM UTC

I'd like to point out that llama 3.1 is not open source[1] (I was recently made aware of that fact by [2], when it was on HN front page) While it's very nice to see a peak of interest for local, "open-weights" LLMs, this is an unfortunate choice of words, as it undermines the quite important differences between llama's license model and open-source. The license question does not seem to be addressed at all in the article.

[1]: https://www.llama.com/llama3_1/license/

[2]: https://csvbase.com/blog/14

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#109

Post by schmidtleonard » Sat, Sep 21, 2024, 4:23 PM UTC

toddmorey wrote: post #102

I narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company. [1] It fee…

Button-toggled voice notes in the iPhone Notes app are a godsend for taking measurements. Rather than switching your hands between probe/equipment and notes repeatedly, which sucks badly, you can just dictate your readings and maaaaybe clean out something someone said in the background. Over the last decade, the microphones + speech recognition became Good Enough for this. Wake-word/endpoint models still aren't there yet, and they aren't really close, but the stupid on/off button in the Notes app 100% solves this problem and the workflow is now viable.

I love it and I sincerely hope that "Apple Intelligence" won't kill the button and replace it with a sub-viable conversational model, but I probably ought to figure out local whisper sooner rather than later because it's probably inevitable.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#110

Post by pimeys » Sat, Sep 21, 2024, 4:24 PM UTC

Has anybody found a good way to utilize ollama with an editor such as zed to do things like "generate rustdoc to this method" etc. I use ollama daily for a ton of things, but for code generation, completion and documentation 4o is still much better than any of the local models...