For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…
Forget ChatGPT: why researchers now run small AIs on their laptops
141–150 of 385 posts
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#142Does anyone know of a local "Siri" implementation? Whisper + Llama (or Phi or something else), that can run shortcuts, take notes, read web pages etc.? PS: for reading web pages I know there's voices integrated in the browser/OS but those are horrible
Apple intelligence?
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#143For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…
Isn’t there also some Firefox AI integration that’s being tested by one dev out there? I forgot the name and wonder if it got any traction.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#144Does anyone know of a local "Siri" implementation? Whisper + Llama (or Phi or something else), that can run shortcuts, take notes, read web pages etc.? PS: for reading web pages I know there's voices integrated in the browser/OS but those are horrible
Apple intelligence?
It isn't clear if you can know when the task gets handed off to their servers. But yeah that'd be the closest I know. I'm not sure it would build a local knowledge base though.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#145I narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company. [1] It fee…
Same. My husky/pyr mix needs a lot of exercise, so I'm outside a minimum of a few hours a day. As a result I do a lot of dictation on my phone.
I put together a script that takes any audio file (mp3, wav), normalizes it, runs it through ggerganov's whisper, and then cleans it up using a local LLM. This has saved me a tremendous amount of time. Even modestly sized 7b parameter models can handle syntactical/grammatical work relatively easily.
Here's the gist:
https://gist.github.com/scpedicini/455409fe7656d3cca8959c123...
EDIT: I've always talked out loud through problems anyway, throw a BT earbud on and you'll look slightly less deranged.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#146Earlier quoted context omitted.
which model are you using? what size/quant/etc? thanks!
Not the parent, but I started using Llama 3.1 8b and it's very good. I'd say it's as good as or better than GPT 3.5 based on my usage. Some benchmarks: https://ai.meta.com/blog/meta-llama-3-1/ Looking forward to try other models like Qwen and Phi in near future.
I found it to not be as good in my case for code generation and suggestions. I am using a quantized version maybe that's the difference.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#147The Mistral models are not half as bad for this.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#148I narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company. [1] It fee…
"before having an LLM clean up my ramblings into organized notes and todo lists."
Which local LLM do you use?
Edit:
And self talk is quite a healthy and useful thing in itself, but avoiding it in public is indeed kind of necessary, because of the stigma
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#149I need to have two things of my own that work offline for privacy concerns and cost savings:
1. Local LLM AI models with GUI and command line
2. > Local LLM-based coding tools do exist (such as Google DeepMind’s CodeGemma and one from California-based developers Continue)
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#150I narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company. [1] It fee…
I found one that can isolate speakers, its just okay at that