Forget ChatGPT: why researchers now run small AIs on their laptops

Post by wkat4242 » Sat, Sep 21, 2024, 8:40 PM UTC

wkat4242 wrote: post #181

noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

Yeah I set up a local server with a strong GPU but even without that it's ok, just a lot slower. The biggest benefits for me are the uncensored models. I'm pretty kinky so the regular models tend to shut me out way too much, they all enforce this prudish victorian mentality that seems to be prevalent in the US but not where I live. Censored models are just unusable to me which includes all the hosted models. It's jus…

Post by ycombinatrix » Sat, Sep 21, 2024, 8:48 PM UTC

zozbot234 wrote: post #123

noman-land wrote: post #107

Earlier quoted context omitted.

You basically need as much RAM as the size of the model.

You actually need a lot less than that if you use the mmap option, because then only activations need to be stored in RAM, the model itself can be read from disk.

Post by whimsicalism » Sat, Sep 21, 2024, 9:01 PM UTC

101008 wrote: post #87

sparky_ wrote: post #71

Earlier quoted context omitted.

This is the best take on the generative AI fad I've yet seen. I wish I could upvote this twice.

I had the same impression. I have been suffering a lot lately about the future for engineers (not having work, etc), even habing anxiety when I read news about AI , but these comments make me feel better and relaxed. I even considered blocking HN.

Post by emadda » Sat, Sep 21, 2024, 9:07 PM UTC

toddmorey wrote: post #178

alyandon wrote: post #105

Earlier quoted context omitted.

I would be greatly interested in knowing how you set all that up if you felt like sharing the specifics.

My hope is to make this easy with a GH repo or at least detailed instructions. I'm on a Mac and I found the easiest way to run & use local models is Ollama as it has a rest interface: https://github.com/ollama/ollama/blob/main/docs/api.md I just have a local script that pulls the audio file from Voice Memos (after it syncs from my iPhone), runs it through openai's whisper (really the best at voice to speech; excellen…

Post by adamsmith » Sat, Sep 21, 2024, 9:10 PM UTC

vunderba wrote: post #145

toddmorey wrote: post #102

I narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company. [1] It fee…

Same. My husky/pyr mix needs a lot of exercise, so I'm outside a minimum of a few hours a day. As a result I do a lot of dictation on my phone. I put together a script that takes any audio file (mp3, wav), normalizes it, runs it through ggerganov's whisper, and then cleans it up using a local LLM. This has saved me a tremendous amount of time. Even modestly sized 7b parameter models can handle syntactical/grammatical…