Forget ChatGPT: why researchers now run small AIs on their laptops

Post by CooCooCaCha » Sun, Sep 22, 2024, 12:04 AM UTC

ryukoposting wrote: post #223

noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

Not only are they the only future worth living in, incentives are aligned with client-side AI. For governments and government contractors, plumbing confidential information through a network isn't an option, let alone spewing it across the internet. It's a non-starter, regardless of the productivity bumps stuff like Copilot can provide. The only solution is to put AI compute on a cleared individual's work computer.

Post by jaggederest » Sun, Sep 22, 2024, 12:30 AM UTC

unethical_ban wrote: post #175

vunderba wrote: post #170

Earlier quoted context omitted.

If you're gonna go with a VS code extension and you're aiming for privacy, then I would at least recommend using the open source fork VS Codium. https://vscodium.com/

It is true that VS Code has some non-optional telemetry, and if VS Codium works for people, that is great. However, the telemetry of VSCode is non-personal metrics, and some of the most popular extensions are only available with VSCode, not with Codium.

Post by Jedd » Sun, Sep 22, 2024, 12:51 AM UTC

kaoD wrote: post #194

noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

Well this was my experience... User: Hey, how are you? Llama: [object Object] It's funny but I don't think I did anything wrong?

Post by antifa » Sun, Sep 22, 2024, 1:14 AM UTC

vunderba wrote: post #158

pimeys wrote: post #140

Earlier quoted context omitted.

Zed has also support for ollama, but all the local models I tried do not really work so well to things like "write docs for this method"... Also local editor autocomplete in the style of github copilot would be great, without needing to use proprietary Microsoft tooling...

There's a lot of plugins/IDEs for assistant style LLMs, but the only TAB style autocompletion ones I know of are either proprietary (Github Copilot), or you need to get an API key (Codestral). If anyone knows of a local autocomplete model I'd love to hear about it. The Continue extension (Jetbrains, VSCodium) lets you set up assistant and autocompletion independently with different API keys.

Post by Havoc » Sun, Sep 22, 2024, 1:17 AM UTC

atentaten wrote: post #276

I'm interested in running locally, but I haven't found consistent advice on hardware specs for optimal performance. I would like to build a server with the best GPU and tons of RAM to run and experiment with these models.