Earlier quoted context omitted.
That's just meat CoT (chain of thought) - right?
I do not understand?
[deleted]
Forget ChatGPT: why researchers now run small AIs on their laptops
221–230 of 385 posts
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#222Earlier quoted context omitted.
From this data we must conclude that within our lifetimes all matter in the universe will eventually be reprogrammed in JavaScript.
I'm not sure I want to live in that reality.
If the simulation hypothesis is real, perhaps it would follow that all the dark matter and dark energy in the universe is really just extra cycles being burned on layers of interpreters and JIT compilation of a loosely-typed scripting language.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#223For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…
Not only are they the only future worth living in, incentives are aligned with client-side AI. For governments and government contractors, plumbing confidential information through a network isn't an option, let alone spewing it across the internet. It's a non-starter, regardless of the productivity bumps stuff like Copilot can provide. The only solution is to put AI compute on a cleared individual's work computer.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#224Earlier quoted context omitted.
I would be greatly interested in knowing how you set all that up if you felt like sharing the specifics.
My hope is to make this easy with a GH repo or at least detailed instructions. I'm on a Mac and I found the easiest way to run & use local models is Ollama as it has a rest interface: https://github.com/ollama/ollama/blob/main/docs/api.md I just have a local script that pulls the audio file from Voice Memos (after it syncs from my iPhone), runs it through openai's whisper (really the best at voice to speech; excellen…
Have you tried the Shortcuts app? On phone and mac. Should be able to make one that finds and moves a voice memo when run. You can run them on button press or via automation.
Also what kind of local machine do you need? I have an imac pro, wondering if this will run the models or if I ought to be on an apple silicon machine? I have an M1 macbook air as well.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#225My Thesis: Small language models (SLM)— models so compact that you can run them on a computer with just 4GB of RAM — are the future. SLMs are efficient enough to be deployed on edge devices, while still maintaining enough intelligence to be useful.
https://www.lycee.ai/blog/why-small-language-models-are-the-...
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#226[flagged]
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#227Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#228For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…
Yeah I set up a local server with a strong GPU but even without that it's ok, just a lot slower. The biggest benefits for me are the uncensored models. I'm pretty kinky so the regular models tend to shut me out way too much, they all enforce this prudish victorian mentality that seems to be prevalent in the US but not where I live. Censored models are just unusable to me which includes all the hosted models. It's jus…
Any tips you can give for like minded folks? Besides grimjim (checking it out).
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#229Earlier quoted context omitted.
If you're gonna go with a VS code extension and you're aiming for privacy, then I would at least recommend using the open source fork VS Codium. https://vscodium.com/
It is true that VS Code has some non-optional telemetry, and if VS Codium works for people, that is great. However, the telemetry of VSCode is non-personal metrics, and some of the most popular extensions are only available with VSCode, not with Codium.
From the documentation (https://code.visualstudio.com/docs/getstarted/telemetry ) it seems there is a supported way to completely turn off telemetry. Is there something else in VSCode that doesn't respect this setting?
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#230Earlier quoted context omitted.
Have you tried a Llamafile? Not sure what platform you are using. From their readme: > … by combining llama.cpp with Cosmopolitan Libc into one framework that collapses all the complexity of LLMs down to a single-file executable (called a "llamafile") that runs locally on most computers, with no installation. Low cost to experiment IMO. I am personally using MacOS with an M1 chip and 64gb memory and it works perfectl…
With 64GB can you run the 70B size llama models well?
I should have qualified the meaning of “works perfectly” :) No 70b for me, but I am able to experiment with many quantized models (and I am using a Llama successfully, latency isn’t terrible)