Curious with the hardware used in this article.
I want everything local – Building my offline AI workspace
221–230 of 294 posts
Re: I want everything local – Building my offline AI workspace
#222Earlier quoted context omitted.
Speak for yourself! If it took me 500GB to store my vectors , on top of all my existing data, it would be a huge barrier for me.
A 4tb external drive is £100. A 1TB sd card or usb stick a similar cost. Maybe Im too old to appreciate what “fast” means, but storage doesnt seem an enormous cost once you stripe it.
This "...doesn't seem an enormous cost once you stripe it." gave me an idea.
I KNOW that I will come back to link a blog post about it in the future.
Re: I want everything local – Building my offline AI workspace
#223local/edge is the most under-valued space at the moment. incredible computing power that dwarfs datacenters, zero latency, zero cost, private, distributed and resilient
I guess you imagine a world like Skype supernodes (Skype gave that up more than a decade ago) or Tor nodes (Tor is used by a tiny fraction of internet users). Not saying it can't be done, but the effort is humongous.
no, I mean I saw multiple companies at this point with their entier K8S cluster... is smaller than single new macbook pro :/
now, if you have 100,000 users with latest iPhone, say you use 10GB RAM in each, using A16 chip with 1.9 TFLOPS, each with 5G connection
this is 1 Peta-Byte RAM + 0.25 Peta-FLOPs GPU + 4 TB / second bandwidth
at zero cost (no-upfront, no-maintenance, users pay for, upgrade, and maintain their phones working, pay for internet, charging with electricity, cooling? - thanks!)
... it goes even wilder if you use macbooks
... and if you consider say mid-size town in China with population of 15 million, you go Exa-scale
and consider that for now iPhones are just sitting idle. for now.
Re: I want everything local – Building my offline AI workspace
#224Great effort, a strong self-hosting community for LLMs is going to be similarly important as the FLOSS movement imho. But right now I feel the bigger bottleneck is on the hardware side rather than software. The amount of fast RAM that you need for decent models (80b+ params) is just not something that's commonly available for consumer hardware right now, not even gaming machines. I heard that Macs (minis) are great for the purpose, but you don't really get them with enough RAM or at prices that don't really qualify as consumer-grade anymore. I've seen people create home clusters (eg using Exo [0]), but I wouldn't really call it practical (single digit token/sec for large models, and the price isn't exactly accessible either). Framework (the modular laptop company) has announced a desktop that can be configured up to 128GB unified RAM, but it's still going to come in at around 2-2.5k depending on your config.
Re: I want everything local – Building my offline AI workspace
#225Local is important for compliance with GDPR and closed source software
I hate sending my code to openAI or my client's code.
I find local llms to be usable for short snippets but still too slow for a lot of things.
I just spent hours debugging code mistral ai gave me and had multiple errors, rtfm is still most of the times better than relying on an llm
Re: I want everything local – Building my offline AI workspace
#226Earlier quoted context omitted.
Mostly because native apps can track you far more invasively than web apps can, and companies are hungry for your private data.
Not sure I agree with that. It's a lot better on battery life and superior experience, especially if you are traveling or around areas with bad cell service. Cookies track me around on websites all the time + modern telemetry is pretty crazy.
>Not sure I agree with that.
It's a lot better on battery life...
The parent is talking about privacy and your first counter argument is privacy irrelevant battery life?
The tracking and telemetry abundance in native far exceeds the browser. Nevermind a lot of apps remain running in background because the user forgets or can't be bothered to close them.
Follow the money. Why are random companies begging me to download their mobile app and get ridiculous discounts in the process whenever I use their website? Why are weather apps known to be spyware vectors but weather websites don't have that stigma?
Re: I want everything local – Building my offline AI workspace
#227So there are models to download but:
a) on what data that things was trained ?
b) any reproducible builds projects ? ;)
Re: I want everything local – Building my offline AI workspace
#228This is fantastic work. The focus on a local, sandboxed execution layer is a huge piece of the puzzle for a private AI workspace. The `coderunner` tool looks incredibly useful. A complementary challenge is the knowledge layer: making the AI aware of your personal data (emails, notes, files) via RAG. As soon as you try this on a large scale, storage becomes a massive bottleneck. A vector database for years of emails c…
I'm gonna put it here for visibility: Use patchright instead of Playwright: https://github.com/Kaliiiiiiiiii-Vinyzu/patchright
What problem does patchright solve?
Re: I want everything local – Building my offline AI workspace
#229Local AI is awesome, but without beefy hardware it’s like trying to run a marathon in flip-flops.
Re: I want everything local – Building my offline AI workspace
#230[dead]