Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I want everything local – Building my offline AI workspace

instavm.io

Post reply on HN
Print view View original

151–160 of 294 posts

Re: I want everything local – Building my offline AI workspace

#151

Post by jokethrowaway »

withinboredom wrote: post #119
zwnow wrote: post #41

Earlier quoted context omitted.

You know there's a ceiling to all this with the current LLM approaches right? They won't become that much better, its even more likely they will degrade. There are cases of bad actors attacking LLMs by feeding it false information and propaganda. I dont see this changing in the future.

I seeded all over the internet that a friend of mine was an elephant with the intention of poisoning the well, so to speak. (with his permission, of course) That was in 2021. Today if you ask who my friend is, it tells you that he is an elephant, without even doing a web search. I wouldn’t be surprised if people are doing this with more serious things.

Looks like they patched it (tested on Claude, ChatGPT; I assume it's Rob) but your point is very valid.

Re: I want everything local – Building my offline AI workspace

#152

Post by esseph »

thorncorona wrote: post #92
com2kid wrote: post #55

> Even with help from the "world's best" LLMs, things didn't go quite as smoothly as we had expected. They hallucinated steps, missed platform-specific quirks, and often left us worse off. This shows how little native app training data is even available. People rarely write blog posts about designing native apps, long winded medium tutorials don't exist, heck even the number of open source projects for native desktop…

I mean outside of HPC why would you when the browser is the world’s most ubiquitous VM?

Because the browser is gross and you can reclaim lot of performance and security when you don't need to use it.

Re: I want everything local – Building my offline AI workspace

#153

Post by esseph »

esseph wrote: post #152
thorncorona wrote: post #92

Earlier quoted context omitted.

I mean outside of HPC why would you when the browser is the world’s most ubiquitous VM?

Because the browser is gross and you can reclaim lot of performance and security when you don't need to use it.

I mean, why aren't the apps on your phone all just webapps, right? (Also, eww)

Re: I want everything local – Building my offline AI workspace

#154

Post by nemomarx »

esseph wrote: post #147
jauntywundrkind wrote: post #128

Earlier quoted context omitted.

Thanks for the numbers. Valuable contribution for sure!! There's been a huge lag for PCIe adoption, and imo so so much has boiled down "do people need it"? In the past 10 years I feel like my eyes have been opened that every high tech company's greatest highest most compelling desire is to slow walk the release out. To move as slow as the market will bear, to do as little as possible, to roll on and on with minor inc…

All of these extremely high end technologies are so far away from hitting the consumer market. Is there any desire for most people? What's the TAM?

Probably small consumer market of enthusiasts (notice Nvidia barely caters to gaming hardware lately) but if you can get better memory throughput on servers isn't that a large industry market?

Re: I want everything local – Building my offline AI workspace

#155

Post by wkat4242 »

Imustaskforhelp wrote: post #29
frank_nitti wrote: post #22

Earlier quoted context omitted.

I'd assume they are referring to being able to run your own workloads in a home-built system, rather then surrendering that ownership to the tech giants alone

Also you get a sort of complete privacy that the data never leaves your home too whereas at best you would have to trust the AI cloud providers that they are not training or storing that data. Its just more freedom and privacy in that matter.

> whereas at best you would have to trust the AI cloud providers that they are not training or storing that data.

Yeah, about that. They even illegally torrented entire databases, hide their crawlers. Crawl entire newspaper archives without permission. They didn't respect the rights of big media companies. But they're going to respect the little guy's of course because it says to in the T&Cs. Uh-huh.

Also, openai already admitted that they do store "deleted" content and temporary chats.

Re: I want everything local – Building my offline AI workspace

#157

Post by varenc »

notsylver wrote: post #149
andylizf wrote: post #72

Earlier quoted context omitted.

Yeah, that's a fair point at first glance. 50GB might not sound like a huge burden for a modern SSD. However, the 50GB figure was just a starting point for emails. A true "local Jarvis," would need to index everything: all your code repositories, documents, notes, and chat histories. That raw data can easily be hundreds of gigabytes. For a 200GB text corpus, a traditional vector index can swell to >500GB. At that poi…

You already need very high end hardware to run useful local LLMs, I don't know if a 200gb vector database will be the dealbreaker in that scenario. But I wonder how small you could get it with compression and quantization on top

> You already need very high end hardware to run useful local LLMs

A basic macbook can run gpt-oss-20b and it's quite useful for many tasks. And fast. Of course Macs have a huge advantage for local LLMs inference due to their shared memory architecture.

Re: I want everything local – Building my offline AI workspace

#158

Post by OneDeuxTriSeiGo »

notsylver wrote: post #149
andylizf wrote: post #72

Earlier quoted context omitted.

Yeah, that's a fair point at first glance. 50GB might not sound like a huge burden for a modern SSD. However, the 50GB figure was just a starting point for emails. A true "local Jarvis," would need to index everything: all your code repositories, documents, notes, and chat histories. That raw data can easily be hundreds of gigabytes. For a 200GB text corpus, a traditional vector index can swell to >500GB. At that poi…

You already need very high end hardware to run useful local LLMs, I don't know if a 200gb vector database will be the dealbreaker in that scenario. But I wonder how small you could get it with compression and quantization on top

You can already do A LOT with an SLM running on commodity consumer hardware. Also it's important to consider that the bigger an embedding is, the more bandwidth you need to use it at any reasonable speed. And while storage may be "cheap", memory bandwidth absolutely is not.

Re: I want everything local – Building my offline AI workspace

#160

Post by jakelazaroff »

esseph wrote: post #153
esseph wrote: post #152

Earlier quoted context omitted.

Because the browser is gross and you can reclaim lot of performance and security when you don't need to use it.

I mean, why aren't the apps on your phone all just webapps, right? (Also, eww)

Mostly because native apps can track you far more invasively than web apps can, and companies are hungry for your private data.
Post reply on HN