Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: LocalGPT – A local-first AI assistant in Rust with persistent memory

github.com

Post reply on HN
Print view View original

81–90 of 165 posts

Re: Show HN: LocalGPT – A local-first AI assistant in Rust with persistent memory

#82

Post by PeterStuer »

PunchyHamster wrote: post #75

if you have to put API key in it, it's not local

Most local systems use an OpenAI compatible API. This requires an API key to be set, even if it is not used. Just set it to "not-needed" or whatever you fancy.

Re: Show HN: LocalGPT – A local-first AI assistant in Rust with persistent memory

#83

Post by benob »

What local models shine as local assistants? Is there an effort to evaluate the compromise between compute/memory and local models that can support this use case? What kind of hardware do you need to not feel like playing with a useless shiny toy?

Re: Show HN: LocalGPT – A local-first AI assistant in Rust with persistent memory

#85

Post by T-A »

fy20 wrote: post #38
dvt wrote: post #4

So weird/cool/interesting/cyberpunk that we have stuff like this in the year of our Lord 2026: ├── MEMORY.md # Long-term knowledge (auto-loaded each session) ├── HEARTBEAT.md # Autonomous task queue ├── SOUL.md # Personality and behavioral guidance Say what you will, but AI really does feel like living in the future. As far as the project is concerned, pretty neat, but I'm not really sure about calling it "local-firs…

> Say what you will, but AI really does feel like living in the future. Love or hate it, the amount of money being put into AI really is our generation's equivalent of the Apollo program. Over the next few years there are over 100 gigawatt scale data centres planned to come online. At least it's a better use than money going into the military industry.

The Apollo program was peanuts in comparison:

https://www.wsj.com/tech/ai/ai-spending-tech-companies-compa...

https://www.reuters.com/graphics/USA-ECONOMY/AI-INVESTMENT/g...

Re: Show HN: LocalGPT – A local-first AI assistant in Rust with persistent memory

#86

Post by Sharlin »

jazzyjackson wrote: post #47
dvt wrote: post #4

So weird/cool/interesting/cyberpunk that we have stuff like this in the year of our Lord 2026: ├── MEMORY.md # Long-term knowledge (auto-loaded each session) ├── HEARTBEAT.md # Autonomous task queue ├── SOUL.md # Personality and behavioral guidance Say what you will, but AI really does feel like living in the future. As far as the project is concerned, pretty neat, but I'm not really sure about calling it "local-firs…

IMHO it doesn't make sense, financially and resource wise to run local, given the 5 figure upfront costs to get an LLM running slower than I can get for 20 USD/m. If I'm running a business and have some number of employees to make use of it, and confidentiality is worth something, sure, but am I really going to rely on anything less then the frontier models for automating critical tasks? Or roll my own on prem IT to…

That’s probably true only as long as subscription prices are kept artificially low. Once the $20 becomes $200 (or the fast-mode inference quotas for cheap subs become unusably small), the equation may change.

Re: Show HN: LocalGPT – A local-first AI assistant in Rust with persistent memory

#89

Post by Sharlin »

dvt wrote: post #35
embedding-shape wrote: post #17

Earlier quoted context omitted.

Ah come on, lazy? As long as it works with the runtime you wanna use, instead of hardcoding their own solution, should work fine. If you want to use Candle and have to implement new architectures with it to be able to use it, you still can, just expose it over HTTP.

I think one of the major problems with the current incarnation of AI solutions is that they're extremely brittle and hacked-together. It's a fun exciting time, especially for us technical people, but normies just want stuff to "work." Even copy-pasting an API key is probably too much of a hurdle for regular folks, let alone running a local ollama server in a Docker container.

Unlike in image/video gen, at least with LLMs the "best" solution available isn’t a graph/node-based interface with an ecosystem of hundreds of hacky undocumented custom nodes that break every few days and way too complex workflows made up of a spaghetti of two dozen nodes with numerous parameters each, half of which have no discernible effect on output quality and tweaking the rest is entirely trial and error.
Post reply on HN