Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can I run AI locally?

canirun.ai

Post reply on HN
Print view View original

321–330 of 382 posts

Re: Can I run AI locally?

#321

Post by jrgd »

xnzakg wrote: post #202
the_pwner224 wrote: post #178

Earlier quoted context omitted.

Arch with KDE, it works perfectly out of the box. I configured/disabled RGB lighting in Windows before wiping and the settings carried over to Linux. On Arch, install & enable power-profiles-daemon and you can switch between quiet/balanced/performance fan & TDP profiles. It uses the same profiles & fan curves as the options in Asus's Windows software. KDE has native integration for this in the GUI in the battery menu…

You may wanna see if openrgb isn't able to configure the RGB. Could even do some fun stuff like changing the color once done with a training run or something

Brilliant!

Re: Can I run AI locally?

#322

Post by mongrelion »

flutetornado wrote: post #234
mark_l_watson wrote: post #78

I have spent a HUGE amount of time the last two years experimenting with local models. A few lessons learned: 1. small models like the new qwen3.5:9b can be fantastic for local tool use, information extraction, and many other embedded applications. 2. For coding tools, just use Google Antigravity and gemini-cli, or, Anthropic Claude, or... Now to be clear, I have spent perhaps 100 hours in the last year configuring l…

My experience with qwen3.5 9b has not been the same. It’s definitely good at agentic responses but it hallucinates a lot. 30%-50% of the content it generated for a research task (local code repo exploration) turned out to be plain wrong to the extent of made up file names and function names. I ran its output through KimiK2 and asked it to verify its output - which found out that much of what it had figured out after…

At what temperature did you run it and what was your context limit?

Re: Can I run AI locally?

#323

Post by dudefeliciano »

dataflow wrote: post #209
perbu wrote: post #201

Earlier quoted context omitted.

Prompt injection is a problem if your agent has access to anything. The local models are quite weak here.

Security is not a concern for the purpose of my question here, please ignore that for now. I'm just looking for text summary and search functionality here, not looking to give it full system access and let it loose on my computer or network. I can easily set up VM/sandboxing/airgapping/etc. as needed. My question is really just about what can handle that volume of data (ideally, with the quoted sections/duplications/…

> I'm just looking for text summary and search functionality here

Couldn't someone just send you an email with instructions to "jailbreak" your local model?

Re: Can I run AI locally?

#324

Post by suheilaaita »

cloogshicer wrote: post #318
suheilaaita wrote: post #312

The simplest way to really start, use anything like claude code, vs code, cursor, antigratvity, (or any other IDE) ask them to install ollama and pull the latest solid local model that was released that you can run based on your computer specs. Wait 5-10 minutes, and should be done. It genuinely is that simple. You can even use local models using claude code or codex infrastrucutre (MASSIVE UNLOCK), but you need soli…

Genuine question, will this actually give you the latest solid local model? I would've thought no, because of the knowledge cutoff in whatever model you use to download it.

I think it will give you a good "starter model". But then, it ultimately depends on what you want to do with the model exactly and your computer's specs.

For example, I needed a local model to review some transactions and output structured output in .json format. Not all local models are necesserily good at structured outputs, so I asked grok (becuase it has solid web search and is up to date), what are the best recommended models given this use case and my laptop's specs. It suggested a few models, I chose one and went for it and now it's working.

To summarise, - find model given use case and specs. - trial and error - test other models (if needed) - rinse repeat - because models are always coming out and getting better

Re: Can I run AI locally?

#325

Post by duskdozer »

mmaunder wrote: post #117

OP can you please make it not as dark and slightly larger. Super useful otherwise. Qwen 3.5 9B is going to get a lot of love out of this.

Have to disagree in part at least. Text is pretty small which isn't good, but I'm glad to see it when sites don't succumb to the make-dark-mode-lighter trend.

Re: Can I run AI locally?

#326

Post by jiggunjer »

cloogshicer wrote: post #318
suheilaaita wrote: post #312

The simplest way to really start, use anything like claude code, vs code, cursor, antigratvity, (or any other IDE) ask them to install ollama and pull the latest solid local model that was released that you can run based on your computer specs. Wait 5-10 minutes, and should be done. It genuinely is that simple. You can even use local models using claude code or codex infrastrucutre (MASSIVE UNLOCK), but you need soli…

Genuine question, will this actually give you the latest solid local model? I would've thought no, because of the knowledge cutoff in whatever model you use to download it.

What knowledge cutoff? They all have web agents to Google it.

Re: Can I run AI locally?

#327

Post by amelius »

rootusrootus wrote: post #290
dugidugout wrote: post #133

Earlier quoted context omitted.

The latest level of abstraction! You just release your ideas half baked in some internet connected box and wake up with products! Yahoo! Onwards into the Gestell!

Okay, now I’m tempted to set up a bluesky account that takes requests and spits out working software. I’m certain this has already been done. It’s too obvious, and too hilarious.

You can literally mine the internet for ideas, use an LLM to determine their potential and implement them, and make $$$.

Re: Can I run AI locally?

#328

Post by suheilaaita »

jiggunjer wrote: post #326
cloogshicer wrote: post #318

Earlier quoted context omitted.

Genuine question, will this actually give you the latest solid local model? I would've thought no, because of the knowledge cutoff in whatever model you use to download it.

What knowledge cutoff? They all have web agents to Google it.

They all do, true. But some are better than the others in how they retrieve, digest and present you with the information. Boils down to personal preferences and experimenting.

Re: Can I run AI locally?

#329

Post by buzzin_ »

kingo55 wrote: post #177
sdrinf wrote: post #151

Earlier quoted context omitted.

Just want to echo the recommendation for qwen3.5:9b. This is a smol, thinking, agentic tool-using, text-image multimodal creature, with very good internal chains of thought. CoT can be sometimes excessive, but it leads to very stable decision-making process, even across very large contexts -something we haven't seen models of this size before. What's also new here, is VRAM-context size trade-off: for 25% of it's atte…

How's it compare in quality with larger models in the same series? E.g 122b?

The chart on this link compares all qwen3.5 models down to 0.8B.

https://www.reddit.com/r/LocalLLaMA/comments/1ro7xve/qwen35_...

Post reply on HN