Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can I run AI locally?

canirun.ai

Post reply on HN
Print view View original

251–260 of 382 posts

Re: Can I run AI locally?

#251

Post by constantinum »

girvo wrote: post #184
echelon wrote: post #144

Earlier quoted context omitted.

Shouldn't we prioritize large scale open weights and open source cloud infra? An OpenRunPod with decent usage might encourage more non-leading labs to dump foundation models into the commons. We just need infra to run it. Distilling them down to desktop is a fool's errand. They're meant to run on DC compute. I'm fine with running everything in the cloud as long as we own the software infra and the weights. This is co…

I run Qwen3.5-plus through Alibaba’s coding plan (Model Studio): incredibly cheap, pretty fast, and decent. I can’t compare it to the highest released weight one though.

I also want to try Qwen 3.5 plus. I have a doubt, I see almost same pricing for both Qwen and Claude code(the difference being the highest pro plan looks cheaper), and not for the lower plans. Am I missing something, when you say “cheaper” ??

Re: Can I run AI locally?

#253

Post by steve_adams_86 »

tempoponet wrote: post #236
steve_adams_86 wrote: post #176

Earlier quoted context omitted.

I've been building a harness for qwen3.5:9b lately (to better understand how to create agentic tools/have fun) and I'm not going to use it instead of Opus 4.6 for my day job but it's remarkably useful for small tasks. And more than snappy enough on my equipment. It's a fun model to experiment with. I was previously using an old model from Meta and the contrast in capability is pretty crazy. I like the idea of finding…

What kind of small tasks do you find it's good at? My non-coding use of agents has been related to server admin, and my local-llm use-case is for 24/7 tasks that would be cost-prohibitive. So my best guess for this would be monitoring logs, security cameras, and general home automation tasks.

That's about it. The harness is still pretty rudimentary so I'm sure the system could be more capable, and that might reveal more interesting opportunities. I don't really know.

So far I've got it orchestrating a few instances to dig through logs, local emails, git repositories, and github to figure out what I've been doing and what I need to do. Opus is waayyy better at it, but Qwen does a good enough job to actually be useful.

I tried having it parse orders in emails and create a CSV of expenses, and that went pretty badly. I'm not sure why. The CSV was invalid and full of bunk entries by the end, almost every time. It missed a lot of expenses. It would parse out only 5 or 6 items of 7, for example. Opus and Sonnet do spectacular jobs on tasks like this, and do cool things like create lists of emails with orders then systematically ensure each line item within each email is accounted for, even without prompting to do so. It's an entirely different category of performance.

Automation is something I'd like to dabble in next, but all I can think of it being useful for is mapping commands (probably from voice) to tool calls, and the reality is I'd rather tap a button on my phone. My family might like being able to use voice commands, though. Otherwise, having it parse logs to determine how to act based on thresholds or something would also be far better implemented with simple algorithms. It's hard to find truly useful and clear fits for LLMs

Re: Can I run AI locally?

#255

Post by wilkystyle »

the_pwner224 wrote: post #178
wilkystyle wrote: post #168

Earlier quoted context omitted.

What operating system are you using? I was looking at this exact machine as a potential next upgrade.

Arch with KDE, it works perfectly out of the box. I configured/disabled RGB lighting in Windows before wiping and the settings carried over to Linux. On Arch, install & enable power-profiles-daemon and you can switch between quiet/balanced/performance fan & TDP profiles. It uses the same profiles & fan curves as the options in Asus's Windows software. KDE has native integration for this in the GUI in the battery menu…

Really appreciate this response! Glad to hear you are running Arch and liking it.

I've been a long-time Apple user (and long-time user of Linux for work + part-time for personal), but have been trying out Arch and hyprland on my decade+ old ThinkPad and have been surprised at how enjoyable the experience is. I'm thinking it might just be the tipping point for leaving Apple.

Re: Can I run AI locally?

#256

Post by dsr_ »

scronkfinkle wrote: post #248
dsr_ wrote: post #213

Earlier quoted context omitted.

[flagged]

Do you also require computers to grow legs when they "run"? "Thinking" is just a term to describe a process in generative AI where you generate additional tokens in a manner similar to thinking a problem through. It's kind of a tired point to argue against the verb since it's meaning is well understood at this point

I am a professional in the information technology field, which is to say a pedantic extremist who believes that words have meanings derived from consensus, and when people alter the meanings, they alter what they believe.

Using "thinking", "feeling", "alive", or otherwise referring to a current generation LLM as a creature is a mistake which encourages being wrong in further thinking about them.

Re: Can I run AI locally?

#257

Post by scottmf »

threecheese wrote: post #220
sdrinf wrote: post #151

Earlier quoted context omitted.

Just want to echo the recommendation for qwen3.5:9b. This is a smol, thinking, agentic tool-using, text-image multimodal creature, with very good internal chains of thought. CoT can be sometimes excessive, but it leads to very stable decision-making process, even across very large contexts -something we haven't seen models of this size before. What's also new here, is VRAM-context size trade-off: for 25% of it's atte…

You can really see the limitations of qwen3.5:9b in reasoning traces- it’s fascinating. When a question “goes bad”, sometimes the thinking tokens are WILD - it’s like watching the Poirot after a head injury. Example: “what is the air speed velocity of a swallow?” - qwen knew it was a Monty Python gag, but couldnt and didnt figure out which one.

As a person who also knows there's a connection between that phrase and Monty Python and not much more information beyond that, I'm not sure how to feel.

Re: Can I run AI locally?

#258

Post by casey2 »

Something notable is that Qwen3.5:0.8B does better on benchmarks than GPT3.5. Runs much faster on local hardware than GPT3.5 on release. However Qwen3.5:0.8B dumber and slower than GPT3.5. It's dumber: it can do 3*3, but if asked to explain it in terms of the definition (i.e. 3+3+3=9) it fails. It's slower: It's a thinking model so your 900T/S are mainly spent "thinking" most of the time it will just repeat until it hangs.

It pretty obvious that this reasoning scaling is a mirage, parameters are all you need. Everything else is mostly just wasting time while hardware get better.

Re: Can I run AI locally?

#259

Post by colechristensen »

dsr_ wrote: post #256
scronkfinkle wrote: post #248

Earlier quoted context omitted.

Do you also require computers to grow legs when they "run"? "Thinking" is just a term to describe a process in generative AI where you generate additional tokens in a manner similar to thinking a problem through. It's kind of a tired point to argue against the verb since it's meaning is well understood at this point

I am a professional in the information technology field, which is to say a pedantic extremist who believes that words have meanings derived from consensus, and when people alter the meanings, they alter what they believe. Using "thinking", "feeling", "alive", or otherwise referring to a current generation LLM as a creature is a mistake which encourages being wrong in further thinking about them.

I'd suggest spending more time studying words to relive your extremism. The meanings of words move incredibly quickly and a tremendous number of words have little to no relation to previous meanings.

Words such as nice, terrific, awful, manufacture, naughty, decimate, artificial, bully... and on and on.

Re: Can I run AI locally?

#260

Post by 998244353 »

dsr_ wrote: post #254
fragmede wrote: post #242

Earlier quoted context omitted.

What do you imagine the psychiatrist will do? That's an incredibly dismissive take.

Accept it in the spirit it was meant: if you have mental illnesses like this, you need treatment.

Ok but no one here actually implied that they think like this.
Post reply on HN