Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can I run AI locally?

canirun.ai

Post reply on HN
Print view View original

241–250 of 382 posts

Re: Can I run AI locally?

#241

Post by peddling-brink »

dsr_ wrote: post #213
sdrinf wrote: post #151

Earlier quoted context omitted.

Just want to echo the recommendation for qwen3.5:9b. This is a smol, thinking, agentic tool-using, text-image multimodal creature, with very good internal chains of thought. CoT can be sometimes excessive, but it leads to very stable decision-making process, even across very large contexts -something we haven't seen models of this size before. What's also new here, is VRAM-context size trade-off: for 25% of it's atte…

[flagged]

Rebooting a machine running an LLM isn’t noticed by the LLM.

Would you feel comfortable digitally torturing it? Giving it a persona and telling it terrible things? Acts of violence against its persona?

I’m not confident it’s not “feeling” in a way.

Yes its circuitry is ones and zeros, we understand the mechanics. But at some point, there’s mechanics and meat circuitry behind our thoughts and feelings too.

It is hubris to confidently state that this is not a form of consciousness.

Re: Can I run AI locally?

#242

Post by fragmede »

dsr_ wrote: post #213
sdrinf wrote: post #151

Earlier quoted context omitted.

Just want to echo the recommendation for qwen3.5:9b. This is a smol, thinking, agentic tool-using, text-image multimodal creature, with very good internal chains of thought. CoT can be sometimes excessive, but it leads to very stable decision-making process, even across very large contexts -something we haven't seen models of this size before. What's also new here, is VRAM-context size trade-off: for 25% of it's atte…

[flagged]

What do you imagine the psychiatrist will do? That's an incredibly dismissive take.

Re: Can I run AI locally?

#243

Post by RagnarD »

I have an RTX 6000 Pro Max-Q, which has 96GB VRAM. It identified the hardware correctly but incorrectly thought it had 4GB, at least if I interpret the RAM dropdown correctly.

Then it shows the full resolution models, which are completely unnecessary to run quality inference. Quantized models are routine for local inference and it should realize that.

Needs work.

Re: Can I run AI locally?

#245

Post by hatthew »

mongrelion wrote: post #194
mkagenius wrote: post #80

Literally made the same app, 2 weeks back - https://news.ycombinator.com/item?id=47171499

What front-end framework did you use? I find the UI so visually appealing

FWIW, while I find it appealing, I also strongly associate it with "vibe coded webapp of dubious quality," so personally I'm not gonna try to replicate it myself.

Re: Can I run AI locally?

#246

Post by shrinks99 »

Someone1234 wrote: post #134
deanc wrote: post #13

Earlier quoted context omitted.

Yes. But llmfit is far more useful as it detects your system resources.

I feel like they both solve different issues well: - If you already HAVE a computer and are looking for models: LLMFit - If you are looking to BUY a computer/hardware, and want to compare/contrast for local LLM usage: This You cannot exactly run LLMFit on hardware you don't have.

Yes, but you can get LLMFit to recommend hardware requirements with `llmfit plan --context `.

Re: Can I run AI locally?

#248

Post by scronkfinkle »

dsr_ wrote: post #213
sdrinf wrote: post #151

Earlier quoted context omitted.

Just want to echo the recommendation for qwen3.5:9b. This is a smol, thinking, agentic tool-using, text-image multimodal creature, with very good internal chains of thought. CoT can be sometimes excessive, but it leads to very stable decision-making process, even across very large contexts -something we haven't seen models of this size before. What's also new here, is VRAM-context size trade-off: for 25% of it's atte…

[flagged]

Do you also require computers to grow legs when they "run"?

"Thinking" is just a term to describe a process in generative AI where you generate additional tokens in a manner similar to thinking a problem through. It's kind of a tired point to argue against the verb since it's meaning is well understood at this point

Re: Can I run AI locally?

#249

Post by dxxvi »

Not sure if there's anybody like me. I use AI for only 2 purposes: to replace Google Search to learn something and to generate images. I wonder where there are not many models that do only 1 thing and do it well. For example, there's this one https://huggingface.co/Fortytwo-Network/Strand-Rust-Coder-14... for Rust coding. I haven't used it yet, so don't know how it's compared to the free models that Kilo Code provides.

Re: Can I run AI locally?

#250

Post by naasking »

philipkglass wrote: post #182
mandeepj wrote: post #167

Earlier quoted context omitted.

> I formerly used that cost hundreds or thousands of dollars for a license Azure Doc Intelligence charges $1.50 for 1000 pages. Was that an annual/recurring license? Would you mind sharing your OCR model? I'm using Azure for now, as I want to focus on building the functionality first, but would later opt for a local model.

I took a long break from document processing after working on it heavily 20 years ago. The tools I used before were ABBYY FineReader and PrimeOCR. I haven't tried any of the commercial cloud based solutions. I'm currently using GLM-OCR, Chandra OCR, and Apple's LiveText in conjunction with each other (plus custom code for glue functionality and downstream processing). Try just GLM-OCR if you want to get started quick…

How does GLM-OCR compare to Qwen 3 VL? I've had good experiences with Qwen for these purposes.
Post reply on HN