Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can I run AI locally?

canirun.ai

Post reply on HN
Print view View original

291–300 of 382 posts

Re: Can I run AI locally?

#291

Post by colechristensen »

hnfong wrote: post #268
colechristensen wrote: post #262

Earlier quoted context omitted.

I'm not entirely opposed to the kind of animism that assigns a certain amount of soul, consciousness, or being to everything in a spectrum between a rock and a philosopher... but even so. Multiplying large matrices over and over is very much towards the "rock" end of that scale.

If we accept the Church-Turing thesis, a philosopher can be simulated by a simple Universal Turing machine. If one day we are able to create a philosopher from such a rudimentary machine (and a lot of tape), would you consider that very much towards the "rock" end as well?

Can a Turing machine of any sort truly indistinguishably simulate a nondeterministic system?

If a Turing machine can truly simulate a full nondeterministic system as complex as a philosopher but it would take dedicating every gram of matter in the visible universe for a trillion years to simulate one second, is this meaningfully different than saying it cannot?

I suggest the answer to both questions are no, but the second one makes the answer at worst "practically, no".

My feeling is that consciousness is a phenomenon deeply connected to quantum mechanics and thus evades simulation or recreation on Turing machines.

Re: Can I run AI locally?

#292

Post by dyauspitr »

dxxvi wrote: post #249

Not sure if there's anybody like me. I use AI for only 2 purposes: to replace Google Search to learn something and to generate images. I wonder where there are not many models that do only 1 thing and do it well. For example, there's this one https://huggingface.co/Fortytwo-Network/Strand-Rust-Coder-14... for Rust coding. I haven't used it yet, so don't know how it's compared to the free models that Kilo Code provide…

For learning and general searching I find ChatGPT to be the best.

Nano Banana Pro for anything image and video related.

Grok Imagine for pretty decent porn generation.

Re: Can I run AI locally?

#293

Post by akersten »

dsr_ wrote: post #256
scronkfinkle wrote: post #248

Earlier quoted context omitted.

Do you also require computers to grow legs when they "run"? "Thinking" is just a term to describe a process in generative AI where you generate additional tokens in a manner similar to thinking a problem through. It's kind of a tired point to argue against the verb since it's meaning is well understood at this point

I am a professional in the information technology field, which is to say a pedantic extremist who believes that words have meanings derived from consensus, and when people alter the meanings, they alter what they believe. Using "thinking", "feeling", "alive", or otherwise referring to a current generation LLM as a creature is a mistake which encourages being wrong in further thinking about them.

I think you are still missing the point. No one in this thread is making an anthropological assertion. "Thinking" here is just shorthand for Chain of Thought[0], which some models have and some models don't. This model, being a "thinking" model, has it.

[0]: https://en.wikipedia.org/wiki/Prompt_engineering#Chain-of-th...

Re: Can I run AI locally?

#294

Post by gentleman11 »

sxates wrote: post #4

Cool thing! A couple suggestions: 1. I have an M3 Ultra with 256GB of memory, but the options list only goes up to 192GB. The M3 Ultra supports up to 512GB. 2. It'd be great if I could flip this around and choose a model, and then see the performance for all the different processors. Would help making buying decisions!

ask apple to graciously allow you to install your own ram in the computer you "own"

Re: Can I run AI locally?

#296

Post by albedoa »

dsr_ wrote: post #256
scronkfinkle wrote: post #248

Earlier quoted context omitted.

Do you also require computers to grow legs when they "run"? "Thinking" is just a term to describe a process in generative AI where you generate additional tokens in a manner similar to thinking a problem through. It's kind of a tired point to argue against the verb since it's meaning is well understood at this point

I am a professional in the information technology field, which is to say a pedantic extremist who believes that words have meanings derived from consensus, and when people alter the meanings, they alter what they believe. Using "thinking", "feeling", "alive", or otherwise referring to a current generation LLM as a creature is a mistake which encourages being wrong in further thinking about them.

[flagged]

Re: Can I run AI locally?

#298

Post by ActorNightly »

sxates wrote: post #4

Cool thing! A couple suggestions: 1. I have an M3 Ultra with 256GB of memory, but the options list only goes up to 192GB. The M3 Ultra supports up to 512GB. 2. It'd be great if I could flip this around and choose a model, and then see the performance for all the different processors. Would help making buying decisions!

>. I have an M3 Ultra with 256GB of memory,

Im sorry but spending this kind of money when you could have just built yourself a dual 3090 workstation that would have been better for pretty much everything including local models is just plain stupid.

Hell, even one 3090 can now run Gemma 3 27b qat very fast.

Re: Can I run AI locally?

#299

Post by ActorNightly »

scorpioxy wrote: post #265

Besides trying to run on your own hardware, anybody have recommendations for running some decent models on one of the many "AI clouds" providers? This is for sporadic use and so maybe one of the "serverless" providers that bill by the hour or minute or similar as opposed to monthly renting GPUs. There are quite a few of them but their marketing is just confusing and full of buzz words. I've been tinkering with OpenRo…

I mean AWS bedrock fits your use case pretty much. They have a bunch of models that are serverless that you can use on a per token pricing cost.

Gemini api use also comes with a free tier.

Re: Can I run AI locally?

#300

Post by lilactown »

dataflow wrote: post #188
mark_l_watson wrote: post #78

I have spent a HUGE amount of time the last two years experimenting with local models. A few lessons learned: 1. small models like the new qwen3.5:9b can be fantastic for local tool use, information extraction, and many other embedded applications. 2. For coding tools, just use Google Antigravity and gemini-cli, or, Anthropic Claude, or... Now to be clear, I have spent perhaps 100 hours in the last year configuring l…

Thanks for sharing this, it's super helpful. I have a question if you don't mind: I want a model that I can feed, say, my entire email mailbox to, so that I can ask it questions later. (Just the text content, which I can clean and preprocess offline for its use.) Have any offline models you've dealt with seemed suitable for that sort of use case, with that volume of content?

If your inbox is as big as mine, you won’t be able to load all the text content into a prompt even with SotA cloud hosted models.

Instead you should give it tools to search over the mailbox for terms, labels, addresses, etc. so that the model can do fine grained filters based on the query, read the relevant emails it finds, then answer the question.

Post reply on HN