Can I run AI locally?

Post by scorpioxy » Sat, Mar 14, 2026, 9:06 AM UTC

ActorNightly wrote: post #299

scorpioxy wrote: post #265

Besides trying to run on your own hardware, anybody have recommendations for running some decent models on one of the many "AI clouds" providers? This is for sporadic use and so maybe one of the "serverless" providers that bill by the hour or minute or similar as opposed to monthly renting GPUs. There are quite a few of them but their marketing is just confusing and full of buzz words. I've been tinkering with OpenRo…

I mean AWS bedrock fits your use case pretty much. They have a bunch of models that are serverless that you can use on a per token pricing cost. Gemini api use also comes with a free tier.

Post by Stronz » Sat, Mar 14, 2026, 9:31 AM UTC

Post by xiconfjs » Sat, Mar 14, 2026, 9:38 AM UTC

ActorNightly wrote: post #298

sxates wrote: post #4

Cool thing! A couple suggestions: 1. I have an M3 Ultra with 256GB of memory, but the options list only goes up to 192GB. The M3 Ultra supports up to 512GB. 2. It'd be great if I could flip this around and choose a model, and then see the performance for all the different processors. Would help making buying decisions!

>. I have an M3 Ultra with 256GB of memory, Im sorry but spending this kind of money when you could have just built yourself a dual 3090 workstation that would have been better for pretty much everything including local models is just plain stupid. Hell, even one 3090 can now run Gemma 3 27b qat very fast.

Post by plank » Sat, Mar 14, 2026, 9:48 AM UTC

cassianoleal wrote: post #277

threecheese wrote: post #220

Earlier quoted context omitted.

You can really see the limitations of qwen3.5:9b in reasoning traces- it’s fascinating. When a question “goes bad”, sometimes the thinking tokens are WILD - it’s like watching the Poirot after a head injury. Example: “what is the air speed velocity of a swallow?” - qwen knew it was a Monty Python gag, but couldnt and didnt figure out which one.

African or European?

Post by krelian » Sat, Mar 14, 2026, 10:09 AM UTC

dsr_ wrote: post #256

scronkfinkle wrote: post #248

Earlier quoted context omitted.

Do you also require computers to grow legs when they "run"? "Thinking" is just a term to describe a process in generative AI where you generate additional tokens in a manner similar to thinking a problem through. It's kind of a tired point to argue against the verb since it's meaning is well understood at this point

I am a professional in the information technology field, which is to say a pedantic extremist who believes that words have meanings derived from consensus, and when people alter the meanings, they alter what they believe. Using "thinking", "feeling", "alive", or otherwise referring to a current generation LLM as a creature is a mistake which encourages being wrong in further thinking about them.