231–240 of 620 posts
Models that you can run at home (Like Qwen 35B) aren't remotely close to Opus or GPT 5.5. Not even close. The only open models that are in that neighbor are around 1T params, so forget about running at home. It's kind of like driving a shitbox. It can often drive you from A to B, and some people will try to convince you it's fine. It's not. There's no logical reason other than absolutely requiring the privacy, doing…
Do you recommend Ollama or bare llama.cpp?
Yes. I use Owen on my MacBook m1 (16gb) daily, running inside Ollama. Works well. Is not particularly fast, and I need to create a custom imagem that sets the temperature of the model to zero starting, so I don't get over creative with its bullshit, but it works reasonable week.
The secret to actually good agentic outputs even with small models? Llamacpp has support for this little known sampler called "top-n sigma". You should use that, set it to 1 and set temperature to literally whatever you want (it could be infinity) and your model will just magically work to your maximum context window. That's because long context generation is a sampling problem.
Earlier quoted context omitted.
i have a 128gb m4 max macbook pro i've been wanting to tinker with this stuff but genuinely never find the time. any mac users in here running similar to the above that can share their experience? i always see great debates with local stuff but the space is constantly moving goalposts and all the vernacular is pretty unfamiliar to me. i'd love to understand what people with objective experience feel they've traded aw…
If you have a 128GB Mac you really ought to try out: https://github.com/antirez/ds4 by the creator of redis. This is probably as close to it gets to state-of-the-art local LLM + agentic coding.
For personal use, yes. I replaced a $100/m subscription to claude in favor of running pi harness pointed at unsloth studio, using both qwen (unsloth/Qwen3.6-35B-A3B-MTP-GGUF) and gemma (unsloth/gemma-4-26B-A4B-it-GGUF) models, depending on my mood. I have a machine I built about 5 years ago with dual RTX3090s in it (I was going to build a new gaming machine anyways, and the llama release had just dropped so I tacked…
I have! I care about data privacy and LLMs being free. I'm using the Pi coding harness but containerized and sandboxed, to make sure it's running completely offline. On my Mac Studio with 128GB RAM (or MacBook with 36GB RAM) I'm using Qwen3.6 35b, with only 3b active parameters so that it runs really fast. I've done a complete redesign for my website's homepage and blog with Django + Wagtail. The latter is interestin…
Sounds super cool, don't get me wrong, but I suppose for most people the bar is higher than HTML/CSS.
We truly live in the dumbest timeline.
Models that you can run at home (Like Qwen 35B) aren't remotely close to Opus or GPT 5.5. Not even close. The only open models that are in that neighbor are around 1T params, so forget about running at home. It's kind of like driving a shitbox. It can often drive you from A to B, and some people will try to convince you it's fine. It's not. There's no logical reason other than absolutely requiring the privacy, doing…
I don't need a Ferrari to get to work
Earlier quoted context omitted.
2x RTX3090 are around $4400. Without any electricity costs or other parts, that's 3.6 years of $100/m claude.
Yes, today is not a great time to purchase hardware. When I bought, I paid $850 a piece. And I needed one anyways for the gaming I was going to do. My guess is the next good time to buy is going to be 24-36 months from now, depending on how the AI bubble goes. --- I'll add to this, I personally don't like Apple hardware (not so much related to the hardware as their company philosophy) but their machines with unified…
Yes, running a local model on a natural wetware substrate here. Recommended setup: plenty of nutrients, some caffeine and a quiet environment. Performance - not currently measured in tokens: roughly average.
I have been running this stack since well before Claude Code became popular. It works OK but I've found it to be very slow; and despite having a big context window, it seems to lose track of what it's working on and goes down a rabbit hole (or just wastes tokens trying to use the web browser) for hours and is hard to get back on track. I even tried spinning up two sub-agents but even after years of trying to prompt t…