41–50 of 196 posts
https://www.youtube.com/@AZisk
At this point, pretty much all he does is review workstations for running LLM's and other machine-learning adjacent tasks.
I'm not his target demographic, but because I'm a dev, his videos are constantly recommended to me on YouTube. He's a good presenter and his advice makes a lot of sense.
Real-world workflows? I'm all for local LLM, tinker with it all the time, but for productive coding use no local LLM approaches cloud and it's not even close. There's no magic trick or combination of pieces, it just turns out that a quarter million dollars worth of H200s is just much, much better than anything a normal person could possibly deploy at home. Give it time, we'll get there, but not anytime soon.
My development flow takes a lot of RAM (and yes I can run it minimally editing in the terminal with language servers turned off), so I wouldn't consider running the local LLM on the same computer.
> Which model(s) are you running (e.g., Ollama, LM Studio, or others) and which open-source coding assistant/integration (for example, a VS Code plugin) you’re using? Open-source coding assistant: VT Code (my own coding agent -- github.com/vinhnx/vtcode) Model: gpt-oss-120b remote hosted via Ollama cloud experimental > What laptop hardware do you have (CPU, GPU/NPU, memory, whether discrete GPU or integrated, OS) and…
Wait ollama cloud has a free tier? Sounds too good. Where's the catch? And is it private?
On an RTX 3080 Ti+Ryzen 9
I am here to hear from folks running LLM on Framework desktop (128GB). Is it usable for agentic coding ?
1. Thermal considerations are important due to throttling for thermal protection. Apple seems best at this but $$$$. The Framework (AMD) seems a reasonable compromise (you can have almost 3 for 1 Mini). Laptops will likely not perform as well. NVIDIA seems really bad at thermal/power considerations.
2. Memory model matters and AMD's APU design is an improvement. NVIDIA GPUs where designed for graphics but where better than CPUs for AI so they got used. Bespoke AI solutions will eventually dominate. That may or may not be NVIDIA in the future.
My primary interest is AI at the edge.
I love local models for some use cases. However for coding there is a big gap between the quality of models you can run at home and those you can't (at least on hardware I can afford) like GLM 4.6, Sonnet 4.5, Codex 5, Qwen Coder 408.
What makes local coding models compelling?
On a Macbook pro 64GB I use Qwen3-Coder-30B-A3B Q4 quant with llama.cpp. For VSCode I use continue.dev as it allows to set my own (short) system prompt. I get around 50token/sec generation and prompt processing 550t/s. When giving well defined small tasks, it is as good as any frontier model. I like the speed and low latency and the availability while on the plane/train or off-grid. Also decent FIM with the llama.cpp…
Here's my ollama config:
https://github.com/woile/nix-config/blob/main/hosts/aconcagu...
I'm not an AI power user. I like to code, and I like the AI to autocomplete snippets that are "logical", I don't use agents, and for that, it's good enough.
Models
gpt-oss-120b, Meta Llama 3.2, or Gemma
(just depends on what I’m doing)
Hardware
- Apple M4 Max (128 GB RAM)
paired with a GPD Win 4 running Ubuntu 24.04 over USB-C networking
Software
- Claude Code
- RA.Aid
- llama.cpp
For CUDA computing, I use an older NVIDIA RTX 2080 in an old System76 workstation.
Process
I create a good INSTRUCTIONS.md for Claude/Raid that specifies a task & production process with a task list it maintains. I use Claude Agents with an Agent Organizer that helps determine which agents to use. It creates the architecture, prd and security design, writes the code, and then lints, tests and does a code review.Models gpt-oss-120b, Meta Llama 3.2, or Gemma (just depends on what I’m doing) Hardware - Apple M4 Max (128 GB RAM) paired with a GPD Win 4 running Ubuntu 24.04 over USB-C networking Software - Claude Code - RA.Aid - llama.cpp For CUDA computing, I use an older NVIDIA RTX 2080 in an old System76 workstation. Process I create a good INSTRUCTIONS.md for Claude/Raid that specifies a task & production process with a task…