541–550 of 620 posts
Earlier quoted context omitted.
There is a bug in llama-cpp for qwen/gemma models, use vLLM instead
what bug and it affects what?
There are other reasons to prefer vllm to llama-cpp as well
No. I've tried all the OS models up to Qwen 480B and Kimi (the biggest models). None come even close to Claude. I do mostly scripting, devops, data processing and systems stuff (ansible playbooks, managing network devices, deploying new software for various things that involves reading docs, writing helm charts, modifying existing ones etc). All other models Gemini, Chatgpt, grok and all OS models don't come even clo…
I've also used Composer2.5 on hobby projects and it is definitely on-par with Opus 4.8 (thinking mode: medium), but much faster.
Do you think you're getting better results with Claude because your agent stack (skills, MCPs, etc.) are configured for it and not for the others?
Earlier quoted context omitted.
I think the idea that code should last decades is now questionable, if not problematic. If we can now produce code at 10x the rate, that means we can have 10x more code (probably not desirable) or we can have 10x as many revisions. Whoever inherits the code can have it rewritten to their liking and understanding. Nothing helps better in understanding a system than to rebuild it, even if just by handholding an LLM.
Only for simple problems. As the problem becomes complex you can't remember all the requirements to prompt the AI with.
I'm using 4x RTX 5070's and first-gen AMD threadripper (1950X) to run Qwen3.6 27B (MTP) Q6_K with llama.cpp and it works great as a daily driver with Pi. Around 50-60 toks/sec. I also connect a few other applications to it such as OpenWeb UI and recently set up Bifrost, an LLM gateway, to be the primary access point for the models I serve. I've tried other models such as Qwen3.6 35B A3B and I've found that 27B works…
This is interesting to me - why'd you go with the 5070 for your 4x build? At first thought, they are quite skewed toward compute (vs VRAM), which is great for gamers but not so great for running LLMs. (I run a 5070 in my desktop)
I did some math/shopping as well. To get 48GB of VRAM you can get 2x 3090s but that is $3k. A single 5090 is $4k but has 32GB, great for running models like Qwen 27B but maybe nothing else depending on your model settings. Already having 2x 5070, where each card is around $600, it made sense for me to get two more which was $1200 and the memory speeds aligned.
The best value option if you're building from scratch is go with 5060 ti (16GB VRAM). Each of those cards are $570/each on Amazon, cheaper than 4x 5070's. Only downside is memory speed is slightly slower, but you wind up with 64GB of VRAM and you can run big models and small models alongside each other comfortably.
In my setup I ran Qwen3.5 9B for fast inference on simple things and Qwen3.6 27B Q6 for coding work. But I ran into stability issues, so I use llama-swap to dynamically swap models. But with 64GB of VRAM, you wouldn't have that issue. There is overhead to loading LLMs into VRAM that isn't clear, so having extra VRAM is a helpful buffer.
Earlier quoted context omitted.
Only for simple problems. As the problem becomes complex you can't remember all the requirements to prompt the AI with.
As the problem becomes complex, you can't remember all the requirements, period.
Earlier quoted context omitted.
Microsoft have not maintained a monopoly on search, mobile, or maps, and they seem to mostly maintain their large market segments based on familiarity, not hype.
? I was speaking historically, not now
I have! I care about data privacy and LLMs being free. I'm using the Pi coding harness but containerized and sandboxed, to make sure it's running completely offline. On my Mac Studio with 128GB RAM (or MacBook with 36GB RAM) I'm using Qwen3.6 35b, with only 3b active parameters so that it runs really fast. I've done a complete redesign for my website's homepage and blog with Django + Wagtail. The latter is interestin…
I've read a bit on what the various components are. What I don't see in your comment is what you're using to run your model locally. Ollama?