481–490 of 651 posts
I’ve been playing around with qwen3.6-35b-a3b and managed to boost it significantly by leveraging my own custom harness [0]. It is quite astonishing to see how far local models have progressed, and I think that if you enjoy tinkering a bit, you can save a good bit of money (if you happen to have the hardware lying around anyways). Overall it’s still hard to beat the the cost/convenience combination of a cloud based m…
Earlier quoted context omitted.
This is basically my experience as well. I have a moderately recent but high spec desktop (Radeon 6900 XT with 16 GB VRAM, Ryzen 9 7900X 12-core, 64 GB system RAM), and I tried out some recommended models with ollama a month or two ago. Anything not geared specifically towards coding seemed to struggled with actually making tool calls instead of just stating the actions they would take without making them (and trying…
> The best "free" experience I've found is using OpenCode with Big Pickle. I have absolutely zero interest in free. I honestly don't think I'm even remotely in the same demographic as people using free tiers / models. I want to pay. I don't want my data used for training. I want it to be open. I want it to be consistently up (more than Claude!). I want it to be fast. I don't want it to be subsidized as that's just an…
Earlier quoted context omitted.
You need to switch out the prompts and work with it differently. I posted this yesterday https://github.com/day50-dev/petsitter I use it with https://github.com/day50-dev/simple-llm-cli And modify the "tricks" until my evals get to good numbers. It's a model by model basis. This is what the larger firms are doing - they have custom prompts per model
Petsitter's default tricks doesn't seem to do much for Qwen3.6, right? JSON mode could be useful I suppose, but that's not really going to make it better at writing code. Do you have any other example tricks? I'm having a hard time understanding how I would apply them.
I haven't include more sophisticated ones because they are complicated and I wanted to avoid the friction
Can anybody recommend sub $10k hardware that can run the models mentioned in TFA at something faster than a snails-pace?
Also the R9700 rocm is 32gb, 1350, available now. It's like 1/3 the price of what 5090s go for and you can get the slimmer models for that price so you can pack more in.
If I had to build right this second I'd do small form factor strix halo with a Radeon card.
You can get all those parts in like 3 days, msrp, no hassles. the only thing you're paying out the nose for is the ran
Good news is mobo manufacturers are adding more slots so you don't have to get robbed paying for 32 or 64gb modules
Earlier quoted context omitted.
what kind of hardware do you need in order to run qwen3.6-27b
Depends on which variant you pull down, but a single 5090 GPU (I know these are insanely expensive, but for context) could run either the Q8 or Q4_K_M version. It will not fit the 52GB version (BF16) on the other hand. So any modern Mac with a Pro or better processor and more than 52GB of RAM (don't forget VRAM for context window also matters!) would suffice, as someone else noted, probably a 128GB model would do the…
Earlier quoted context omitted.
what kind of hardware do you need in order to run qwen3.6-27b
I recommend MacBook M5 Max with 128 GB of RAM to run it comfortably and fast. If you have something like a regular M4, go with qwen3.6-35b-a3d - the Mixture of Expert architecture makes it run 2-3x faster than the 27b version.
Earlier quoted context omitted.
The opposite of that has been happening for 20 years now with cloud compute. It won't happen with AI models either. It's almost ingrained in the American business model now. Outsource everything. Nobody wants to manage a room full of servers when they can spend 2-3x as much and outsource that headache along with the responsibility for it. Same will happen with AI. Whether that means paying Anthropic that premium or p…
> The opposite of that has been happening for 20 years now with cloud compute. It won't happen with AI models either. AI is different. Cloud computing genuinely is cheaper on average. It's better than paying for cisco servers, and at scale, it's cheaper than managed platforms (ala Heroku), and it's a coin toss for when you're in the middle ground and constantly approaching the point of rebuilding poor-man versions of…
Earlier quoted context omitted.
> The best "free" experience I've found is using OpenCode with Big Pickle. I have absolutely zero interest in free. I honestly don't think I'm even remotely in the same demographic as people using free tiers / models. I want to pay. I don't want my data used for training. I want it to be open. I want it to be consistently up (more than Claude!). I want it to be fast. I don't want it to be subsidized as that's just an…
If you think your data isn’t being hoovered up I’d like to point out that every model is possible due to federal crimes committed to obtain the information they were trained on. Regardless of how much you are paying, your data is worth another petty civil infraction.
Earlier quoted context omitted.
Hard to guarantee it's private if you don't keep it local... I don't have a lot of trust for companies in this space.
I trust AWS in this space. I'm 100% sure that they will be precisely honoring the terms of service for Bedrock (I've never looked to see whether they claim to train on your data though).
Bedrock in fact does not train on your data. It was a big deal when it was announced that they share data with Anthropic for Fable, but even then it was gated away where you’d have to explicitly allow it.