Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.8 27B is excellent, but it defaults to overthinking things

simonwillison.net

Post reply on HN
Print view View original

121–130 of 411 posts

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#121

Post by mattmaroon »

ipdashc wrote: post #114
refactor_master wrote: post #33

Earlier quoted context omitted.

Unlike cloud infra in general which offers things like automatic backups, regional redundancy, and effectively unlimited scalability, it seems like the value proposition of cloud LLM gets ever shakier. * Many businesses don't need frontier level intelligence anyway. * It's completely stateless. If your local LLM machine catches fire? Nothing was lost. Buy another.

I mean, I might be missing something, but isn't part of the idea with cloud infra that you can scale down as well? Large orgs with significant demand might go out and buy local LLM hardware, but most businesses probably don't want to bother dropping $2k on a box with a beefy GPU and would rather just pay the lowest subscription tier so their employees can occasionally make queries. Plus, you know, the whole economies…

All white collar work will be done by AI soon, there won’t be scaling down, just scaling up. I do food trucks and festivals and I’ve got AI doing so much of my non-meatspace work now that I’m buying $10-$20 a day in tokens. At that rate, hardware starts to look cheap, and I have less of a use case for it than most white collar workers.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#122

Post by pdyc »

madduci wrote: post #87
chvid wrote: post #66

“The fact that a 17GB file can do all of this stuff on my home machines is a miracle. Once again, I’m delighted and amazed at how much progress local models have made this year.” I think that should be the blinking headline - this shows what can be done with consumer hardware.

Tried yesterday on my own laptop (a UltraCore 7 255H without dedicated GPU,with 32 GB RAM), it wasn't even starting thinking, even on a small context window (65k)

i have same 255h and i was able to run it with low token speed 6-8tg/s with approx similar context window 60k

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#124

Post by pyrale »

madduci wrote: post #105
aphroz wrote: post #91

Earlier quoted context omitted.

I think not much can run without a dedicated GPU

Till now I was using successfully Qwen 3.5 and Gemma 4 at a reasonable speed

There is no way you would run a dense 27b model on that spec. I ran 3.6 27b on a 64gb ram, 24 gb vram, and it felt like the lower limit for this model with a decent context window.

If you want a better experience, maybe wait for either a moe model (like 3.6 35b A3) or a model with less parameters (like 9b). Qwen has been releasing those in the past, so maybe we’ll have them for 3.8 too.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#126

Post by ipdashc »

mattmaroon wrote: post #121
ipdashc wrote: post #114

Earlier quoted context omitted.

I mean, I might be missing something, but isn't part of the idea with cloud infra that you can scale down as well? Large orgs with significant demand might go out and buy local LLM hardware, but most businesses probably don't want to bother dropping $2k on a box with a beefy GPU and would rather just pay the lowest subscription tier so their employees can occasionally make queries. Plus, you know, the whole economies…

All white collar work will be done by AI soon, there won’t be scaling down, just scaling up. I do food trucks and festivals and I’ve got AI doing so much of my non-meatspace work now that I’m buying $10-$20 a day in tokens. At that rate, hardware starts to look cheap, and I have less of a use case for it than most white collar workers.

Genuine question, what are you spending that on? The $20/month ChatGPT/Codex subscription has largely been enough for me as an IT worker.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#127

Post by mattmaroon »

ipdashc wrote: post #126
mattmaroon wrote: post #121

Earlier quoted context omitted.

All white collar work will be done by AI soon, there won’t be scaling down, just scaling up. I do food trucks and festivals and I’ve got AI doing so much of my non-meatspace work now that I’m buying $10-$20 a day in tokens. At that rate, hardware starts to look cheap, and I have less of a use case for it than most white collar workers.

Genuine question, what are you spending that on? The $20/month ChatGPT/Codex subscription has largely been enough for me as an IT worker.

There. I have the business plan with two seats and I use them both and blow through it pretty fast. I think it’s because much of what I have it do involves using a browser. For instance I have it pull various permits from cities and there’s no API for that.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#128

Post by redrove »

NhanH wrote: post #103
refactor_master wrote: post #33

Earlier quoted context omitted.

Unlike cloud infra in general which offers things like automatic backups, regional redundancy, and effectively unlimited scalability, it seems like the value proposition of cloud LLM gets ever shakier. * Many businesses don't need frontier level intelligence anyway. * It's completely stateless. If your local LLM machine catches fire? Nothing was lost. Buy another.

The whole cloud story lies on two aspects: - Hyperscaling “we are going to serve billions of people in our applications”, which is becoming increasing unlikely as regional tech companies become more dominant than than the global one (this one is as much about geopolitics as technology) - Operations is hard, in which case non-frontier models should be increasingly capable. Devops for small-ish deployment is one of the…

I don’t disagree but saying “cloud will change in the next 8-20y-ish” is a bit of a non-argument, you’re not really stating any thesis to speak of; Change is a given over that time frame.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#129

Post by redrove »

mattmaroon wrote: post #121
ipdashc wrote: post #114

Earlier quoted context omitted.

I mean, I might be missing something, but isn't part of the idea with cloud infra that you can scale down as well? Large orgs with significant demand might go out and buy local LLM hardware, but most businesses probably don't want to bother dropping $2k on a box with a beefy GPU and would rather just pay the lowest subscription tier so their employees can occasionally make queries. Plus, you know, the whole economies…

All white collar work will be done by AI soon, there won’t be scaling down, just scaling up. I do food trucks and festivals and I’ve got AI doing so much of my non-meatspace work now that I’m buying $10-$20 a day in tokens. At that rate, hardware starts to look cheap, and I have less of a use case for it than most white collar workers.

I would also be interested in what you use AI for. Is it marketing? bureaucracy?

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#130

Post by SahAssar »

ipdashc wrote: post #114
refactor_master wrote: post #33

Earlier quoted context omitted.

Unlike cloud infra in general which offers things like automatic backups, regional redundancy, and effectively unlimited scalability, it seems like the value proposition of cloud LLM gets ever shakier. * Many businesses don't need frontier level intelligence anyway. * It's completely stateless. If your local LLM machine catches fire? Nothing was lost. Buy another.

I mean, I might be missing something, but isn't part of the idea with cloud infra that you can scale down as well? Large orgs with significant demand might go out and buy local LLM hardware, but most businesses probably don't want to bother dropping $2k on a box with a beefy GPU and would rather just pay the lowest subscription tier so their employees can occasionally make queries. Plus, you know, the whole economies…

I'm pretty sure a small-to-medium org has plenty of things that could be queued/scheduled to run when there is downtime. It requires some planning and thought though and I don't think most orgs are there yet.
Post reply on HN