Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)

modelscope.cn

Post reply on HN
Print view View original

11–20 of 178 posts

Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)

#13

Post by sscaryterry »

pwython wrote: post #11

I was already rolling around the idea of a 128GB M5 Max MBP. Now this! A 4-bit MLX quant with 128k window should fit perfectly, in the 50-70 tok/s range.

I have a 128GB M5 Max, and it sucks at this stage. 50-70 tok/s might be something...

Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)

#14

Post by ddtaylor »

I enjoy the Qwen models a lot, but building things on top of them with OpenRouter has been painful.

OpenRouter does a lot of great work and I really enjoy being able to use different models so easily. I like when a provider is phasing out an older model that still works for my needs and the price is much lower. It seems like such a good win-win.

However, the problem is that many Qwen models have almost no capacity or is so flaky you literally have to just litter your code with a blacklist/whitelist of providers. OpenRouter has some attempts to solve this, but they don't work. In fact, OpenRouter has a lot of really cool stuff that is documented, but if you read the code it's not yet implemented or isn't actually there yet, which is a shame.

I tried to get in contact with them at OpenRouter about this and I was interested in working with them in the past, but it's difficult to get in touch with the right people and they are growing very fast. I expect being acquired by Stripe will accelerate those problems in some ways. I have no doubt they will resolve all of these issues eventually and scaling that much that quickly is really hard, so kudos to them, but the road has been pretty lame and taken some wind out of my sails.

Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)

#17

Post by irthomasthomas »

ddtaylor wrote: post #14

I enjoy the Qwen models a lot, but building things on top of them with OpenRouter has been painful. OpenRouter does a lot of great work and I really enjoy being able to use different models so easily. I like when a provider is phasing out an older model that still works for my needs and the price is much lower. It seems like such a good win-win. However, the problem is that many Qwen models have almost no capacity or…

Openrouter was pretty great before prompt caching became common. Now it is extremely expensive for most individual workflows, unless you spend a lot of work customizing router preferences, and then you still get a worse cache hit rate than using the provider directly. I only keep $5-$10 in OR for occasional testing.

Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)

#18

Post by irthomasthomas »

pwython wrote: post #11

I was already rolling around the idea of a 128GB M5 Max MBP. Now this! A 4-bit MLX quant with 128k window should fit perfectly, in the 50-70 tok/s range.

IDK, prefill speed is a bigger concern for most wokflows, like agent coding, and I heard that this is quite low on macs?

Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)

#19

Post by smcleod »

irthomasthomas wrote: post #18
pwython wrote: post #11

I was already rolling around the idea of a 128GB M5 Max MBP. Now this! A 4-bit MLX quant with 128k window should fit perfectly, in the 50-70 tok/s range.

IDK, prefill speed is a bigger concern for most wokflows, like agent coding, and I heard that this is quite low on macs?

That was mainly before the M4 generation when they didn't have matmul instructions.

Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)

#20

Post by smcleod »

sscaryterry wrote: post #13
pwython wrote: post #11

I was already rolling around the idea of a 128GB M5 Max MBP. Now this! A 4-bit MLX quant with 128k window should fit perfectly, in the 50-70 tok/s range.

I have a 128GB M5 Max, and it sucks at this stage. 50-70 tok/s might be something...

50-70tk/s is what I get on my m5 max on a 5-6bit Qwen 3.8 27B?
Post reply on HN