Alibaba is giving sleepless nights to the tech giants
To be fair, Alibaba IS a tech giant, one of the biggest in fact.
They are just giving sleepless nights to the western tech giants.
Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)
101–110 of 178 posts
Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)
#102Earlier quoted context omitted.
I can do traditional server stuff on any old computer with a big hard disk and a decent amount of RAM. That's not worth $3500-$4000. When RAMpocalypse is over and we can buy a Strix Halo for under $2000 again, the math starts mathing. It becomes a pretty great desktop computer that also happens to run AI pretty well at a pretty good price.
Yeah, but that computer can’t also do the AI stuff. And not everybody has a desktop with multiple 32GB GPUs available. I’ll admit though I’m biased because I bought my board for $1600 back before the prices went crazy.
Oh no a tough constraint that will lead to further innovation like deepseek. How terrible.
Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)
#103Finally, a reason to own a 128GB Strix Halo or GB10 device. Or a reason to consider the new Mac Studio. I have a Strix Halo and dual 32GB GPUs in my desktop, and the latter is pretty much always better for running local models because it's quite a bit faster due to higher memory bandwidth. There simply haven't been any models that are better than Qwen 27B or Gemma 31B, which run comfortably in 64GB with big context.…
how much performance (tok/s) can you expect from 128GB Strix Halo? assuming this model will be released with FP8 also can you use it for fine tuning?
You can only use up to 90gb for the GPU, so it doesn't fit
Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)
#104+1 to the long list of people hoping for Qwen3.8-27b A3B.
You probably meant Qwen3.8-35B-A3B. But judging from some of the words from their team, it seems unlikely unfortunately.
They normally release a 35b dense and an 27b moe (4B active per token)
For context 35B on my m4 runs at 10 tokens a second, 27B moe runs 50-60 tokens a second.
Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)
#105Earlier quoted context omitted.
how much performance (tok/s) can you expect from 128GB Strix Halo? assuming this model will be released with FP8 also can you use it for fine tuning?
You can only use up to 90gb for the GPU, so it doesn't fit
No. In Linux, you can ignore the BIOS (well, set it to 512MB) and set GTT to allow almost the entirety of memory for GPU. If you don't run a desktop GUI (and why would you), you don't need memory for much of anything else. Just a few GB for the OS and llama.cpp.
These are roughly the settings I use: https://github.com/kyuz0/amd-strix-halo-toolboxes#kernel-par...
Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)
#106Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)
#107Earlier quoted context omitted.
You can already run it locally its just not the same. It is still slow, a lot slower than what you are used to with claude and co. And as soon as you increase context size, your memory requirements jump. Then when it runs for 30 minutes for something claude needs 5, your device will get hot. And even a used 3090 is apparently now between 1-2k.
> It is still slow, a lot slower than what you are used to with claude and co. That really depends on the model, I run a few models locally. All at speeds comparable to or faster than Opus. In general we haven't reached the ceiling for what performance we can get out of consumer hardware. As evidence by FreeToken which hasn't even added MTP/speculative drafting support yet, which will add another boost. > Then when i…
Which models when run locally come close to Sol and Opus, from your experience? And which harness do you use?
Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)
#108Finally, a reason to own a 128GB Strix Halo or GB10 device. Or a reason to consider the new Mac Studio. I have a Strix Halo and dual 32GB GPUs in my desktop, and the latter is pretty much always better for running local models because it's quite a bit faster due to higher memory bandwidth. There simply haven't been any models that are better than Qwen 27B or Gemma 31B, which run comfortably in 64GB with big context.…
A 3060ti 8gb, released in 2020, has 448 GB/s of bandwidth compared to the Halo 256 GB/s The 3080ti is 912.4 GB/s
Enjoy Gemma 4 E2B at blistering speeds, I guess?
Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)
#109I have an m1 max 64gb macbook. Anything I can do to get 3.8 27b at more than 10 tok/s or am I relegated? 3.6 a3b is good but its not as good
Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)
#110Earlier quoted context omitted.
I built a router that lets you route between local and cloud models. Link in my profile.
Yeah, I also built my own "router" for this: if (process.env.LOCAL_MODEL { http('localhost:3000/v1/completions')... } else { http('api.openrouter.ai/v1/completions')... }
Brilliant! I'd like to invest $10M. Who should I make the check out to?