I can't wait for Qwen 27B on an ASIC
Qwen 3.8 27B available on Cerebras at 1500 tokens/s
221–230 of 240 posts
Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s
#222I was wondering whether this was any good for programming, but it is too fast for its own good. There is a limit of 450,000 tokens per minute. I hit this limit in about 90 seconds and burned through $1.10 while doing so. This is because cached tokens count towards the token limit. For comparison, I ran the same task with DeepSeek-V4-Flash, which finished in 172 seconds and cost $0.024 with a final context window size…
Qwen 3.8 on xhigh defaulr needs 128k context minimum or you'll spend most of your time compacting context.
Also make sure you use the instruct temperatures/etc for implementation.
Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s
#223I can't wait for Qwen 27B on an ASIC
I wonder how much it would cost?
It's something that's been bugging me for a while: once we reach a "good enough" small model, and qwen 3.8 27b is already getting damn close to it, does it make sense to just bake weights and everything directly into an ASIC, and use that for highly optimized inference? AFAIK only groq and cerebras are moving in that direction, and only to be providers themselves... It would be a dream to buy one for <$1k.
Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s
#224Tried it limits are shit
Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s
#225Earlier quoted context omitted.
Nah, Taalas was putting the weights into silicon as a mask ROM. Their demo chip was hardwired to serve Llama 3.1 8B, and could never be updated. New models, even new versions without any architectural/size changes meant new tape outs. But in exchange, you get insane speed and great energy efficiency. I could see it being a great approach for basic "good enough" models. They may have had a little flexibility by suppor…
To qualify "insane speed" for anyone unaware, think a 10x improvement over even Cerebras. On the order of ~15,000 tokens/s. Not saying their approach scales well enough to keep pace with the various frontiers, but using their demo alone feels like a paradigm shift.
This seems like it would be an altogether different experience than the common experience of using an LLM, which is characterized by the person spending a lot of time waiting on the machine.
Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s
#226I can't wait for Qwen 27B on an ASIC
I wonder how much it would cost? It's something that's been bugging me for a while: once we reach a "good enough" small model, and qwen 3.8 27b is already getting damn close to it, does it make sense to just bake weights and everything directly into an ASIC, and use that for highly optimized inference? AFAIK only groq and cerebras are moving in that direction, and only to be providers themselves... It would be a drea…
This is exactly what Taalas has already done, and the reason they got quickly acquired by AMD. Their chip runs Llama 3.1 8B at 17k tok/s, and even if llama 3.1 is dated, I can think of many problems I could use it for, especially at those speeds. They're certain to be working on a newer set of weights by now.
Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s
#227Earlier quoted context omitted.
Yeah, their public service isn't a serious/competitive offering. They don't have the capacity to serve all the customers who might want to use them at that speed. The public service exists so they get some users on OpenRouter, and that shows them as #1 on speed, which proves their tech is very fast, which gets them billions in hardware sales/licensing. If you have big enough pockets they can probably dedicate capacit…
GPUs can't reach these speeds. You could build a supercomputing cluster and still not reach these speeds.
Celeris reaches ~50% of the speed on commodity hardware. celeris-magnus-1 is based on qwen3.8-27b. Maybe we will get there without custom silicon!
Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s
#228Yep, dumber than Deepseek4 at 10x the price. Cool tech demo though.
Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s
#229[deleted]
Re: Qwen 3.8 27B available on Cerebras at 1500 tokens/s
#230Earlier quoted context omitted.
To qualify "insane speed" for anyone unaware, think a 10x improvement over even Cerebras. On the order of ~15,000 tokens/s. Not saying their approach scales well enough to keep pace with the various frontiers, but using their demo alone feels like a paradigm shift.
This seems like it would be an altogether different experience than the common experience of using an LLM, which is characterized by the person spending a lot of time waiting on the machine.
Yeah, it'd be a lot easier to maintain flow, less need to work on more than one session at once, etc. And then tool calls would be the limiting factor, especially network access. I hope AMD keeps the project moving forward post acquisition.