Viewing profile — zackangelo

Blackwell supports nvfp4 natively.

what was the concurrency limitation? that node should be able to support a lot more

I don't believe safetensors has a native int4 dtype, so they packed 4 int4s into a bf16 in this checkpoint.

If you're in SF and weighing this decision, it's easy to get tilted in the buy direction because the rental stock is so horrific. Landlords have very little incentive to update pro…

With the 3.5 release, the Plus model was just a rebrand of the open weight 397B. But I suspect that will change going forward. They haven’t released the weights for 3.6 but they di…

absolutely not, take Kimi K2.6 for a spin

Isn't Kimi K2.6 natively INT4?

I don’t think this is true across Blizzard. Overwatch is the best it’s ever been.

I give them a try about twice a year. I write a lot of Rust which should be squarely in their wheelhouse. This last time I was pleasantly surprised to find they mostly fixed their …

They are but the IDE needs to be integrated with them. Qwen specifically calls out FIM (“fill in the middle”) support on the model card and you can see it getting confused and post…

17b per token. So when you’re generating a single stream of text (“decoding”) 17b parameters are active. If you’re decoding multiple streams, it will be 17b per stream (some tokens…

This uses Nvidia’s CUDA snapshot API under the hood, but you have to pair it with a host side snapshot as well. Modal uses gVisor for this, which is notoriously high overhead. Does…

You’re right I misunderstood. I’m not sure if it would be of much utility because this would presumably be for tensor parallel workloads. In that case you want the ranks in your cl…

Sparks are built for this and actually have Connect-X 7 NICs built in! You just need to get the SFPs for them. This means you can natively cluster them at 200Gbps.

No you use tensor parallelism in both cases. The way it typically works in an attention block is: smaller portions of the Q, K and V linear layers are assigned to each node and are…

What 1T parameter base model have you seen from any of those labs?

Wouldn't you be able to test nccl if you had 2 of these?

Just a bit of feedback: > Instead of one brittle giant, we orchestrate a Mixture of Experts… “mixture of experts” is a specific term of art that describes an architectural detail o…

Because it depends on how much better “best” is. If it’s only incrementally better than open source models that have other advantages, why would you bother? OpenAI’s moat will only…

Not quite a frontier model but definitely built by a frontier lab: Grok 2 was recently open sourced and I believe it uses a fairly standard MHA architecture with MoE.

I feel a bit silly for not noticing this before. Over the last year or so I've often wondered when ssh added protocol-level support for session resume. I'd open my laptop on a new …

Curious what issues you were having. The kernel should compile natively if you pass nvcc the correct arch flags, although it probably won't take advantage of any new hardware featu…

GPT-OSS will run even faster on Blackwell chips because of its hardware support for fp4. If anyone is working on training or inference in Rust, I'm currently working on adding fp8 …