Viewing profile — dlewis1788

Looks like more than KV is having an issue. Just tried to load dash.cloudflare.com and no bueno.

100% - probably why vLLM is now the default back-end in Dynamo.

100% valid - Nvidia is trying to address that now with cuTile and the new Python front-end for CUTLASS.

CUDA is an entire ecosystem - not a single programming language extension (C++) or a single library, but a collection of libraries & tools for specific use cases and optimizations …

Just curious what your issues with Triton were. We've done OK with it using it to serve LLM models w/ a classifier head via HF Transformers pipeline & Flash Attention 2, as well as…

For training, yes, but no indications on inference workloads. Apple has said they would use their own silicon for inference in the cloud.

Someone commented below that with enough batchnorm/layernorm/etc. and/or gradient clipping you can manage it, but BF16 just makes life easier if you can live without some precision…

I didn't even know about Apple's AMX instructions until I clicked on your link. Very interesting - thanks!

My understanding is for certain types of networks BF16 will train better than FP16, given the additional protection against exploding gradients and loss functions with the extended…

Confirmed Apple M1 lacks bfloat16 support completely - M1: hw.optional.arm.FEAT_BF16: 0 vs M2: hw.optional.arm.FEAT_BF16: 1

Somehow missed this from WWDC23, but it looks like Sonoma will add support for bfloat16 with Metal, and there's an active PR to add support with the PyTorch MPS back-end (PR #99272…