Viewing profile — AlekseiSavin

yeah, tokens per second

it looks like llama.cpp has some performance issues with bf16

right now, we support AWQ but are currently working on various quantization methods in https://github.com/trymirai/lalamo

You're right, modern edge devices are powerful enough to run small models, so the real bottleneck for a forward pass is usually memory bandwidth, which defines the upper theoretica…

already) https://github.com/trymirai/uzu-swift