> you pass the --n-gpu-layers 35 flag (or whatever value is appropriate) to enable GPU This is a bit like specifying how large your strings will be to a C program. That was maybe accepted in the old days, but not anymore really.
That's not the limitation introduced in Llamafile. It's actually a feature of all gguf models. If not specified, GPU is not used at all. Optionally, you can offload some work to the GPU. This allows to run 7b models (zephyr, mistral, openhermes) on regular PCs, it just takes a bit more time to generate the response. What other API would you suggest?
Assuming increasing vram leads to an appreciable improvement in model speed, it should default to using all but 10% of the vram of the largest GPU, or all but 1GB, whichever is less.
If I've got 8GB of vram, the software should figure out the right number of layers to offload and a sensible context size, to not exceed 7GB of vram.
(Although I realise the authors are just doing what llama.cpp does, so they didn't design it the way it is)