Their 7b model performs much better than the 13b model I have running locally with alpaca.cpp. I think this is because (looking at their code) they stick with the float16 parameters while alpaca.cpp is using the quantizized 4bit parameters. The difference is quite noticeable.
> float16 parameters vs 4bit parameters What does that mean?
Models are basically giant matrices of decimals in the range 0-1 and they represent the strengths of the paths between the neurons.
Greater precision (float 16) will give better predictions. You lose accuracy in predictions when you reduce the precision (float 4) but it makes the model much easier to run on lower spec hardware, as it requires less RAM to store lower precision floats.
https://en.m.wikipedia.org/wiki/Half-precision_floating-poin...