Viewing profile — amanrs

This is harder than it looks. First "token-healing" doesn't work. Consider the case "app" where the most likely options are "ap|praisal" or "apple|sauce". You can't just sample all…

It edits the file based on the "plan" laid out by a smarter language model

The key misconception about many quantization methods is that lower precision = better speed. I believe GPT-Q is not much faster than bf16 from skimming the AWQ paper - https://arx…