I was yesterday years old when I learned that those open weight models need custom code to run. Somehow I expected inference engines are generic LLM runtimes that can execute any weight. So, to get this right. Someone trains a model. They release the weights and a reference implementation of the model architecture. Then a provider has to host this model either by running inference via the reference implementation, an…
Absolutely. The worst part is that ML being probabilistic means the models still output something comprehensible meaning you can't determine if it's correct or not without a lot of scrutiny and engineering.