Earlier quoted context omitted.
The cost per token served has been falling steadily over the past few years across basically all of the providers. OpenAI dropped the price they charged for o3 to 1/5th of what it was in June last year thanks to "engineers optimizing inferencing", and plenty of other providers have found cost savings too. Turns out there was a lot of low-hanging fruit in terms of inference optimization that hadn't been plucked yet. >…
> "engineers optimizing inferencing" are we sure this is not a fancy way of saying quantization?
The same is happening in AI research now.