Running a 125M model on-device at that speed is impressive. How much did you have to optimize the model to get that performance on an iPhone?
For the current model I’m using Core ML, which optimizes the kernels the first time you run it. I haven’t actually spent that much time tuning performance beyond that.