Low responsiveness of ML models to critical or deteriorating health conditions

PAC Learning is about compression, KF/EKF is more like Taylor expansion.

The specific types of PAC Learning that this paper covers has problems with a simplicity bias, and fairly low sensitivity.

While based on UHATs, this paper may provide some insights.

https://arxiv.org/abs/2502.02393

Obviously LLM and LRMs are the most studied, but even the recent posts on here from anthropic show that without a few high probability entries in the k-top results, confabulations are difficult for transformers.

Obviously there are PAC Learning methods that target anomaly detection, but they are very different than even EKF + Mc

You will note in this paper that even highly weighted features exhibited low sensitivity.

While the industry may find some pathological cases that make the approach usable, autograd and the need for parallelism make the application of this papers methods to tiny variations to multivariate problem ambitious.

They also only trained on medical data. Part of the reason the foundation models do so well is that they encode verifiers from a huge corpus that invalidates the traditional bias variance tradeoffs from the early 90's papers.

But they are still selecting from the needles and don't have access to the hay in the haystack.

The following paper is really not related except it shows how compression exacerbates that problem.

https://arxiv.org/abs/2205.06977

Chaitin's constant encoding the Halting problem, and that it is normal and uncomputable is the extreme top end of computability, but relates to the compression idea.

EKFs have access to the computable reals, and while non-linear, KF and EKFs can be thought of linearization of the approximations as a lens.

If the diagnostic indicators were both ergodic and Markovian, this paper's approach would probably be fairly reliable.

But these efforts are really about finding a many to one reduction that works.

I am skeptical about it in this case for PAC ML, but perhaps they will find a pathological case.

But the tradeoffs between statistical learning and expansive methods are quite different.

Obviously hype cycles drive efforts, I encourage you to look at this years AAAI conference report and see that you are not alone with the frustration on the single minded approach.

IMHO this paper is a net positive, showing that we are moving from a broad exploration to targeted applications.

But that is just my opinion.

As an aside, I'm trying to train transformers for some classification tasks on audio data. The models are "small" (like 1M-15M params at most) and I find they are very finicky to train. Below 1M parameters I find them hard to train at all. I have thrown all sorts of learning rate schedules at them and the best I can get is the network learns for a bit and then plateaus, after which I can't do anything to get them out of that minima. Training an LSTM/GRU on the same data gives me a much better loss value.

I couldn't find many papers on training transformers at that scale. The only one I was able to find was MS's TinyStories [0], but that paper didn't delve much into how they trained the models and whether they trained from scratch or distilled from a larger model.

At those scales, I find LSTMs and CNNs are a lot more stable. The few online threads I've found comparing LSTMs and Transformers had the same thing to say - Transformers need a lot more data and model size to achieve parity and exceed LSTMs/GRUs/CNNs, maybe because the inductive bias provided is hard to beat at those scales. Others can comment on what they've seen.

[0] - https://arxiv.org/abs/2305.07759

I remember getting cornered by somebody in a statistics class and interrogated about whether I thought neural networks were statistical techniques. In that situation I’ll only answer yes, they are statistical techniques. As far as I can tell, a big chunk of what we do with machine learning is create complicated models with a large number of parameters. We’re not creating something other than statistical models. We just draw a kind of cultural line between traditional statistics and machine learning techniques.

Now that I think about it, maybe if you asked me about the line between traditional statistics and machine learning, I would say that in traditional statistics, you can understand the parameters.