So I'm not ML guru or anything but what I learned was that if you have m features on n samples you want n > m to prevent over-fitting, no? Also, with so few samples, how do you do your hyperparameter tuning and validation? I mean you could eliminate certain features in isolation but that doesn't capture dependent features. And how would you do dimensionality reduction?
Honestly I didn't prioritize hyperparameter tuning enough. I pretty much went with one of the first models I identified.
Could you elaborate on the idea of not capturing dependent features please?