What does it really mean to properly understand p-value? What I remember is that if the p-value is less than 0.05, the research result is considered statistically significant. That's about as far as my memory goes. I know that's actually a misunderstanding, but that's how most people understand it. I'm not sure how much I need to know to say I truly understand it.
Low p-value basically means how surprising your data would be if there were actually no effect (ie less than 5% of the time you’ll get this due to randomness if there is no change — which is rather impossible)
Sample size matters heavily. With more observations, estimates become more precise, so increasingly small differences can become statistically significant. With a large sample, you can therefore get a tiny, practically meaningless effect with a very small p-value.
Eg effect of $1 can be statistically significant (not random) which does not matter in practical terms if average is like $10000.
So the key point here is not only to look at the p-value but also at an actual change. If a drug gives you only 0.01% more hair, it doesn’t matter to you that it is guaranteed.