The downside is you have no one to recount the experience with later on, you might tell people about it but no one will ever understand what it was like unless they were there with you.
How's that a downside? Furthermore, why's it necessary for someone else to know about it? Doesn't it hold value even if you only experienced it?
I just really enjoy recalling shared memories with people. It reminds you of your shared past and helps you bond.