It all works this way (though a certain female family member would disagree, claiming to remember conversations word-for-word years later). But my memory works this way. A summary "party at so-and-so's house, weather was nice, overall vibe was ___". The rest is context. You know what the house/backyard is like, you know the general feel of that time of year, you know the crowd that usually comes, you can easily synth…
I think a large part of it is just that you store what is important to you. To me the day-to-day politeness is just filler. I don't care if they had black coffee or a latte. If someone was struggling with something and poured out their heart over multiple conversations, I'm going to remember what arguments and concerns they had and the mental model I built up around that situation. The filler is just unimportant and doesn't stick around.
My wife is the opposite. Signs of weakness are an embarrassment to be forgotten. She lives for the day-to-day.