I think there's some accidental synecdoche here.
At least for those of us endowed with the gift/curse of forming memories as life rushes by, they can appear to be composed of details our senses perceived around us enmeshed with details about how our wetware experienced that moment. Sometimes, we even have false memories of things that didn't happen, or that happened differently than we remember.
There's a separate world of practice where people develop mnemonic devices (spaced repetition, narratives, memorable nonsense phrases, memory palaces, songs/rhyme/prosody, hand gestures) to help memorize things better than we could do by accident, or just by thinking "remember X".
I would guess most people who are able to form memories can find a mnemonic device that works for them when they need to remember things. The very idea of a "memory palace", however, is so absolutely foreign to my experience of the world that I don't think I could have invented it on my own. Once someone describes it, I have an abstract understanding that the ability to arrange things in visual space could be a powerful mnemonic. I can, after all, remember at least a few things about many thousands of physical places I've been at some point in my life, even though I can't really picture any of them vividly. But, as a mnemonic, it's useless to me--I can't conjure it.
My partner can remember things like what shirt she was wearing when we went to a specific restaurant a decade ago. I have abstract knowledge that we have been to that place, but I couldn't tell you anything else about it. For her, memories like this are a springboard into conjuring many other details about the event. Sometimes her recall injects enough context to jog my own. Sometimes it's all lost in the fog.
Whether you are "good" or "bad" at memorizing things in an intentional way, unless you have infinite time to creative narrative mnemonics as life rushes by, I suspect aphantasia impacts your ability to recall daily life.