It's also a really common thing that people do.
Like, I buy stuff on Amazon for my parents, friends, and family all the time.
Not to mention the common cases of shipping to work address, having multiple residence addresses (e.g. students who have a home and a dorm address), shopping while visiting family out-of-state.
It's a really common fraud scheme because it's not uncommon for regular people to do that. Otherwise, we would not have a need for a "shipping address" at all: all the purchases could go to the belling address, right?
So, basically, you are exactly the kind of bad seller that ruins it for everyone else. Your inability to deal with fraud adds so much more headache to the experience.
I hope that, as an independent seller, you have it in bold letters on the front page that you only ship to the billing address. This way, people who you won't ship to anyway won't waste their time (and people who you won't ship to will not be inconvenienced anyway).