I was in the same situation as you a few years ago. My Mac app was making enough money to be helpful but not quite enough to be worth the trouble. I seriously considered open sourcing it. But I stuck with it and, four years later, it grew and began supporting my family full-time. If you truly believe there's no chance of it ever growing larger, than by all means, retire it. I've sunsetted many apps that never caught…
That's a great story! What's the app if I may ask?
VirtualHostX was my first app, the one that grew slowly over a few years and then took off. Hostbuddy is a small, companion app that does moderately well. Hobo is my newest, still in beta, that I hope to kick off in February.