As somebody with experience in developing many OSS projects of different sizes (Redis, Hping, Jim Tcl, Visitors web analyzer, and many additional smaller ones), I think that the solution is simpler than it appears: just do what you want. When you are inside the flames of a successful open source project, you may think that the solution space is binary: don't do anything, or do everything people are demanding from you. Instead you can just keep doing what you want, cherry picking what issues you want to address, reply to, what features you want to add, and so forth. Just give you a fixed amount of time to spend on the project (in my case most of the time it was "all the time I've in a day, but up to 6/8 max", but it can be even just 10 minutes every day), and in this time do the things you like to do and ignore all the others. A few issues/PRs will be perfectly aligned to what you feel is right and you'll enjoy taking care of them. Others will not, and who cares?
And anyway, doing things this way I was able to write a database that beaten, in the market, products developed with hundreds of developers while I was alone, so there must be some merit in what the original author feels is worth investing into. So, just do what you want, but:
1. Don't fall in the trap of thinking that who asks you for things is doing some kind of mistake or abuse just because (for example) they are not paying you. Nope, they are fine asking for things, you are fine to ignore the requests.
2. Don't fall in the trap that you are not accountable about the quality of the software just because it's free software: do only want you want, but ship finished work that is reasonably well written and well documented. To do what you want, at your own peace and according to your own personal expectation has NOTHING to do with the quality of your work. Software fails, but one thing is to ship terrible stuff just because "Hey it's free", another thing is to do things the way you want, but with love.
3. When people attack you, reply gently saying what you think. Don't get trapped into fights, don't feed the troll, remember that many criticizing you, if you are stealing money from the table providing something free, have specific agendas (but sometimes they are just assholes), and their goal is to mount a big case. Stop them replying carefully and without anger, then let the discussion end, or continue without you.
4. Make good friends in the process. They'll help you immensely when there are hard times. Remember: the smartest people 99% of the times have a big hearth and are the most friendly.