This is weird because there's great variety between books and publishers, and there are lots of hidden costs. The one most self-publishers miss is marketing and sales---at the big publishers, there are
I am not a publishing industry insider, so these numbers are an educated SWAG, but let's see if they shed any light.
Let's say a top editor at a big-5 genre publisher makes $100k a year. That's way high, but we'll roll with it, 'cuz it makes the math work out nicely. Let's say they work on ten books a year, also probably high, but that's $10k/book. If they make $50k and work on five books a year that's still the right ballpark.
The art and production departments are shared across a publisher's whole line, or possibly even multiple imprints. Let's say the department costs $300k total and does thirty books a year, for $10k/book, which includes the cost of soliciting and paying the artist for the cover art, ebook and physical production, copyediting and proofreading.
We're up to $20k now.
Now we get to the fun part. Your big-5 publisher literally pays people to go read your book and then drive to all the bookstores in a geographic area and convince them to buy it. Figure that there are 10 of them in the US, each of them pulls in $50k, and they're repping 25 books a quarter, so 100 a year. That's another $5k on top just for their salaries, not counting their travel and expenses, for which $15k a person a year seems not unreasonable. And this isn't counting ad buys or any marketing, just plain sales. So figure half your book's total budget is sales.
So that's $40k/book for a midlist, 100k-word genre title from a well-regarded editor before we get to the author or the physical production.
(Things I'm leaving out: legal department, some kind of business/management structure, some kind of web site???)
Professional rates in fiction are a princely seven cents a word, so for an author who's worked a year on a book (not unreasonable) that's a whole $7k---but this book was acquired by a well-regarded editor at a good house, so let's say the author is getting a whole twenty cents a word, or $20k, for this year of labor. That's a third of the cost of the book so far. Accounting around authors in publishing gets weird---advances versus residuals, etc.; actually accounting in publishing is just plain weird, period---and a $20k advance is princely, but not unreasonably so.
So then the publisher has spent $60k on your book, and wants to earn its money back and then some.
My sense is that 5k sales is a low number, and certainly you aren't going to get a $20k advance next time, if your next book gets picked up at all. Publishing is a hits business---most sales happen in the first few months---so rates matter more than absolute numbers. 5k sales over ten years might finally earn out your advance, but 5k sales in the first month will get the second book of your trilogy picked up. 100k sales is John Scalzi territory, and my sense is that 30k sales is comfortable midlist territory.
Now, 30k sales on $60k spent out of the gate works out to $2/book, or $8/book profit at $10/book ($13 list with an Amazon/wholesale cut of 30%), but remember that in a hits business not nearly every book will sell even that many copies, but no one can predict ahead of time which ones, so publishers need to invest in many more books which won't sell to find the ones that will. It's anecdotally reported in the publishing industry---and I think it's evident in the financials as well---that breakout, millions-of-copies-sold success 50 Shades of Gray subsidized essentially all of Random House's other books for a year or two.
It depends on your hits business how bad the ratio is, but in genre publishing figure that less than half of books earn out for the author, so at $60k/book that's 6k sales minimum, which not every book makes. (Classic scenario: publisher gives you a big advance on your outline but then decides not to sell or market the book when you turn it in and they don't like it, but they don't want to pay the kill fee, so they rush it through production, ignore it in marketing, print the minimum required copies, and hope nobody buys them. Whoops.)
The key expense self-publishing discussions usually miss is really the sales and marketing one, followed closely by editorial. And freelance editors do exist and are quite good, so if you're self-publishing, consider one. I don't know of any freelance book salespeople, though. For some reason salespeople are very good about getting paid.
I should add that this is not to say that, if you're self-publishing, you should expect to spend $60k on your book. My sense is that even $10k/book is on the high side. Certainly out here in the freelance world I would tell an author expecting to put that much up to self-publish her book that she was being taken for a ride, and not the fun kind.
Remember that you're giving up sales and marketing, and you're not paying yourself an advance, so you're down to $20k already, and then you're working with newer artists and freelance professionals. Depending on how good you are at the various aspects of the work (and don't kid yourself here, he says mostly to himself), figure $3-5k is a decent ballpark for an anthology or a full-length novel. It's not going to sell 6k copies either, but then paying more probably wouldn't help its sales any. Still, do your research, don't shortchange your business partners, pay for quality where it counts, and don't pay for vanity publishers (Author Solutions, etc.)