Blame the (metal) compositor unions which back in the day bargained for sinecures where all their members were guaranteed perpetual employment rather than choosing to participate in the digital revolution.
Fortunately, some folks did work to preserve the craft and beauty of books --- Dr. Donald Knuth taking a decade off from writing _The Art of Computer Programming_ to create TeX (though initially he thought he'd do it over a sabbatical) is one shining example.
Robert Bringhurst's authoring _The Elements of Typographic Style_ also made a huge difference (I've lost count of how many copies I've given as gifts to folks).
A further issue is that doing a good page layout over an entire chapter (or book if the pagination is continuous) is an NP-hard problem --- I've had a chapter come out correctly on a first pass exactly once in my career (fastest 40 minutes of my life). The usual work-flow is something like:
- check all characters to ensure that hyphens are properly set, en and em dashes replace them where appropriate, and correct the setting of any instances of what should be special characters such as prime or double primes
- assign all formatting and ensure that all heads and paragraphs have settings which will forbid widows/orphans and verify that the callouts for all figures/photos/tables are correct
- review the entire chapter from beginning to end, page by page, verifying that each ends as it should at the bottom of the page, and that a referenced element shows on that page spread
- for instances where things don't work out, check to see which paragraphs can be adjusted to run longer or shorter by one or more lines, adjusting this until one finds a set of adjustments which results in a proper appearance for the page/spread --- repeat for all future pages --- if a particular spread/figure placement is a problem, back up and see if changing previous pages will fix it --- check the last page to ensure that it is full enough, if not, adjust previous spread, if that doesn't work, see if running the entire chapter long or short by a line will fix it.
- review the entire chapter again to ensure that there are no bad breaks or stacks, add discretionary hyphens or non-breaking spaces or adjust paragraph settings as necessary, ensuring that pages still base-align
If someone wants to write an ePub reader or page formatter which can do that, I'd be glad to see it.