A few things come to mind.
Each use case—typesetting and word-processing—have enough standardized tools to fill an entire application's interface. I think making the compromises needed to do be able to do both adequately would be too much to make either task worthwhile. Maybe that wouldn't necessarily be true for digital, like ePub, but I think that would quickly become true for print.
I don't feel a typesetting interface/context is amenable to writing long-form documents. When I take into consideration multi-column layouts, running headers and footers, art wrapping, page flow across multiple pages, there are what appears to be a lot of potential distractors from the writing process. Word processing documents, when I hide as much of the interface as I can, allow for the tight focus needed for long-form writing. Someone else in this thread suggested having writing and editing happen in another window, but that's still creating an abstraction from the layout, and not really having edits happen "in real time" if only because then editing is not happening actually within the layout.
Finally, I think there would be a significant barrier to training for many authors trying to work in layouts. There are applications, like QuarkXpress, that enable word-processing-like tools in the layout. But I couldn't imagine asking an author to learn enough about the above to ask them to write a long-form paper, much less an entire book, in that environment. Fonts are handled differently, styles are handled differently, etc.