[Author here] The big way TogetherJS differs from many of the other realtime products out there, like Firebase or (to a lesser degree) ShareJS, is that it doesn't do any storage. It coordinates real-time sessions, but it's not there to take the place of the application, instead it augments the application. If your application has persistence, it has traditional persistence. Whatever auth system you have is your own. At the same time while it dodges any persistence, it extends the scope beyond what those other tools typically do to include more with session management and the standard tools that I think are needed for collaboration. There's nothing exciting about text chat, but you need it anyway.
Still, ignoring persistence I think will help keep the surface area of the TogetherJS smaller compared to some other tools. Or at least focused on other things.
People have mentioned the idea of being able to chop TogetherJS up before, but I have a hard time seeing how that would actually work. For instance, the architecture is built around a pretty dumb server, so hooking into Streams would seem counter to that principle. And just allowing for the possibility would add a lot of complexity. Ultimately the UI is actually the bulk of what makes up the TogetherJS code (or maybe it's the part I find hard and so I feel like I spend a lot of time with it), and I don't know how to make this kind of UI while being modular. Without the UI I don't think it's nearly as compelling a project (becomes more like ShareJS, or Firebase, or Google Drive Realtime API – neat but a big investment, and sometimes only appropriate for greenfield projects.
Anyway, I'm certainly interested in how people want to extend or integrate with TogetherJS. I'm skeptical of splitting it up into pieces, as it is theoretically easier to do things that way, but I expect in practice it will just leave a confusing pile of pieces that aren't fun to put back together. But that doesn't mean that there aren't use cases worth exploring... and with some creativity there might be other approaches to handling those cases.