> The issue I see with Elite/Privateer/etc is that its hard to encourage group play which tends to be a near requirement for most MMOs to succeed.
I don't think it would have to be "massive", multiplayer with a dozen of players or so could be just fine if we talk about coop (aka PvE) only: you could form convoys with escorts, attack/defend bigger objectives than in solo play, etc.
Competitive multiplayer (PvP) brings a lot of issues, compromises and downsides; one of them being the ability to customize your game. I casually play Oolite, and it's key feature is precisely the ability to modify almost anything in the game. Those mods for instance let you escort an AI ship or be escorted by AI wingmen, which compensates a bit for the lack of multiplayer.
> If ships are the primary draw how do you group people up
When you look at it, this genre is not very different from an ARPG: you upgrade your ship just like to equip your character; your ship can be a freighter or a light fighter and it is similar to character classes. If you need more variety you can for instance introduce distinct damage types (energy damage with lasers and kinetic damage for missile) with specific defences (shields vs lasers and armour vs missiles). Or you can reserve certain capabilities like interstellar jumps to certain ships.