Here are the problems with the current options:
-- phpBB and vBulletin are self-hosted. The new platform needs to be SaaS.
-- Both platforms suffer from feature bloat, especially vBulletin. Forums aren't just products, they are tools for managing communities. Whatever product is designed to replace phpBB and vBulletin needs to begin with the basic premise that the forum participants are the users and not the forum administrator.
-- phpBB and vBulletin right out of the box come with too many useless features enabled. Most are features you don't need or that you don't need until the forum already has a large number of users. They complicate the forum in the early days and hinder growth. A forum is like a shark. If their isn't movement/activity it dies.
-- One key mistake almost all first time admins make is making too many forum "rooms" too early in the game. A forum should start with no more than maybe two to three "rooms". The software shouldn't prevent you from making more, but it should advise you against creating more rooms when the "activity level per room" is still low. The company who builds this software should actually discover the ideal amount of activity per room level to promote growth via aggregate A/B testing across all its clients forums.
-- In fact, this previous suggestion is the first in a long list of features that help "prevent new forum administrators from shooting themselves in the foot" kind of like a safety switch on a gun.
-- The new forum needs to be designed first and foremost to help its admins grow the user base and help stimulate user participation. This is key. Most new forum administrators don't have a clue on how to do this. I would say that the startup that tackles this problem would need to try a Mint Blog approach, which the blog educates the product's users on the topic of community management.
-- Were I the product manager for such a product, I would read Clay Shirky's "A group is its own worst enemy" several times before starting and regularly while designing the product. The idea in that essay is really important to understanding the state of forum software today and the problem you are trying to solve. Avatars are an example of a problem with forum software today. The avatars in both programs can be modified to be any size and can include animated gifs and whatnot. Basically, the avatar features are currently configuration over convention. On top of that the current defaults for avatars leads to a situation where avatars are so big that they create a lot of whitespace between responses that interrupt the feeling of the conversation threads. Allowing animated gifs results in distractions from the primary activity.
-- Invite features - Last time I checked, none of the current forum softwares are designed for viral/social growth.
-- Migration tools are necessary. You should make it easy for phpBB users and vBulletin users to migrate to your new forum platform.
-- Social network integration - The software should be designed from day one to have an API and to integrate with Facebook, Twitter, etc.
-- Constant email notifications by default to stimulate user participation just like Quora does. However this needs to be simpler to "dial down" as a user becomes more active as it eventually clogs up the email inbox of that user. Quora's control panel for this isn't very user friendly as it is very granular. A better approach would be to sort the types of notification emails by historical frequency from those that user has gotten the most of to those that the user has gotten few or none of. This makes the decision very easy for the user and promotes them to only turn off the notifications that are actually inconveniencing them.
-- I can cite at least one example that demonstrates that this is a product missing in the market: Zendesk. Zendesk comes with forum features, not because forum features are a necessary feature of a helpdesk, but because companies that need help desks typically need forums and there was no company that Zendesk could plug into via API for plug'n'play forum support. The forum software in Zendesk is adequate for most users' needs, but it doesn't scale well because it was always designed to be bolt on to the helpdesk itself.
-- I haven't used GetSatisfaction yet, but I would imagine that this is a shared issue with them as well.
-- Forums basically have two personas: hobbyists and shared-interest communities (i.e. head-fi.org, stevehoffman.tv, thewolfweb.com, church groups, etc.) and companies.
That all being said, I would say that an similar product that is also needed, especially inside companies, is QaaS. Quora as a Service.
A lot of companies want a Quora type solution first and foremost than a forum. I would say that Zendesk needs a product like a Quora-as-a-Service instead of a forum.
I'm currently working on my own startup right now, but if someone were to want to build this, I would be very happy to serve as an advisor/board member for a small stake of equity. This is a problem that I've given a LOT of thought to and that I would love to see someone solve.