Earlier quoted context omitted.
I'm peripherally in the industry, but even I can say middleware is huge. Speedtree, as you mentioned, shows how niche the suites can be. You also have FMV players (especially on mobile) and of course physics engines are huge. Look for Havok and you'll see it powers most AAA games. I don't know specific pain points you can ease, but if you can generalize it, game companies will probably pay for it.
potatolicious mentioned animation systems http://news.ycombinator.com/item?id=2303453 Is this an area that is just not very important (wont buy) or one that just is low on the list of todo items? speedtree begs the question of what about speedFOOBAR such as buildings, cities, caves, worlds, fish or how about speedanimal (generating animal + animation)? Is the big place to find a nitch in Art or are there better place…
If you want to know what sorts of libraries the game industry wants to license, take a look at RAD Game Tools, they are the masters of this stuff. Some of the stuff they are currently focused on are large data-set compression/decompression (Charles Bloom) and I would guess some sort of tool/library to generically handle sparse virtual texture rendering (Sean Barrett) which is similar to id's megatexture technology... though the SVT guess is more speculation on my part than public knowledge.
In any case, this market is probably not a great one to get into unless you are a deep subject matter expert in one of these areas (or you want to become one and have a lot of time to spend doing so), the bar is set pretty high already and few game development studios are willing to put any effort into exploring solutions that aren't either well-proven or from people they already know and trust (the game industry is still rather incestuous).