(very off topic)
How do you store, query, and manipulate data that includes 45 million new rows per day?Perhaps one shouldn't? I've always been a bit fascinated by the analytical tools in brokerage software - with a pretty good understanding of DSP and an appreciation for the fact that asset prices are somewhat periodic, it's hard to overlook the fundamental similarities between stock graphs and audio waveforms. Once you start performing FFTs or wavelet transforms and get a 'feel' for dealing with signals, patterns become very seductive...possibly too seductive: http://en.wikipedia.org/wiki/Pareidolia and http://en.wikipedia.org/wiki/Tetris_effect
Might there be another approach? You wouldn't prepare to go to the store by reviewing and analyzing the 1287 individual footsteps of your previous trip, or try to predict the content of a HN thread by textual analysis of all previous threads. Do we do so at a subconscious level, then? Not really - or rather, our subconscious tends to forget about things as soon as they cease to be important, which in the case of things like walking is a period of seconds or less. Processing large volumes of data is computationally expensive, but it turns out that simple rules can yield results that are both complex and useful, as in flocking and swarming behavior: see http://en.wikipedia.org/wiki/Boids and http://en.wikipedia.org/wiki/Swarm_Intelligence, plus everything from the wisdom of crowds to nonlinear dynamic systems (aka chaotic ones) like Newton's basin or the logistic equation.
I feel there are two other fundamental problems with the massive dataset + analysis approach. One is that you're not working in a closed system, and there's no sensible way to quantify unexpected events. 'Bigcorp CEO in Sex Scandal!' might cause the price of Bigcorp to tank if it's a major distraction or their largest customer base is among rural conservatives. If Bigcorp makes racing cars, it might just be good publicity! Now you can do some kinds of interesting posthoc analysis (eg for news stories that contain a stock symbol, measure the correlation between # of textually similar stories and stock volume/prices using a distributed windowing function) but we're a long way from having a browser plugin that trades based on the contents of your RSS feed.
Another problem is that of feedback. As you've discussed so ably above, people who spot an arbitrage opportunity will mine the hell out of it. And as we all know, traders are extremely subject to herding behavior even though all training suggests they do otherwise. Sure there are systematic contrarians, but I bet that if you just want to do academic analysis you could find a contrarian coefficient and quantify its damping effect on price or volume movements.
So rather than crunching vast quantities of stored data, I wonder if it might be better to treat price movements not as absolutes which you hope will reach a particular ceiling or floor, but as differential vector data with a short half-life. So far AI and modeling approaches seem to have focused on prediction (surprise) and don't perform especially well. I think it would be more interesting to map correlation variations for as large a number of nodes (listed securities) as possible - think how we intuitively appreciate the dynamics of a school of fish when watching a nature documentary, without performing any detailed analysis of individual fish trajectories.
Of course this still involves processing a lot of data, but storing it is less important because are only seeking to become more familiar with high-level behaviors inside that system. There's more to fishing than running trawlers!