Fast Database Emerges from MIT Class, GPUs and Student’s Invention

The question is what kind of processing he did. Usually database systems are heavily constraint by the I/O bandwidth of the system - even the cheapest consumer CPU is able to process data orders of magnitude faster than discs are able to deliver new data.

That is why database servers usually have the fastest discs money can buy, as much memory as you can fit in there in order to keep as much data as possible in main memory and the largest possible caches to avoid hitting the relatively slow memory bus as often as possible. The situation becomes much worse if the database is not read-only because all changes have to be persisted and you have to hit the disc on every change.

Therefore I think using GPUs will buy you nothing for common database application - processor speed was never your problem. But this does not exclude the possibility that there are operation, for example correlating millions of data points from your data set, where the processing power of GPUs comes handy, but this is most likely not what you will see in one of your line of business applications.

The number 700,000 is probably the result of a highly tuned GPU implementation versus a general purpose CPU implementation. A quad-core Core i7 at 3.0 GHz has a theoretical peak performance of 96 GFLOPs [1], the currently fastest supercomputer peaks at 27,112,500 GFLOPS (Titan, 560,640 cores, 8.2 MW [2]). 700,000 times the mentioned Core i7 is still more than twice the performance of Titan, 67,200,000 GFLOPs.

[1] http://stackoverflow.com/questions/15655835/flops-per-cycle-... [2] http://www.top500.org/lists/2012/11/

Here's a technical talk: http://www.youtube.com/watch?v=WSvh5ZPrR4w I couldn't find a paper, but fortunately this video is only 10 minutes.

He's got an in-memory column store, so (AFAIK) those are immediate advantages over PostGIS. He's also using pre-rendering of shapes into bitmaps (a form of caching/indexing) along with GPU hardware to accelerate intersection queries. It looks like this is a case where a problem (2D geo data) maps quite naturally to GPU hardware (2D textures) if only you can discover the mapping.

The same story was submitted to /., and the person who wrote the database answered several questions about it there. See http://search.slashdot.org/story/13/04/22/2225240/harvardmit... for more.

Hi all, MapD creator here - I'd be happy to answer any questions. The 700K figure was done by rendering polygon files to textures and using them as lookup tables - it ran pretty fast on the CPU so its not all the GPU there. However if a point falls on a border - perhaps 1% of cases for say a medium-sized raster of the US - you have to do a geometric lookup as usual - which I didn't benchmark. Probably better would be to do some tiled implementation - basically something like textured quad-trees.

And as I told the journalist I wasn't sure if I did the PostGIS indexing right - I feel like the quote got taken a little out of context - but the speedups were "high" to "quite high" regardless.

It appears from the graphs (yeah, I know) that he's probably doing trig (to locate the tweet) and collision detection (to determine the area of the city). He could also have more control over single precision algorithm choice than he would have via his original solution. It could be something as simple as pre-calculating the geolocation with a parallel CUDA kernel during ETL into PostGIS and bypassing its index generation.

I also have an itchy feeling that there may have been some confusion regarding conversions between "times" and "percent", possibly multiple times. [1]

Extrapolating that an iterative solution would take 40 days, without running it for 40 days, is tricky as well. Given multiple tweets in close proximity, PostGIS should benefit from caching the R-tree index.

I'm not assuming the inventor is doing these things, as his db prof was impressed, as well as CSAIL and the body awarding his prize. [2] I think, if anything, the reporting is intended to be high-level enough to have more broad appeal, which makes it more difficult for us to evaluate the results in a Gedankenversuch. "Reduce slowness here..."

[1] The sidebar says,'Some statistical algorithms run 70 times faster compared to CPU-based systems like MapReduce,' while the quote in the article says,'“The speed ups over PostGIS … I’m not an expert, and I’m sure I could have been more efficient in setting up the system in the first place, but it was 700,000 times faster,” Mostak said. “Something that would take 40 days was done in less than a second.”'

[2] Hey, logical fallacies do not necessarily mean the argument is wrong!