Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fast Database Emerges from MIT Class, GPUs and Student’s Invention

data-informed.com

Post reply on HN
Print view View original

21–30 of 44 posts

Re: Fast Database Emerges from MIT Class, GPUs and Student’s Invention

#21

Post by phyalow »

tmostak wrote: post #17

Hi all, MapD creator here - I'd be happy to answer any questions. The 700K figure was done by rendering polygon files to textures and using them as lookup tables - it ran pretty fast on the CPU so its not all the GPU there. However if a point falls on a border - perhaps 1% of cases for say a medium-sized raster of the US - you have to do a geometric lookup as usual - which I didn't benchmark. Probably better would be…

Thanks for the clarification! I was highly suspicious of the 700k figure, it makes a lot more sense with the context of precomputing and then looking up.

Re: Fast Database Emerges from MIT Class, GPUs and Student’s Invention

#22

Post by throwaway1979 »

tmostak wrote: post #17

Hi all, MapD creator here - I'd be happy to answer any questions. The 700K figure was done by rendering polygon files to textures and using them as lookup tables - it ran pretty fast on the CPU so its not all the GPU there. However if a point falls on a border - perhaps 1% of cases for say a medium-sized raster of the US - you have to do a geometric lookup as usual - which I didn't benchmark. Probably better would be…

Heh ... when I saw the speedup numbers, the first thing I wondered was ... index? Regardless, this is impressive work. Do you have a tech report you can point to for someone to get more details? I'd love to learn more about your setup. Also, would you mind describing your programming background? I have a cs PhD and I admit I never had the guts to get into GPUs! You've given me a whole lot of motivation!

Re: Fast Database Emerges from MIT Class, GPUs and Student’s Invention

#23

Post by intended »

tmostak wrote: post #18

(MapD creator again) - Also, check out http://worldmap.harvard.edu/tweetmap for a live demo - even though its been a bit buggy under Slashdot load and I had to disable GetFeatureInfo requests (i.e. click to get tweet). Also see here for a demo of a new interface I'm messing with that does animation - http://www.youtube.com/watch?v=Foo8pYbSPv4 .

And roughly half an hour ago, I was hoping for something that could help me map GIS data and animate the changes over time so that I could see before and after effects.

Look forward to seeing this released.

Re: Fast Database Emerges from MIT Class, GPUs and Student’s Invention

#24

Post by tmostak »

throwaway1979 wrote: post #22
tmostak wrote: post #17

Hi all, MapD creator here - I'd be happy to answer any questions. The 700K figure was done by rendering polygon files to textures and using them as lookup tables - it ran pretty fast on the CPU so its not all the GPU there. However if a point falls on a border - perhaps 1% of cases for say a medium-sized raster of the US - you have to do a geometric lookup as usual - which I didn't benchmark. Probably better would be…

Heh ... when I saw the speedup numbers, the first thing I wondered was ... index? Regardless, this is impressive work. Do you have a tech report you can point to for someone to get more details? I'd love to learn more about your setup. Also, would you mind describing your programming background? I have a cs PhD and I admit I never had the guts to get into GPUs! You've given me a whole lot of motivation!

Thanks! Yeah I'd like to revisit this benchmark soon (I'm actually working on rendering choropleths on the GPU now). I'm working on a database paper that will hopefully be ready for VLDB submission by next month - that would probably be the best.

In terms of CS background - nothing formal except for AP comp sci I took in high school like 12 years ago. However I've always loved hacking on things on the side, and a combination of my research on the Middle East/burning desire to make an Iphone app kind of got me back into it. Then to be fair I took a GPU Graphics class (basically OpenGL) at Harvard and then a Database course at MIT while doing my Masters in ME Studies, which is where this project started.

Re: Fast Database Emerges from MIT Class, GPUs and Student’s Invention

#25

Post by tmostak »

intended wrote: post #23
tmostak wrote: post #18

(MapD creator again) - Also, check out http://worldmap.harvard.edu/tweetmap for a live demo - even though its been a bit buggy under Slashdot load and I had to disable GetFeatureInfo requests (i.e. click to get tweet). Also see here for a demo of a new interface I'm messing with that does animation - http://www.youtube.com/watch?v=Foo8pYbSPv4 .

And roughly half an hour ago, I was hoping for something that could help me map GIS data and animate the changes over time so that I could see before and after effects. Look forward to seeing this released.

Is your data relational? I can import anything from Postgres, MySQL, Solr (but has to be flattened), or shapefiles. If you tell me more perhaps I could put it into the system for you and we could try it out.

Re: Fast Database Emerges from MIT Class, GPUs and Student’s Invention

#26

Post by intended »

tmostak wrote: post #25
intended wrote: post #23

Earlier quoted context omitted.

And roughly half an hour ago, I was hoping for something that could help me map GIS data and animate the changes over time so that I could see before and after effects. Look forward to seeing this released.

Is your data relational? I can import anything from Postgres, MySQL, Solr (but has to be flattened), or shapefiles. If you tell me more perhaps I could put it into the system for you and we could try it out.

Its company data, so I use that greatest of all databases: Excel.

I'll have to think/work on this a bit - my test sets are clean but not massive enough yet for animation, while my data source is messy messy data.

Do note I'm not in the states - I'm new to this, but my experience so for has been that getting usable geolocation data for the region I am looking at is not easy.

(Incidentally, having some contact info on your profile page would make it a little easier for anyone who wanted to get in touch with you)

Re: Fast Database Emerges from MIT Class, GPUs and Student’s Invention

#27

Post by slavak »

danbruc wrote: post #11
eelsen wrote: post #3

The claim of a 700,000x speedup makes me suspicious of pretty much everything else about the work.

The question is what kind of processing he did. Usually database systems are heavily constraint by the I/O bandwidth of the system - even the cheapest consumer CPU is able to process data orders of magnitude faster than discs are able to deliver new data. That is why database servers usually have the fastest discs money can buy, as much memory as you can fit in there in order to keep as much data as possible in main…

This is what I was wondering as well. I work on a BigData BI solution, and in my experience IO and not processing time is the final bottleneck. I would expect working on the GPU to be slower, not faster, than working on CPU, because of the extra time needed to copy the data/results into/out of the VRAM. Also 40M tweets is what I would consider the very bottom of the BigData scale. I would guess the use-case here is one where the calculation is very complex, much more so than the data itself.

Re: Fast Database Emerges from MIT Class, GPUs and Student’s Invention

#28

Post by zzleeper »

tmostak wrote: post #25
intended wrote: post #23

Earlier quoted context omitted.

And roughly half an hour ago, I was hoping for something that could help me map GIS data and animate the changes over time so that I could see before and after effects. Look forward to seeing this released.

Is your data relational? I can import anything from Postgres, MySQL, Solr (but has to be flattened), or shapefiles. If you tell me more perhaps I could put it into the system for you and we could try it out.

Hi Todd! Do you have an email? (mine is sergio.correia@duke.edu )

I'm working on a project linking bank credit with the geographic distance between individuals/firms and bank branches. It's a fairly big dataset (~4MM geocoded individuals with matched credit data) and MapD would seem like a cool tool to visualize how the data changes with time.

Cheers

Re: Fast Database Emerges from MIT Class, GPUs and Student’s Invention

#29

Post by sendob »

laurentoget wrote: post #2

He is not the first person to try to implement a GPU base database, and this is not a trivial undertaking by any means. http://gpgpu.org/tag/databases Does anybody know any source with more details on this work?

not his, but I found this interesting: http://www-cs.ccny.cuny.edu/~jzhang/papers/ACMGIS10_Extended...

Re: Fast Database Emerges from MIT Class, GPUs and Student’s Invention

#30

Post by tmostak »

zzleeper wrote: post #28
tmostak wrote: post #25

Earlier quoted context omitted.

Is your data relational? I can import anything from Postgres, MySQL, Solr (but has to be flattened), or shapefiles. If you tell me more perhaps I could put it into the system for you and we could try it out.

Hi Todd! Do you have an email? (mine is sergio.correia@duke.edu ) I'm working on a project linking bank credit with the geographic distance between individuals/firms and bank branches. It's a fairly big dataset (~4MM geocoded individuals with matched credit data) and MapD would seem like a cool tool to visualize how the data changes with time. Cheers

Hi, yes im tmostak at csail dot mit dot edu (does that fool the spambots these days)? Yeah email me and we can chat!
Post reply on HN