1.1B taxi rides benchmarked on distributed GPU-powered MapD

Post by menegattig » Wed, Apr 26, 2017, 4:14 PM UTC

tmostak wrote: post #4

arnon wrote: post #3

1.1B records = 500GB of raw CSV data. This fits into RAM quite easily on a machine like the P2.8xlarge, especially when compression is used (like MapD uses). I'd like to see how well this performs on a dataset that doesn't fit in the RAM.

Its honestly a pretty small dataset for us. MapD can easily do sub-100ms queries on 100B+ records with a single rack. You can fit a lot of GPU + CPU memory on a small cluster. That RAM goes even further when you use compression like you mentioned. We're fast at pulling data off of disk but we've never really aimed to be a traditional disk-based data warehouse. There are already great systems for that.

Post by tmostak » Wed, Apr 26, 2017, 4:16 PM UTC

Bedon292 wrote: post #7

mmrezaie wrote: post #5

Is there a bridge for using MapD with Spark interface or somehow combining them? This can be interesting for the clusters with a lot of GPUs and a lot of data to do data manipulations.

That actually sounds amazing, I would totally use that. I am actually hoping that they will create GDAL bindings for geospatial data.

Post by Bedon292 » Wed, Apr 26, 2017, 5:09 PM UTC

tmostak wrote: post #13

Bedon292 wrote: post #7

Earlier quoted context omitted.

That actually sounds amazing, I would totally use that. I am actually hoping that they will create GDAL bindings for geospatial data.

We actually already can import geospatial formats via GDAL (shapefiles, geojson, kml). We can render points and polygon data. More geospatial abilities to come!

Post by menegattig » Wed, Apr 26, 2017, 5:29 PM UTC

kornish wrote: post #16

menegattig wrote: post #11

Earlier quoted context omitted.

At what point do you think starts making sense cost-wise to use MapD (or GPU in general) instead of Redshift or BigQuery?

My guess is that it makes more sense when interactive query latency (hundreds of ms) is of the utmost concern.

Post by tmostak » Wed, Apr 26, 2017, 6:01 PM UTC

menegattig wrote: post #17

kornish wrote: post #16

Earlier quoted context omitted.

My guess is that it makes more sense when interactive query latency (hundreds of ms) is of the utmost concern.

Right, I agree with you. I was just wondering what kind of companies (except from financial sector) would be willing to spend hundreds of thousands to get their latency from hundreds of ms to dozens of ms. I'm saying that because if you have a very well-tuned Redshift cluster, you can easily get dozens of ms for your queries, spending thousands of dollars, not hundreds of thousands.