Short answer: It fits in RAM if it's <= 12288 GB
The original Twitter thread is funny. This site doesn't add anything. It looks like SEO more than anything, and now that HN has linked to it, it's been successful.
Does my data fit in RAM?
11–20 of 167 posts
Re: Does my data fit in RAM?
#12Short answer: It fits in RAM if it's <= 12288 GB
> Short answer: It fits in RAM if it's For a few given system configurations, and a given definition of "fits".
Re: Does my data fit in RAM?
#13Legit question: I have a dataset that's a terabyte in size spread over multiple tables, but my queries often involve complex self joins and filters; for various reasons, I'd prefer to be able to write my queries in SQL (or spark code) because it's the most expressive system I've seen. What tool should I use it to load this dataset on RAM and run these queries?
There are a few steps to consider before you are loading data into ram.
Can you partition the data in any useful way? For example if queries use separate ranges of dates, then you can partition data so that queries only need to touch the relevant date range. Can you pre-process any computations? Sometimes tricky things done within the context of multiple joins can be done once and written to a table for later use. Can you materialize any views? Do you have the proper indexes set up for your joins and filters? Are you looking at execution plans for your queries? Sometimes small changes can speed up queries by many orders of magnitude.
Smart queries + properly structured data + a well tuned postgres DB is an incredibly powerful tool.
Re: Does my data fit in RAM?
#14Funny story, about 6 years ago we got a HP DL980 server with 1TB of memory to move from an Itanium HP-UX server. The test database was Oracle and about 600GB in size. We loaded the data and they had some query test they would run and the first time took about 45 minutes (which was several hours faster than the HPUX), They made changes and all the rest of the runs took about 5 minutes for their test to complete. Final…
What an unfortunate cache-filling algorithm, though. With eight drive slots, 40 minutes of IO is about 31 megabytes per disk per second.
Re: Does my data fit in RAM?
#15Legit question: I have a dataset that's a terabyte in size spread over multiple tables, but my queries often involve complex self joins and filters; for various reasons, I'd prefer to be able to write my queries in SQL (or spark code) because it's the most expressive system I've seen. What tool should I use it to load this dataset on RAM and run these queries?
You should use the cache or persist call on the spark data frame/dataset. Persist gives you more control.
This is what most people in my org do, this is orders of magnitude slower than running queries on the same dataset on redshift with optimized presorting and distribution. Redshift doesn't scale for tens or hundreds of parallel users though, so looking for options
Re: Does my data fit in RAM?
#16Earlier quoted context omitted.
I am not a big-data guy but wouldn't it be along the lines of A) get a big honking server B) fire up "X" SQL server C) Allocate 95-98% of the RAM to DB cache?
A single terabyte is a few magnitudes from what you need big-data-anything for. You could probably work with that just fine on your average 64GB ram desktop with an SSD.
A small computer with 1 SSD will take at least 10-20 minutes to make a pass over 1TB of data, if everything is perfectly pipelined.
Re: Does my data fit in RAM?
#17I wonder how Abe Vigoda is doing...
Re: Does my data fit in RAM?
#18Fine print: But it might cost you $300K CapEx or $800K/yr OpEx. Hope you have a budget!
Re: Does my data fit in RAM?
#19Fine print: But it might cost you $300K CapEx or $800K/yr OpEx. Hope you have a budget!
You know what else costs? Humongous amount of servers to run silly stuff to orchestrate other silly stuff to autoscale yet else silly stuff to do stuff on your stuff that could fit into memory and be processed on a single server (+ backup, of course).
Add to that small army of people, because, you know, you need specialists of variety of professions just to debug all integration issues between all those components that WOULD NOT BE NEEDED if you just decided to put your stuff in memory.
Frankly, the proportion of projects that really need to work on data that could not fit in memory of a single machine is very low. I work for one of the largest banks in the world processing most of its trades from all over the world and guess what, all of it fits in RAM.
Re: Does my data fit in RAM?
#20Fine print: But it might cost you $300K CapEx or $800K/yr OpEx. Hope you have a budget!
This is about saving you money, so that's not the right fine print.
Look at it this way: This is for the person that's already going to get enough ram sticks to fit the entire data set or multiple of it, across many machines, and deal with the enormous overhead from doing queries across many machines. The revelation is that you can fit that much ram inside a single machine for a much cheaper and faster experience.