$2,000 each for 128GB LRDIMMs, 48 of those will be $100,000 and then you'll need another $20,000 to buy the rest of the server it goes in.
Finally a machine that can run Chrome.
Does my data fit in RAM?
31–40 of 167 posts
Re: Does my data fit in RAM?
#32Legit question: I have a dataset that's a terabyte in size spread over multiple tables, but my queries often involve complex self joins and filters; for various reasons, I'd prefer to be able to write my queries in SQL (or spark code) because it's the most expressive system I've seen. What tool should I use it to load this dataset on RAM and run these queries?
in memory sqlite is pretty fast
Re: Does my data fit in RAM?
#33$2,000 each for 128GB LRDIMMs, 48 of those will be $100,000 and then you'll need another $20,000 to buy the rest of the server it goes in.
Finally a machine that can run Chrome.
Can still “only” have 6000 tabs.
(Someone did it recently with 1.5TiB of ram on a Mac Pro. Then Linus from Linus tech tips did it on Windows with 2TiB)
Re: Does my data fit in RAM?
#34$2,000 each for 128GB LRDIMMs, 48 of those will be $100,000 and then you'll need another $20,000 to buy the rest of the server it goes in.
[deleted]
Re: Does my data fit in RAM?
#35Looks like I'm all set with my -12PB dataset fitting in ram.
12TiB, not PiB!
Hope you read this before you spent $50,000,000 dollars/month (although you might be able to negotiate a discount).
Re: Does my data fit in RAM?
#36$2,000 each for 128GB LRDIMMs, 48 of those will be $100,000 and then you'll need another $20,000 to buy the rest of the server it goes in.
If the result is faster than a $250K Hadoop cluster then you're still ahead.
Re: Does my data fit in RAM?
#37$2,000 each for 128GB LRDIMMs, 48 of those will be $100,000 and then you'll need another $20,000 to buy the rest of the server it goes in.
if you can get by with 4TB, it's less than $30/hr on AWS
Re: Does my data fit in RAM?
#38Earlier quoted context omitted.
You should use the cache or persist call on the spark data frame/dataset. Persist gives you more control.
This is what most people in my org do, this is orders of magnitude slower than running queries on the same dataset on redshift with optimized presorting and distribution. Redshift doesn't scale for tens or hundreds of parallel users though, so looking for options
I'm about 75% joking: restore a new cluster from a snapshot for horizontal scaling. They did launch a feature along those lines, because my joke suggestion probably doesn't scale organizationally: take a look at Concurrency Scaling. This is also a fundamental feature of Snowflake with the separation of compute and storage.
Re: Does my data fit in RAM?
#39Short answer: It fits in RAM if it's <= 12288 GB
The original Twitter thread is funny. This site doesn't add anything. It looks like SEO more than anything, and now that HN has linked to it, it's been successful.
Could you share the link to the original Twitter thread? I guess the link was changed on HN before some of us got here.
Re: Does my data fit in RAM?
#40Fine print: But it might cost you $300K CapEx or $800K/yr OpEx. Hope you have a budget!
I would love to know how you came up with these numbers.