"Can I afford to fit my data in RAM?" is a whole other site, I presume...
"Can you afford to not put your data in RAM" is another one.
"Did you even need all of this data?" is the one after that one
Does my data fit in RAM?
81–90 of 167 posts
Re: Does my data fit in RAM?
#82Earlier quoted context omitted.
Yeah, except Hadoop provides redundancy, easier backups, turnkey solutions for governance and compliance and easier scaling I’m no fan of distributed systems that sit idle at 2% utilisation when a single node would do : BUT, reducing it down to “cost” and “does it fit in RAM” is way too reductive
And reducing it to a single machine means I have an order of magnitude less time spent on setting it up, maintaining it, adapting all my code, debugging, etc. These days I’m firmly of the opinion that if you can make it run on a single machine, you absolutely should, and you don’t get more machines until you can prove you can fully utilise one machine.
What about services where background load causes unacceptable latencies due to other bottlenecks on that single machine? What if you are IO bound and you couldn't possibly use all of the CPU or memory on any system which was given to you?
Re: Does my data fit in RAM?
#83The Amazon listing has an Azure instance type and links to Azure docs. A quick search shows that you can get at least 24TiB from AWS: https://aws.amazon.com/ec2/instance-types/high-memory
Thanks for pointing out my sloppy mistake.
I have chosen for the cloud options to only select virtual instances that can be spun up on demand. The high-memory instances you link to are purpose-build.
On the other hand, it is true, they do exist.
Re: Does my data fit in RAM?
#84"Can I afford to fit my data in RAM?" is a whole other site, I presume...
The point is that the alternative is far more expensive (taking from the tweet linked on the page that inspired it)
Re: Does my data fit in RAM?
#85Short answer: It fits in RAM if it's <= 12288 GB
The original Twitter thread is funny. This site doesn't add anything. It looks like SEO more than anything, and now that HN has linked to it, it's been successful.
The original site yourdatafitsinram.com is now a get rich quick scheme site.
Also, the site that remained wasn't ever updated. So I took some time and updated some links.
I liked updating/creating this because I love the simplicity of the concept / thought behind it.
Re: Does my data fit in RAM?
#86The problem is DRAM price hasn't drop one bit. The lowest price floor per GB has been similar for the past decade. Roughly at $2.8/GB in 2012, 2016, and 2019. And all DRAM manufacturers has been enjoying a very profitable period. And yet our Data size continue to grow. We can fit more Data inside memory not because DRAM capacity has increase, but we are simply increasing memory channels.
Everyone knows that DRAM prices have been in a collapse since early this year, but last week DRAM prices hit a historic low point on the spot market. Based on data the Memory Guy collected from spot-price source InSpectrum, the lowest spot price per gigabyte for branded DRAM reached $2.59 last week. https://thememoryguy.com/dram-prices-hit-historic-low/ You've selected out the low points on the graph: 2012, 2016, and…
> Everyone knows that DRAM prices have been in a collapse
I'm afraid I had no idea that had happened at all.
> since early this year,
Since..when?..now? Feb 12th is pretty early in the year AFAIAC
Re: Does my data fit in RAM?
#87You can fit 48 TB on a HPE MC990 X though I'm pretty sure that's got one of those NUMA architectures that SGI had with the UV 3000 or whatever.
I remember jokingly telling my team to spend the millions of dollars we did expanding our clusters with one of these and just processing in RAM. I honestly don't think I did the analysis to make sure it would be actually better.
It was 'jokingly' because we couldn't afford three of these machines anyway. The clusters had the property that we could lose some large fraction of nodes and still operate, we could expand slightly sub-linearly, etc. which are all lovely properties.
It would have been neat, though. Ahhhh imagine the luxury of just loading things into memory and crunching the whole thing in minutes instead of hours. Gives me shivers.
Re: Does my data fit in RAM?
#88IIRC, the largest IBM z15 can have up to 40 TiB of RAM.
The website needs updating.
Re: Does my data fit in RAM?
#89The problem is DRAM price hasn't drop one bit. The lowest price floor per GB has been similar for the past decade. Roughly at $2.8/GB in 2012, 2016, and 2019. And all DRAM manufacturers has been enjoying a very profitable period. And yet our Data size continue to grow. We can fit more Data inside memory not because DRAM capacity has increase, but we are simply increasing memory channels.
Everyone knows that DRAM prices have been in a collapse since early this year, but last week DRAM prices hit a historic low point on the spot market. Based on data the Memory Guy collected from spot-price source InSpectrum, the lowest spot price per gigabyte for branded DRAM reached $2.59 last week. https://thememoryguy.com/dram-prices-hit-historic-low/ You've selected out the low points on the graph: 2012, 2016, and…
7.5 percent drop since 2012 (using your numbers plus the OP’s number).
That’s not what we in computing refer to as a “collapse”.