Does my data fit in RAM? • Hacker News Board

Re: Does my data fit in RAM?

#41

Post by trhway » Wed, Feb 12, 2020, 8:23 PM UTC

wmf wrote: post #36

bluedino wrote: post #22

$2,000 each for 128GB LRDIMMs, 48 of those will be $100,000 and then you'll need another $20,000 to buy the rest of the server it goes in.

If the result is faster than a $250K Hadoop cluster then you're still ahead.

the way things today, the Hadoop cluster will be just a bit faster on that thing.

25 years ago i was suggesting clients to upgrade from 4MB to 6-8MB as it was improving their experience with our business software, these days i've already suggested a couple of customers to upgrade from 6TB and 8TB respectively ... as it would improve their experience with our business software. What's funny is that customer experience with business software back then was better than today.

Re: Does my data fit in RAM?

#42

Post by imron » Wed, Feb 12, 2020, 8:31 PM UTC

samspenc wrote: post #39

slumdev wrote: post #11

Earlier quoted context omitted.

The original Twitter thread is funny. This site doesn't add anything. It looks like SEO more than anything, and now that HN has linked to it, it's been successful.

Could you share the link to the original Twitter thread? I guess the link was changed on HN before some of us got here.

On the linked page, it says:

"Inspired by this tweet" which links to https://twitter.com/garybernhardt/status/600783770925420546

Re: Does my data fit in RAM?

#43

Post by ksec » Wed, Feb 12, 2020, 8:32 PM UTC

The problem is DRAM price hasn't drop one bit.

The lowest price floor per GB has been similar for the past decade. Roughly at $2.8/GB in 2012, 2016, and 2019. And all DRAM manufacturers has been enjoying a very profitable period.

And yet our Data size continue to grow. We can fit more Data inside memory not because DRAM capacity has increase, but we are simply increasing memory channels.

Re: Does my data fit in RAM?

#44

Post by gameswithgo » Wed, Feb 12, 2020, 8:45 PM UTC

ksec wrote: post #43

The problem is DRAM price hasn't drop one bit. The lowest price floor per GB has been similar for the past decade. Roughly at $2.8/GB in 2012, 2016, and 2019. And all DRAM manufacturers has been enjoying a very profitable period. And yet our Data size continue to grow. We can fit more Data inside memory not because DRAM capacity has increase, but we are simply increasing memory channels.

buying the ram probably cheaper than buying database liceneses and/or the machines to run them and/or the time you spend making things fast enough when using them.

Re: Does my data fit in RAM?

#45

Post by endorphone » Wed, Feb 12, 2020, 8:49 PM UTC

ramraj07 wrote: post #2

Legit question: I have a dataset that's a terabyte in size spread over multiple tables, but my queries often involve complex self joins and filters; for various reasons, I'd prefer to be able to write my queries in SQL (or spark code) because it's the most expressive system I've seen. What tool should I use it to load this dataset on RAM and run these queries?

Every major database will load the hottest data into RAM, where the scope of "hottest" broadens to whatever amount will fit in RAM. A small percentage of them require you to confirm how much RAM it can use for this cache.

Putting the data on a ramdisk just becomes entirely redundant because it's still going to create a second memory cache that it uses.

Many operations do a local cache warming by running the common queries over the database before they it is brought online for processing. As a secondary note, people often under-estimate the size of their data because they don't account for all of the keys, indexes and relationships that also would be memory cached in an ideal situation.

Re: Does my data fit in RAM?

#46

Post by iaabtpbtpnn » Wed, Feb 12, 2020, 8:51 PM UTC

Sure, my data fits in RAM if you take one of those example machines, which we have, and which are currently hosting dozens of production VMs, and instead dedicate the whole thing to my database. I'd love that, but it's never going to happen.

Re: Does my data fit in RAM?

#47

Post by zaltekk » Wed, Feb 12, 2020, 8:54 PM UTC

The Amazon listing has an Azure instance type and links to Azure docs.

A quick search shows that you can get at least 24TiB from AWS: https://aws.amazon.com/ec2/instance-types/high-memory

Re: Does my data fit in RAM?

#48

Post by keanzu » Wed, Feb 12, 2020, 8:58 PM UTC

thedance wrote: post #16

chmod775 wrote: post #6

Earlier quoted context omitted.

A single terabyte is a few magnitudes from what you need big-data-anything for. You could probably work with that just fine on your average 64GB ram desktop with an SSD.

A small computer with 1 SSD will take at least 10-20 minutes to make a pass over 1TB of data, if everything is perfectly pipelined.

Samsung claims their 970 Pro NVMe can read 3.5GB/s sequentially. That's about 300 seconds or 5 minutes per TB.

Re: Does my data fit in RAM?

#49

Post by takeda » Wed, Feb 12, 2020, 9:00 PM UTC

iaabtpbtpnn wrote: post #46

Sure, my data fits in RAM if you take one of those example machines, which we have, and which are currently hosting dozens of production VMs, and instead dedicate the whole thing to my database. I'd love that, but it's never going to happen.

It's still way more efficient to use it for that database, than running a dozen of hadoop nodes as VMs.

Re: Does my data fit in RAM?

#50

Post by [deleted] » Wed, Feb 12, 2020, 9:01 PM UTC

iaabtpbtpnn wrote: post #46

Sure, my data fits in RAM if you take one of those example machines, which we have, and which are currently hosting dozens of production VMs, and instead dedicate the whole thing to my database. I'd love that, but it's never going to happen.

[deleted]