Does my data fit in RAM? • Hacker News Board

Re: Does my data fit in RAM?

#21

Post by w1nk » Wed, Feb 12, 2020, 7:23 PM UTC

chmod775 wrote: post #6

saber6 wrote: post #4

Earlier quoted context omitted.

I am not a big-data guy but wouldn't it be along the lines of A) get a big honking server B) fire up "X" SQL server C) Allocate 95-98% of the RAM to DB cache?

A single terabyte is a few magnitudes from what you need big-data-anything for. You could probably work with that just fine on your average 64GB ram desktop with an SSD.

Another poster already replied with a decent refutation of this claim, but a single pass over a TB of data is often not enough for 'big data' use cases and at tens of minutes per pass, it may very well be infeasible to operate on such at dataset with only 64GB of memory.

In the machine learning world, some of the algorithms that are industrial workhorses will require you to have your dataset in memory (ie: all the common GBM libraries), and will walk over it lots of times.

You may be able to perform some gymnastics and allow the OS to swap your terabyte+ dataset around inside your 64GB of RAM, but the algorithms are now going to take forever to complete as you thrash your swap constantly while the training algorithm is running.

tl;dr - a terabyte dataset in the machine learning context may very well need that much RAM plus some overhead in terms of memory available to be able to train a model on the dataset.

Re: Does my data fit in RAM?

#22

Post by bluedino » Wed, Feb 12, 2020, 7:27 PM UTC

$2,000 each for 128GB LRDIMMs, 48 of those will be $100,000 and then you'll need another $20,000 to buy the rest of the server it goes in.

Re: Does my data fit in RAM?

#23

Post by justizin » Wed, Feb 12, 2020, 7:36 PM UTC

ah and it is so fucking lovely to be hired by a company which has been running everything on a single machine for 2 years.

Re: Does my data fit in RAM?

#24

Post by lunixbochs » Wed, Feb 12, 2020, 7:39 PM UTC

Looks like I'm all set with my -12PB dataset fitting in ram.

Re: Does my data fit in RAM?

#25

Post by qaq » Wed, Feb 12, 2020, 7:40 PM UTC

ghc wrote: post #18

Fine print: But it might cost you $300K CapEx or $800K/yr OpEx. Hope you have a budget!

it will cost you 100K CapEx and will save you many 100s of Ks in opEx and untold amount of money in development cost.

Re: Does my data fit in RAM?

#26

Post by [deleted] » Wed, Feb 12, 2020, 7:41 PM UTC

thedance wrote: post #16

chmod775 wrote: post #6

Earlier quoted context omitted.

A single terabyte is a few magnitudes from what you need big-data-anything for. You could probably work with that just fine on your average 64GB ram desktop with an SSD.

A small computer with 1 SSD will take at least 10-20 minutes to make a pass over 1TB of data, if everything is perfectly pipelined.

[deleted]

Re: Does my data fit in RAM?

#27

Post by jagged-chisel » Wed, Feb 12, 2020, 7:41 PM UTC

lunixbochs wrote: post #24

Looks like I'm all set with my -12PB dataset fitting in ram.

Is that “negative twelve” or did you mean ~?

Re: Does my data fit in RAM?

#28

Post by randyzwitch » Wed, Feb 12, 2020, 7:42 PM UTC

While pithy, the implication that you are going to process 12 TB of data in RAM using mostly single-threaded tools doesn't reflect reality.

Re: Does my data fit in RAM?

#29

Post by lotyrin » Wed, Feb 12, 2020, 7:46 PM UTC

randyzwitch wrote: post #28

While pithy, the implication that you are going to process 12 TB of data in RAM using mostly single-threaded tools doesn't reflect reality.

Where exactly does that implication come from? Are you from a world where you need a map reduce framework + cluster to have parallelism of any kind?

Re: Does my data fit in RAM?

#30

Post by lsb » Wed, Feb 12, 2020, 7:49 PM UTC

ramraj07 wrote: post #2

Legit question: I have a dataset that's a terabyte in size spread over multiple tables, but my queries often involve complex self joins and filters; for various reasons, I'd prefer to be able to write my queries in SQL (or spark code) because it's the most expressive system I've seen. What tool should I use it to load this dataset on RAM and run these queries?

You can get a box with 4TB of ram on EC2 for $4/hr spot, so copy your data into /dev/shm and go hog wild.

For lots of databases, most of their time is spent locking and copying data around, so depending on your workload you might getsignificant speedups in Pandas/Numpy if it's just you doing manipulations, and there are multicore just-in-time compilers for lots of Pandas/Numpy operations (like Numba/Dask/etc).

If you have lots of weird merging criteria and want the flexibility of SQL I'd say use a modern Postgresql with multicore selects on that 4TB box.