Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Does my data fit in RAM?

yourdatafitsinram.net

Post reply on HN
Print view View original

71–80 of 167 posts

Re: Does my data fit in RAM?

#71

Post by ramraj07 »

lsb wrote: post #30
ramraj07 wrote: post #2

Legit question: I have a dataset that's a terabyte in size spread over multiple tables, but my queries often involve complex self joins and filters; for various reasons, I'd prefer to be able to write my queries in SQL (or spark code) because it's the most expressive system I've seen. What tool should I use it to load this dataset on RAM and run these queries?

You can get a box with 4TB of ram on EC2 for $4/hr spot, so copy your data into /dev/shm and go hog wild. For lots of databases, most of their time is spent locking and copying data around, so depending on your workload you might getsignificant speedups in Pandas/Numpy if it's just you doing manipulations, and there are multicore just-in-time compilers for lots of Pandas/Numpy operations (like Numba/Dask/etc). If you…

Currently I have it loaded on redshift with as much optimization as possible, and the queries are far more analytical than end-user like (often having to self join on the same dataset). This works okay, but doesn't scale with more than a handful users at a time. I'll probably run some tests with the postgres suggestion but curious if this is still a better alternative or not

Re: Does my data fit in RAM?

#72

Post by ramraj07 »

typon wrote: post #32
ramraj07 wrote: post #2

Legit question: I have a dataset that's a terabyte in size spread over multiple tables, but my queries often involve complex self joins and filters; for various reasons, I'd prefer to be able to write my queries in SQL (or spark code) because it's the most expressive system I've seen. What tool should I use it to load this dataset on RAM and run these queries?

in memory sqlite is pretty fast

I tried this once, creating an index on a 20 billion row table isn't fast :/

Re: Does my data fit in RAM?

#73

Post by H8crilA »

One of the best things you can often do for latency/throughput improvement is to just mlock() the data. Yes, it fits. Get a more expensive machine and 5x the throughput in a single day with a config change.

Re: Does my data fit in RAM?

#74

Post by threeseed »

wmf wrote: post #36
bluedino wrote: post #22

$2,000 each for 128GB LRDIMMs, 48 of those will be $100,000 and then you'll need another $20,000 to buy the rest of the server it goes in.

If the result is faster than a $250K Hadoop cluster then you're still ahead.

But that Hadoop cluster will also work just fine if the data set does not fit in RAM. And I've never met a data set that didn't expand over time. And with Spark you get a robust, scalable and industry standard way of distributing work amongst the nodes.

Also did you know that Hadoop is open source. So that $250K is purely for hardware.

Re: Does my data fit in RAM?

#75

Post by threeseed »

ramraj07 wrote: post #2

Legit question: I have a dataset that's a terabyte in size spread over multiple tables, but my queries often involve complex self joins and filters; for various reasons, I'd prefer to be able to write my queries in SQL (or spark code) because it's the most expressive system I've seen. What tool should I use it to load this dataset on RAM and run these queries?

You can just write Spark SQL, set the executor memory to whatever the machine is and not worry about whether it's in RAM or not.

Spark will naturally use RAM first and then disk as needed.

Re: Does my data fit in RAM?

#76

Post by malux85 »

FridgeSeal wrote: post #70
malux85 wrote: post #65

Earlier quoted context omitted.

Yeah, except Hadoop provides redundancy, easier backups, turnkey solutions for governance and compliance and easier scaling I’m no fan of distributed systems that sit idle at 2% utilisation when a single node would do : BUT, reducing it down to “cost” and “does it fit in RAM” is way too reductive

And reducing it to a single machine means I have an order of magnitude less time spent on setting it up, maintaining it, adapting all my code, debugging, etc. These days I’m firmly of the opinion that if you can make it run on a single machine, you absolutely should, and you don’t get more machines until you can prove you can fully utilise one machine.

Waiting for a machine to be “fully utilised” before scaling just shows your lack of experience at systems engineering.

Do you know how quickly disks fail if you force them at 100% utilisation, 24/7?

Then what happens when this system dies? How much downtime do you have because you have to replace then hardware then get your hundreds of gigabyte dataset back in RAM and hot again?

I’ve worked as a lead developer at companies where I’ve been personally responsible for hundreds of thousands of machines, and running a node to 100% and THEN thinking about scaling is short sighted and stupid

Re: Does my data fit in RAM?

#77

Post by threeseed »

lmilcin wrote: post #19
ghc wrote: post #18

Fine print: But it might cost you $300K CapEx or $800K/yr OpEx. Hope you have a budget!

You know what else costs? Humongous amount of servers to run silly stuff to orchestrate other silly stuff to autoscale yet else silly stuff to do stuff on your stuff that could fit into memory and be processed on a single server (+ backup, of course). Add to that small army of people, because, you know, you need specialists of variety of professions just to debug all integration issues between all those components th…

I really don't understand comments like this.

Yes your company's data may fit in RAM. But does every intermediate data set also fit in RAM ? Because I've also worked at a bank and we had thousands of complex ETLs often needing tens to hundreds of intermediate sets along the way. There is no AWS server that can keep all of that inflight at one time.

And what about your Data Analysts/Scientists. Can all of their random data sets reside in RAM on the same server too ?

Re: Does my data fit in RAM?

#78

Post by dmos62 »

FridgeSeal wrote: post #70
malux85 wrote: post #65

Earlier quoted context omitted.

Yeah, except Hadoop provides redundancy, easier backups, turnkey solutions for governance and compliance and easier scaling I’m no fan of distributed systems that sit idle at 2% utilisation when a single node would do : BUT, reducing it down to “cost” and “does it fit in RAM” is way too reductive

And reducing it to a single machine means I have an order of magnitude less time spent on setting it up, maintaining it, adapting all my code, debugging, etc. These days I’m firmly of the opinion that if you can make it run on a single machine, you absolutely should, and you don’t get more machines until you can prove you can fully utilise one machine.

Account for parent's concerns for redundancy, backups, scalability.

Re: Does my data fit in RAM?

#80

Post by cldellow »

dijit wrote: post #58
lsb wrote: post #30

Earlier quoted context omitted.

You can get a box with 4TB of ram on EC2 for $4/hr spot, so copy your data into /dev/shm and go hog wild. For lots of databases, most of their time is spent locking and copying data around, so depending on your workload you might getsignificant speedups in Pandas/Numpy if it's just you doing manipulations, and there are multicore just-in-time compilers for lots of Pandas/Numpy operations (like Numba/Dask/etc). If you…

How long does it take to copy your data in? And what’s the bandwidth cost involved? People like to talk about the elasticity or compute, but startup is not free (or even cheap in most cases).

If your data is in S3, my experience is that you can push ~20-40MB/core/sec on most instances.

OP is probably talking about an x1e.32xlarge. According to Daniel Vassalo's S3 benchmark [1], it can do about 2.7GB/sec.

So your 4TB DB might take ~30min to fetch.

Bandwidth is free, you'd pay $2 for the 30 min of compute, and some fractions of pennies for the few hundred S3 requests.

[1]: https://github.com/dvassallo/s3-benchmark

Post reply on HN