61–70 of 167 posts
This was a fairly salient point, and remember circa 2012/2013 struggling to fit large bioinfomatics data into an older iMac with base R.
Sure, my data fits in RAM if you take one of those example machines, which we have, and which are currently hosting dozens of production VMs, and instead dedicate the whole thing to my database. I'd love that, but it's never going to happen.
It's still way more efficient to use it for that database, than running a dozen of hadoop nodes as VMs.
Earlier quoted context omitted.
No, but I frequently see people implying that you can do your data science in Python and R as long as you can fit the data in RAM. As you mention, it's not RAM that's the limiting factor for larger data volumes, it's finding tools that exploit parallelism.
I have done plenty of parallel work in python and R, so I'm still not sure what you mean.
$2,000 each for 128GB LRDIMMs, 48 of those will be $100,000 and then you'll need another $20,000 to buy the rest of the server it goes in.
$2,000 each for 128GB LRDIMMs, 48 of those will be $100,000 and then you'll need another $20,000 to buy the rest of the server it goes in.
If the result is faster than a $250K Hadoop cluster then you're still ahead.
I’m no fan of distributed systems that sit idle at 2% utilisation when a single node would do : BUT, reducing it down to “cost” and “does it fit in RAM” is way too reductive
I would definitely imagine that most workloads rarely need more than a few to a few hundred TB in memory, since you may have petabytes of data but you probably touch very little of it.
Earlier quoted context omitted.
A small computer with 1 SSD will take at least 10-20 minutes to make a pass over 1TB of data, if everything is perfectly pipelined.
Samsung claims their 970 Pro NVMe can read 3.5GB/s sequentially. That's about 300 seconds or 5 minutes per TB.
Legit question: I have a dataset that's a terabyte in size spread over multiple tables, but my queries often involve complex self joins and filters; for various reasons, I'd prefer to be able to write my queries in SQL (or spark code) because it's the most expressive system I've seen. What tool should I use it to load this dataset on RAM and run these queries?
in memory sqlite is pretty fast
Legit question: I have a dataset that's a terabyte in size spread over multiple tables, but my queries often involve complex self joins and filters; for various reasons, I'd prefer to be able to write my queries in SQL (or spark code) because it's the most expressive system I've seen. What tool should I use it to load this dataset on RAM and run these queries?
Most DB engines will use what RAM is available and even if they don't, your OS's page cache will make sure stuff is fast anyways. > What tool should I use it to load this dataset on RAM and run these queries? The question should really be: What tool should I use to make this fast? Postgres can be pretty fast when used correctly and you can make your data fit.
Earlier quoted context omitted.
If the result is faster than a $250K Hadoop cluster then you're still ahead.
Yeah, except Hadoop provides redundancy, easier backups, turnkey solutions for governance and compliance and easier scaling I’m no fan of distributed systems that sit idle at 2% utilisation when a single node would do : BUT, reducing it down to “cost” and “does it fit in RAM” is way too reductive
These days I’m firmly of the opinion that if you can make it run on a single machine, you absolutely should, and you don’t get more machines until you can prove you can fully utilise one machine.