91–100 of 167 posts
Legit question: I have a dataset that's a terabyte in size spread over multiple tables, but my queries often involve complex self joins and filters; for various reasons, I'd prefer to be able to write my queries in SQL (or spark code) because it's the most expressive system I've seen. What tool should I use it to load this dataset on RAM and run these queries?
Earlier quoted context omitted.
This is what most people in my org do, this is orders of magnitude slower than running queries on the same dataset on redshift with optimized presorting and distribution. Redshift doesn't scale for tens or hundreds of parallel users though, so looking for options
I'm about 75% joking: restore a new cluster from a snapshot for horizontal scaling. They did launch a feature along those lines, because my joke suggestion probably doesn't scale organizationally: take a look at Concurrency Scaling. This is also a fundamental feature of Snowflake with the separation of compute and storage.
The problem is DRAM price hasn't drop one bit. The lowest price floor per GB has been similar for the past decade. Roughly at $2.8/GB in 2012, 2016, and 2019. And all DRAM manufacturers has been enjoying a very profitable period. And yet our Data size continue to grow. We can fit more Data inside memory not because DRAM capacity has increase, but we are simply increasing memory channels.
Everyone knows that DRAM prices have been in a collapse since early this year, but last week DRAM prices hit a historic low point on the spot market. Based on data the Memory Guy collected from spot-price source InSpectrum, the lowest spot price per gigabyte for branded DRAM reached $2.59 last week. https://thememoryguy.com/dram-prices-hit-historic-low/ You've selected out the low points on the graph: 2012, 2016, and…
And it is predicted to climb back up this year, due to manufacturers dropping wafer starts at a point in time when a large launch of next-gen consoles is drastically increasing consumption.
Nope, definitely no collusion there. /s
1. If you have to ask, then either it doesn't now, or it doesn't sometimes. So assume it doesn't.
2. If you can use a cluster, then maybe.
3. In some senses, it doesn't matter. How so? Reading and writing from RAM is very slow, latency-wise, for today's processors. If I can bend the truth a little, it's a bit like a fast SSD. So, if you can up the bandwidth to disk enough, it becomes kind of comparable. Well, if you can use 16 PCIe 4.0 lanes, it's roughly 24 GB/sec effective bandwidth, which is roughly half of your memory bandwidth. Now it's true that in real-life systems it's usually just 4 lanes, but it's very doable to change that with a nice card.
4. DIMM-form-factor non-volatile memory may increase memory sizes much more.
Earlier quoted context omitted.
You know what else costs? Humongous amount of servers to run silly stuff to orchestrate other silly stuff to autoscale yet else silly stuff to do stuff on your stuff that could fit into memory and be processed on a single server (+ backup, of course). Add to that small army of people, because, you know, you need specialists of variety of professions just to debug all integration issues between all those components th…
I really don't understand comments like this. Yes your company's data may fit in RAM. But does every intermediate data set also fit in RAM ? Because I've also worked at a bank and we had thousands of complex ETLs often needing tens to hundreds of intermediate sets along the way. There is no AWS server that can keep all of that inflight at one time. And what about your Data Analysts/Scientists. Can all of their random…
There is a trend, when the application is inefficient, to spend huge amount of resources on scaling it instead of making the application more efficient.
Earlier quoted context omitted.
You know what else costs? Humongous amount of servers to run silly stuff to orchestrate other silly stuff to autoscale yet else silly stuff to do stuff on your stuff that could fit into memory and be processed on a single server (+ backup, of course). Add to that small army of people, because, you know, you need specialists of variety of professions just to debug all integration issues between all those components th…
I really don't understand comments like this. Yes your company's data may fit in RAM. But does every intermediate data set also fit in RAM ? Because I've also worked at a bank and we had thousands of complex ETLs often needing tens to hundreds of intermediate sets along the way. There is no AWS server that can keep all of that inflight at one time. And what about your Data Analysts/Scientists. Can all of their random…
$100K has always been "cheap" for a "business computer" and today you can get more computer for that money than ever.
$100K of hardware (per year or so) is small-fry compared to almost every other R&D industry out there. Just compare with the cost of debuggers, oscilloscopes and EMC labs for electronic engineers.
IIRC, the largest IBM z15 can have up to 40 TiB of RAM. The website needs updating.
You can fit 48 TB on a HPE MC990 X though I'm pretty sure that's got one of those NUMA architectures that SGI had with the UV 3000 or whatever. I remember jokingly telling my team to spend the millions of dollars we did expanding our clusters with one of these and just processing in RAM. I honestly don't think I did the analysis to make sure it would be actually better. It was 'jokingly' because we couldn't afford th…
You didn't have to have all the capital. They'd rent you the machine on a long lease and you could say you'd bought a million dollar computer.
Sun tried to get into similar markets but it was always a tougher deal on minis and micros.
IIRC, the largest IBM z15 can have up to 40 TiB of RAM. The website needs updating.