Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Does my data fit in RAM?

yourdatafitsinram.net

Post reply on HN
Print view View original

131–140 of 167 posts

Re: Does my data fit in RAM?

#131

Post by beached_whale »

sevensor wrote: post #116
yongjik wrote: post #105

Even more importantly, does your data have to fit in RAM? There are tons of problems that need to process large data, but touch each item just once (or a few times). You can go a really long way by storing them in disk (or some cloud storage like S3) and writing a script to scan through them. I know, pretty obvious, but somehow escapes many devs.

There's also the "not all memory is RAM" trick: plan ahead with enough swap to fit all the data you intend to process, and just pretend that you have enough RAM. Let the virtual memory subsystem worry about whether or not it fits in RAM. Whether this works well or horribly depends on your data layout and access patterns.

Don't even need to do that. Just mmap it and the virtual memory system will handle it.

Re: Does my data fit in RAM?

#132

Post by FpUser »

quersive wrote: post #101
ramraj07 wrote: post #2

Legit question: I have a dataset that's a terabyte in size spread over multiple tables, but my queries often involve complex self joins and filters; for various reasons, I'd prefer to be able to write my queries in SQL (or spark code) because it's the most expressive system I've seen. What tool should I use it to load this dataset on RAM and run these queries?

[Disclaimer: I worked on BigQuery a couple lives ago] I'd give Google BigQuery a shot. Should work fast [seconds] and scale seamlessly to [much] larger datasets and [many] more users. For a 1 TB dataset, I have a hard time imagining crafting a slow query. Maybe something outlandish like 1000[00?] joins. They also have an in-memory "BI Engine" offering, alas limited to 50GB max. On premise, there is Tableau Data Engin…

"My contact is in my profile"

This makes for a good laugh: Queries for the masses. Contact: f'info@${user}.com'

Re: Does my data fit in RAM?

#133

Post by EdwardDiego »

Telling me that my (example) 24TB data set fits into RAM, because it fits into 24TB on an AWS instance designed for SAP that's so expensive that it's price on inquiry, isn't overly helpful.

May I suggest that a) they consider applications need RAM too and b) that if the price is POI, it's probably better to mention "But you know, 188 m5.8xlarges might be cheaper"

Re: Does my data fit in RAM?

#134

Post by EdwardDiego »

yongjik wrote: post #105

Even more importantly, does your data have to fit in RAM? There are tons of problems that need to process large data, but touch each item just once (or a few times). You can go a really long way by storing them in disk (or some cloud storage like S3) and writing a script to scan through them. I know, pretty obvious, but somehow escapes many devs.

I've spent a lot of time writing Spark code, and its ability to store data in a column oriented format in RAM is the only reason why - disk is goddamned slow.

As soon as you're touching it more than once, sticking it in RAM upon reading makes everything much faster.

Re: Does my data fit in RAM?

#135

Post by adrianN »

lmilcin wrote: post #19
ghc wrote: post #18

Fine print: But it might cost you $300K CapEx or $800K/yr OpEx. Hope you have a budget!

You know what else costs? Humongous amount of servers to run silly stuff to orchestrate other silly stuff to autoscale yet else silly stuff to do stuff on your stuff that could fit into memory and be processed on a single server (+ backup, of course). Add to that small army of people, because, you know, you need specialists of variety of professions just to debug all integration issues between all those components th…

There are a number of cloud database solutions that are very easy to manage and not all that expensive. For example I work for Snowflake and our product doesn't need a small army of people to babysit it.

I mean, I also prefer doing things on a single machine, but if that machine gets expensive enough, or writing a program that can actually use all that power gets too difficult, why not switch to a cloud database?

Re: Does my data fit in RAM?

#136

Post by makapuf »

imron wrote: post #42
samspenc wrote: post #39

Earlier quoted context omitted.

Could you share the link to the original Twitter thread? I guess the link was changed on HN before some of us got here.

On the linked page, it says: "Inspired by this tweet" which links to https://twitter.com/garybernhardt/status/600783770925420546

Note that this is 2015, on the heyday of big data.

Re: Does my data fit in RAM?

#137

Post by bscphil »

mr_toad wrote: post #109
ksec wrote: post #43

The problem is DRAM price hasn't drop one bit. The lowest price floor per GB has been similar for the past decade. Roughly at $2.8/GB in 2012, 2016, and 2019. And all DRAM manufacturers has been enjoying a very profitable period. And yet our Data size continue to grow. We can fit more Data inside memory not because DRAM capacity has increase, but we are simply increasing memory channels.

If it was just a matter of adding more memory people would. A few thousand, or tens of thousands of dollars aren’t much to organisations that have that much data. The trouble is that there are limits to how much memory you can fit on a motherboard.

And how much you can afford. My partner's scientific research group recently upgraded their shared server used for analysis ... to one with 32 GB of RAM and 8 TB of storage. I think they could have done better, personally, but it's telling how thin budgets are stretched in a lot of real world cases.

Re: Does my data fit in RAM?

#138

Post by sriku »

The question of whether my storage mechanism fits budget is more relevant, no? It costs less than 4cents per hour to store 1TB on S3 but an X1 with 1TB of RAM costs $13 or so per hour on Aws. So the issue is whether what you pay for is worth the result you're computing.

Re: Does my data fit in RAM?

#139

Post by LeoPanthera »

Jupe wrote: post #127

Some pedantry... Raw RAM space is not the issue... it's indexing and structure of the data that makes it process-able. If you just need to spin through the data once, there's no need to even put all of it in RAM - just stream it off disk and process it sequentially. If you need to join the data, filter, index, query it, you'll need a lot more RAM than your actual data. Database engines have their own overhead (system…

I often use RAM disks for production services. I'm sure that I shouldn't. It feels very lazy, and I know that I'm probably missing out on the various "right" ways to do things.

But it works so well and it's so easy. It's a really difficult habit to kick.

Obligatory joke: "Oh boy, virtual memory! Now I can have a really big RAM disk!"

Re: Does my data fit in RAM?

#140

Post by malux85 »

FridgeSeal wrote: post #129
malux85 wrote: post #76

Earlier quoted context omitted.

Waiting for a machine to be “fully utilised” before scaling just shows your lack of experience at systems engineering. Do you know how quickly disks fail if you force them at 100% utilisation, 24/7? Then what happens when this system dies? How much downtime do you have because you have to replace then hardware then get your hundreds of gigabyte dataset back in RAM and hot again? I’ve worked as a lead developer at com…

I don't mean "let your production systems spool up to point where you're maxing out a single machine" - that would be exceedingly silly. I mean "when you've proven that the application you've written can fully, or near fully utilise the available power on a single machine, and that when running production-grade workloads, actually does so, then you may scale to additional machines. What this means is not getting a 9-…

And what about redundancy in case of node failure?
Post reply on HN