Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Does my data fit in RAM?

yourdatafitsinram.net

Post reply on HN
Print view View original

111–120 of 167 posts

Re: Does my data fit in RAM?

#111

Post by Aeolun »

threeseed wrote: post #103
andyjpb wrote: post #96

Earlier quoted context omitted.

Buy them a machine each. $100K has always been "cheap" for a "business computer" and today you can get more computer for that money than ever. $100K of hardware (per year or so) is small-fry compared to almost every other R&D industry out there. Just compare with the cost of debuggers, oscilloscopes and EMC labs for electronic engineers.

My company has over 400 Data Scientists and 1000s of Data Analysts. Buy them a machine each at a cost of $40-60 billion ? Or would it make more sense to buy one Spark cluster and then share the resources at a fraction of the cost.

I don’t get your numbers, getting one for 400 people is 40M, for thousands it may be 100-999M.

Still expensive, but much less than 40 billion.

Re: Does my data fit in RAM?

#112

Post by Aeolun »

lotyrin wrote: post #53
randyzwitch wrote: post #52

Earlier quoted context omitted.

No, but I frequently see people implying that you can do your data science in Python and R as long as you can fit the data in RAM. As you mention, it's not RAM that's the limiting factor for larger data volumes, it's finding tools that exploit parallelism.

I have done plenty of parallel work in python and R, so I'm still not sure what you mean.

If you are using python and R, can’t you make your own parallelism?

Re: Does my data fit in RAM?

#113

Post by ghc »

lmilcin wrote: post #19
ghc wrote: post #18

Fine print: But it might cost you $300K CapEx or $800K/yr OpEx. Hope you have a budget!

You know what else costs? Humongous amount of servers to run silly stuff to orchestrate other silly stuff to autoscale yet else silly stuff to do stuff on your stuff that could fit into memory and be processed on a single server (+ backup, of course). Add to that small army of people, because, you know, you need specialists of variety of professions just to debug all integration issues between all those components th…

There's a world outside of web apps and SV tech companies. There's a lot of big datasets out there, most of which never hit the cloud at all.

Story time: I worked on one project where a single (large) building's internal sensor data (HVAC, motion, etc. 100k sensors) would fill a 40TB array every year. They had a 20 year retention policy. So Dell would just add a new server + array every year.

I worked with another company that had 2000 oracle servers in some sort of franken-cluster config. Reports took 1 week to run and they had pricing data for their industry (they were a transaction middleman) for almost 40 years. I can't even guess the data size because nobody could figure it out.

This is not a FAANG problem. This is an everage SME to large enterprise problem. Yeah, startups don't have much data. Most companies out there aren't startups.

By the way, memory isn't the only solution. In the past 15 years, I've rarely worked on projects where everything was in memory. Disks work just fine with good database technology.

Re: Does my data fit in RAM?

#114

Post by ksec »

keanzu wrote: post #57
ksec wrote: post #43

The problem is DRAM price hasn't drop one bit. The lowest price floor per GB has been similar for the past decade. Roughly at $2.8/GB in 2012, 2016, and 2019. And all DRAM manufacturers has been enjoying a very profitable period. And yet our Data size continue to grow. We can fit more Data inside memory not because DRAM capacity has increase, but we are simply increasing memory channels.

Everyone knows that DRAM prices have been in a collapse since early this year, but last week DRAM prices hit a historic low point on the spot market. Based on data the Memory Guy collected from spot-price source InSpectrum, the lowest spot price per gigabyte for branded DRAM reached $2.59 last week. https://thememoryguy.com/dram-prices-hit-historic-low/ You've selected out the low points on the graph: 2012, 2016, and…

>the lowest spot price per gigabyte for branded DRAM reached $2.59 last week.

It would be better to reference this as quoted from the article which was written in November 2019. So not really last week

>most of the time DRAM has not been available at these prices.

I did said price floor.

Re: Does my data fit in RAM?

#115

Post by vasili111 »

skwb wrote: post #61

I remember back in the early 2010's that a large selling point of SAS (besides the ridiculous point that R/python were freeware and therefore cannot be trusted on important projects ) was that it can chew through large data sets that perhaps couldn't be moved into RAM (but maybe it takes a week or whatever....). This was a fairly salient point, and remember circa 2012/2013 struggling to fit large bioinfomatics data i…

I have not tested it myself but now there is a disk.frame ( https://github.com/xiaodaigh/disk.frame ). As far as I know, other options exist too.

Re: Does my data fit in RAM?

#116

Post by sevensor »

yongjik wrote: post #105

Even more importantly, does your data have to fit in RAM? There are tons of problems that need to process large data, but touch each item just once (or a few times). You can go a really long way by storing them in disk (or some cloud storage like S3) and writing a script to scan through them. I know, pretty obvious, but somehow escapes many devs.

There's also the "not all memory is RAM" trick: plan ahead with enough swap to fit all the data you intend to process, and just pretend that you have enough RAM. Let the virtual memory subsystem worry about whether or not it fits in RAM. Whether this works well or horribly depends on your data layout and access patterns.

Re: Does my data fit in RAM?

#117

Post by threeseed »

Aeolun wrote: post #111
threeseed wrote: post #103

Earlier quoted context omitted.

My company has over 400 Data Scientists and 1000s of Data Analysts. Buy them a machine each at a cost of $40-60 billion ? Or would it make more sense to buy one Spark cluster and then share the resources at a fraction of the cost.

I don’t get your numbers, getting one for 400 people is 40M, for thousands it may be 100-999M. Still expensive, but much less than 40 billion.

He said $100k for each user but it's a dumb idea anyway.

We have a Spark cluster which supports all of those users for $10-$20k a month.

Re: Does my data fit in RAM?

#118

Post by efficax »

chmod775 wrote: post #3
ramraj07 wrote: post #2

Legit question: I have a dataset that's a terabyte in size spread over multiple tables, but my queries often involve complex self joins and filters; for various reasons, I'd prefer to be able to write my queries in SQL (or spark code) because it's the most expressive system I've seen. What tool should I use it to load this dataset on RAM and run these queries?

Most DB engines will use what RAM is available and even if they don't, your OS's page cache will make sure stuff is fast anyways. > What tool should I use it to load this dataset on RAM and run these queries? The question should really be: What tool should I use to make this fast? Postgres can be pretty fast when used correctly and you can make your data fit.

Both mysql and pgsql bypass the page cache if they can and maintain their own page caches. You have to do this, otherwise you’re double caching! That is, you’d have your own page cache, which you need to manage calls to read() and to know when to flush pages, while the OS would also have the same pages in its own cache.

(mongodb I believe uses direct mmap access instead of a pagecache, and lmdb does this as well)

Re: Does my data fit in RAM?

#119

Post by bkanber »

ramraj07 wrote: post #72
typon wrote: post #32

Earlier quoted context omitted.

in memory sqlite is pretty fast

I tried this once, creating an index on a 20 billion row table isn't fast :/

I haven't tried this and don't know if it would work -- but depending on the shape of your data and queries, you might not need certain indices. That is, for some workloads (especially if you're thinking of spot instances), it might be overall faster to skip the indexing and allow the query to do a full table scan. It sounds like maybe you never tried the query without the index, so I'm curious to know if there's any weight behind this theory.

Re: Does my data fit in RAM?

#120

Post by bkanber »

Aeolun wrote: post #110
ghc wrote: post #18

Fine print: But it might cost you $300K CapEx or $800K/yr OpEx. Hope you have a budget!

Meh, that’s like half the budget of the databases on our dev environment. We have way too many fucking copies of our DB.

I prefer using mocked data in dev. Smaller dataset, no possibility of PII leaks.
Post reply on HN