Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Your data fits in RAM

yourdatafitsinram.com

Post reply on HN
Print view View original

31–40 of 230 posts

Re: Your data fits in RAM

#31

Post by chao- »

I love it. I was just doing some Fermi estimates for a friend on the data for a project he has in the pipeline. I was curious whether or not it would be cost efficient for his project's budget to go with NVMe SSDs or have to stick with traditional SATA ones, and turns out it doesn't even matter (for now) because at least the first three months of data will fit in 256GB of RAM, even allowing for a 2.5x factor stemming from some (estimated) inefficient storage or data structure use in a scripting language like Ruby or Python.

Edit: And after those first three months he'll know more about the use and performance demands of the project and will be able to make far more accurate decisions about storage categories.

Re: Your data fits in RAM

#32

Post by jacquesm »

gonzo41 wrote: post #27
jacquesm wrote: post #23

Earlier quoted context omitted.

10 servers with 100G will use a lot more power and will require distribution of your algorithm right along with your dataset, so instead of 10 server you will probably end up with a pretty high multiple of 10.

Isn't this what the cloud is for? Surely you just rent an instance of the computing power you need for an hour or two then upload the data, a script and wait.

That really depends on your usecase. Not all analysis is 'one-shot' and not all businesses are free to upload their data into 'the cloud'.

Re: Your data fits in RAM

#33

Post by zimbatm »

mtabini wrote: post #24
FooBarWidget wrote: post #19

Earlier quoted context omitted.

Complexity, sure. But cost? I thought a single 1 TB RAM server is more expensive than 10x 100 GB RAM servers. And many people don't want to deal with physical hardware. Dealing with physical hardware increases operational complexity too. They want to rent a virtual/cloud server. Which provider allows you to rent a virtual server with 1 TB RAM?

Cost follows complexity, even if it's not always immediately obvious. A 1TB RAM server is more expensive than 10x 100GB RAM servers, but the hardware cost is often small compared to the business and technical cost of getting a solution to scale across a cluster. Of course, generalizations are always dangerous—the take-home point here is perhaps that before going to a cluster because “that's the way big data is handle…

> generalizations are always dangerous

I've got to remember this one

Re: Your data fits in RAM

#34

Post by jkot »

Outside of scale-out, scale-up there is also solution: scale-in. Optimize your memory usage, so your data occupies less space.

I work on something like that.

Re: Your data fits in RAM

#35

Post by Leynos »

bshimmin wrote: post #16
jacquesm wrote: post #6

Earlier quoted context omitted.

No, the point is that usually fitting things in RAM lowers the budget. So it's well worth doing proper analysis on whether or not you can (a) fit all your data in RAM and (b) if a cluster of machines does not become it's own reason for existence. Replacing a large number of nodes with a single machine with a lot of RAM is usually a cost savings measure rather than a larger expense (and it saves power too!), and due t…

What does 6TB of RAM go for these days?

If you go by the Dell site linked from the article, a PowerEdge maxed out with 6TB RAM (4 sockets * 24 LRDIMMs * 64 GB) will set you back US$444,000 (or thereabouts). They also list 3.2 TB NVMe PCIe cards for US$11,000 each.

Re: Your data fits in RAM

#36

Post by Smerity »

It's probably worth extending "Your data fits in RAM" to "Your data doesn't fit in RAM, but it does fit on an SSD". So many problems will still work with quite reasonable performance when using an SSD instead. By using a single machine with an array of SSDs, you also avoid the complexity and overhead of distributed systems.

My favourite realization of this: Frank McSherry shows how simplicity and a few optimisations can win out on graph analysis in his COST work. In his first post[1], he shows how to beat a cluster of machines with a laptop. In his second post[2], he applies even more optimizations, both space and speed, to process the largest publicly available graph dataset - terabyte sized with over a hundred billion edges - all on his laptop's SSD.

[1]: http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...

[2]: http://www.frankmcsherry.org/graph/scalability/cost/2015/02/...

Re: Your data fits in RAM

#37

Post by jacquesm »

sytelus wrote: post #30

Looks like 1.5TB RAM with 15 cores costs $50K. But it shouldn't be just about RAM. The problems I'm working on requires 250 cores on similar amount of data. If there was an option to get say 150 cores with 2TB RAM, things would fly for sure.

Another 4 to 6 years and that should be a reality.

Re: Your data fits in RAM

#39

Post by e12e »

jacquesm wrote: post #22
jordanthoms wrote: post #15

Earlier quoted context omitted.

There is no point deploying a heavy, complex (and usually pretty slow due to the overheads involved) distributed database, when you could just buy a server with xTB ram, load any sql database on it, and run your queries in a fraction of the time. If your data is so large that it can't fit in the RAM of a single machine, then distributed databases make more sense (since loading data off disk is very slow, modulo SSD).

For some problems SQL would already be way too much overhead.

Could you give a concrete example?

If your working set is small, say 1TB -- and so fits in RAM -- for what kind of problems would using SQL be so much of an overhead that you need a different approach? And what would that approach be?

I suppose you could have a massive set of linear equations that you might be able to fit into 1TB of RAM, but would be difficult to work with as tables in Postgres?

Re: Your data fits in RAM

#40

Post by cornellphds »

In my opinion the correct answer is 255Gb. (i.e. AWS r3X8 High Memory instances ).

While one can purchase servers with larger memory most likely you will run into limitation on number of cores. Also note that there is at least some overhead in processing data, so you would need at least 2X the size of raw data.

Finally while its a good thing to tweet, joke about and make fun of buzzword while trying to appear smart. The reality is that purchasing such servers (> 255 Gb RAM) is costly process. Further you would ideally need two of them to remove single point of failure. it is likely that the job is batch and while it might take a terabyte of RAM you only need to run it once a week, in all these cases you are much better off relying on a distributed system where each node has very large memory, and the task can easily split. Just because you have cluster does not mean that each node has to be a small instance (4 processors ~16 Gb RAM).

Post reply on HN