Your data fits in RAM

No, the point is that usually fitting things in RAM lowers the budget. So it's well worth doing proper analysis on whether or not you can (a) fit all your data in RAM and (b) if a cluster of machines does not become it's own reason for existence.

Replacing a large number of nodes with a single machine with a lot of RAM is usually a cost savings measure rather than a larger expense (and it saves power too!), and due to a lack of communications overhead and exploitation of the fact that you now have access to all the data in one go you may very well find that your algorithms run much faster.

A distributed solution should be a means of last resort.

Re: Your data fits in RAM

#7

Post by lukegb » Thu, May 21, 2015, 11:10 AM UTC

josephmx wrote: post #5

I sincerely hope nobody is using a tool like this to decide which enterprise servers to buy...

Me too. The links are mostly to back up my claim rather than as a suggestion of servers to buy (or I'd have found some affiliate links!)

Re: Your data fits in RAM

#8

Post by mosselman » Thu, May 21, 2015, 11:10 AM UTC

lukegb wrote: post #2

Inspired by https://twitter.com/garybernhardt/status/600783770925420546

Can someone explain in a bit more detail what this is about? Is the 'joke' that running data computation in RAM is faster than what? From disk?

Re: Your data fits in RAM

#9

Post by jakozaur » Thu, May 21, 2015, 11:11 AM UTC

More accurate title would be fit in RAM of single machine.

Maybe some bonus category:

0. Spreadsheet is all you need.

1. Python script is good enough.

2. Java/Scala is way to go.

3. Need to manage memory (gc doesn't cut), some custom organization.

4. Actually needs a cluster.

Re: Your data fits in RAM

#10

Post by nodata » Thu, May 21, 2015, 11:12 AM UTC

iddqd wrote: post #3

Everything fits in RAM if you have the budget for it.

That's the point of the tool! To remind people to compare the cost of fitting the data in ram compared to the cost of not putting it in ram.