Live data from Hacker News

Your data fits in RAM

yourdatafitsinram.com

91–100 of 230 posts

Re: Your data fits in RAM

#91
post #85
post #64

Earlier quoted context omitted.

Ditto for computationally intensive work: if it is CPU dominated, more CPU's calculating in parallel will be of advantage, even if the data could fit some RAM. There's no a single simple answer, but sure, whenever less computers are enough, less should be used. The recent problem is, some people love "clouds" so much today that they push there the work that could really be done locally.

Part of the problem is that a lot of problems that are CPU dominated on a single system becomes IO dominated once you start distributing it at very low node counts without very careful attention to detail.

Yes, and starting from the distributed version first, one can even miss the fact that everything can be immensely faster once the communication is out of the way.

There's no a single simple answer, except: don't decide about the implementation first, instead competently and without prejudice evaluate your actual possibilities.

Also don't decide the language like "Python" or "Ruby" first. If you expect that the calculations are going to take a month, then code them in Ruby and wait the month for the result, you can miss the fact that you could have had the results in one day by just using another language, most probably without sacrifying the readability of the code much. Only if the task is really CPU-bound, of course.

On another side, if you have a ready solution in Python, and you'd need a month to develop the solution for other language, the first thing you have to consider is how often you plan to repeat the calculations afterwards. Etc.

Re: Your data fits in RAM

#93
post #55
post #9

More accurate title would be fit in RAM of single machine. Maybe some bonus category: 0. Spreadsheet is all you need. 1. Python script is good enough. 2. Java/Scala is way to go. 3. Need to manage memory (gc doesn't cut), some custom organization. 4. Actually needs a cluster.

> 0. Spreadsheet is all you need. I HATE when people use Spreadsheets to do anything besides simple math. http://lemire.me/blog/archives/2014/05/23/you-shouldnt-use-a... TL:DR your work is not reproducible and we can't see what you did to get to your numbers. A million examples of why this is bad. Also > 1. Python script is good enough You mean Python with pandas and numpy? I use R which is also a great choice > 2. J…

Ad 0. I agree. Your article got valid point. I wouldn't do serious research based solely on complicated spreadsheet.

Though in many non-techies things, like daily sales transactions it is a way to go.

Ad 1. pandas/numpy would put it on par with 2.

Ad 2. Would disagree. I know data scientist using Spark. Mostly they like Scala API.

In general, everyone got their favorite weapon of choice and what they feel comfortable. The point is that simpler solutions sometimes are just enough do their job.

Renting r3.4xlarge on AWS for an hour and play with your favorite tool may be an orders of magnitude easier/cheaper/faster than using big data solution.

Re: Your data fits in RAM

#94
post #31

I love it. I was just doing some Fermi estimates for a friend on the data for a project he has in the pipeline. I was curious whether or not it would be cost efficient for his project's budget to go with NVMe SSDs or have to stick with traditional SATA ones, and turns out it doesn't even matter (for now) because at least the first three months of data will fit in 256GB of RAM, even allowing for a 2.5x factor stemming…

Where's 2.5x from? I'd be curious to see any actual data on comparing memory footprint for a problem in C/Go/Rust to Python/Ruby. I'm sure it varies widely, but 2.5x might not be far off.

Re: Your data fits in RAM

#95

Earlier quoted context omitted.

This of course ignores that it's much easier to get your hands on a cluster of average machines than one massive bloody server, and all the non-performance-oriented benefits of running a cluster (availability etc.). Much easier to request a client provisions 20 of their standard machines, or get them from AWS. People don't like custom hardware, and for good reason.

Amazon offers some bloody huge servers... 32 core, 256GB RAM, and 48TB HDD space. d2.8x large

That is 4k a MONTH for 256gb of ram.

If you could do the same job on a fleet of 8-16GB servers.. you can get a lot more CPU for a lot less dollars. Depends if you really need everything on 1 machine or not (as of course nothing will beat same machine in memory locality)

Re: Your data fits in RAM

#96
post #35

Earlier quoted context omitted.

What does 6TB of RAM go for these days?

If you go by the Dell site linked from the article, a PowerEdge maxed out with 6TB RAM (4 sockets * 24 LRDIMMs * 64 GB) will set you back US$444,000 (or thereabouts). They also list 3.2 TB NVMe PCIe cards for US$11,000 each.

Though to be fair, no larger business would ever pay Dell sticker price. I bet they would sell it to a good client for only 200k ;)

Re: Your data fits in RAM

#97
Today I'm working on dataset of 1GB, which fits in memory. But it is not enough. If a variable is category/factor you need to introduce dummy values and your dataset starts picking the weight. Next - do you want apply ML algorithm in parallel? Upst, you need more memory. Done that? Now please use test dataset for prediction. My point that "data in memory" is just the beginning...

Re: Your data fits in RAM

#99
post #86

Earlier quoted context omitted.

There's a slide deck[1] that explains it rather well. Basically, Tachyon acts as a distributed, reliable, in memory file system. To generalise enormously, programs have problems sharing data in RAM. Tachyon lets you share data between (say) your Spark jobs and your Hadoop Map/Reduce jobs at RAM speed, even across machines (it understands data-locality, so will attempt to keep data close to where it is being used). [1…

Neat, thanks for the link, the code examples towards the end make it clear that this is pretty simple to use.

Yeah, most things coming from the Spark team are excellent in that respect.

I've never used Tachyon, but based on the wonderful "getting started" experience Spark gives I'd be confident it would be similarly well thought out.

Post reply on HN