Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Your data fits in RAM

yourdatafitsinram.com

Post reply on HN
Print view View original

51–60 of 230 posts

Re: Your data fits in RAM

#51

Post by cornellphds »

Smerity wrote: post #36

It's probably worth extending "Your data fits in RAM" to "Your data doesn't fit in RAM, but it does fit on an SSD". So many problems will still work with quite reasonable performance when using an SSD instead. By using a single machine with an array of SSDs, you also avoid the complexity and overhead of distributed systems. My favourite realization of this: Frank McSherry shows how simplicity and a few optimisations…

This is classic case of "Algorithm/Problem Selection" if your algorithm/problem is tailored to a task such as PageRank, surely a single threaded highly optimized code will beat a cluster designed for ETL tasks. In real organizations where there are multiple workflows/algorithms, distributed systems always win out. Systems like Hadoop take care of Administration, Redundancy, Monitoring and Scheduling in a manner that a single machine cannot. Sure you can "Grep" faster on a laptop than AWS EMR with 4 Medium instances, but in reality where you have 12 types of jobs which are run by team of 6 people, you are much better off with a distributed system.

Re: Your data fits in RAM

#52

Post by jacquesm »

akie wrote: post #48
jacquesm wrote: post #18

Earlier quoted context omitted.

Less than 6 machines with 1T each ;) (Assuming the 6 will operate only on local data and will never need to communicate in which case you may well end up with more than 6). But seriously: the price of RAM for servers is now ~10$ / G.

That would translate to ~$60k for 6TB of RAM. Plus the cost of the server itself ($10k?)

No, that's a very expensive server, and Dell will charge you a hefty premium for the memory. There are quite a few options that will cost you less than that (of course the maximum capacity will vary).

It would be nice to see an article comparing all the high RAM machines side by side with specs and prices.

The largest machine I have right now will hold 512G and was a run-of-the-mill machine, it was about $5K, I'd expect the more exotic ones to be substantially more expensive but probably not as expensive as the machines linked here.

Re: Your data fits in RAM

#53

Post by baldfat »

smegel wrote: post #38

If you are programming in R, you sure better hope it does!

After reading the title I was sure there was something about R in the comments.

You can program R in Spark you can now program in R http://blog.revolutionanalytics.com/2015/01/a-first-look-at-...

Now you can work directly with SQL Server as announced this week by MS. http://www.computerworld.com/article/2923214/big-data/sql-se...

I have had a ton of arguments about R's "biggest weakness" being that it uses RAM. I haven't once in the almost 3 years of working in R that I ran into this road block, but I am sure others have. Which there are several good distributed choices that will keep getting better and better.

Using RAM instead of Distributed is better in R as well as really any other language in terms of complexity and flexibility.

Re: Your data fits in RAM

#54

Post by cornellphds »

jacquesm wrote: post #42
cornellphds wrote: post #40

In my opinion the correct answer is 255Gb. (i.e. AWS r3X8 High Memory instances ). While one can purchase servers with larger memory most likely you will run into limitation on number of cores. Also note that there is at least some overhead in processing data, so you would need at least 2X the size of raw data. Finally while its a good thing to tweet, joke about and make fun of buzzword while trying to appear smart.…

> Further you would ideally need two of them to remove single point of failure. That's assuming that everything needs to be 'high availability' and buying two of everything is a must. This is definitely not always the case. In plenty of situations buying a single item and simply repairing it when it breaks is a perfectly good strategy.

Its not about having two of everything at all times, but rather about having a capacity whenever you need it. At 244Gb you hit a sweet point where you can have access to large capability at a flexible price (Spot Market / On Demand / On Premise). This is what separates engineers with business acumen from run of the mill "consultants" with a search engine.

Re: Your data fits in RAM

#55

Post by baldfat »

jakozaur wrote: post #9

More accurate title would be fit in RAM of single machine. Maybe some bonus category: 0. Spreadsheet is all you need. 1. Python script is good enough. 2. Java/Scala is way to go. 3. Need to manage memory (gc doesn't cut), some custom organization. 4. Actually needs a cluster.

> 0. Spreadsheet is all you need.

I HATE when people use Spreadsheets to do anything besides simple math.

http://lemire.me/blog/archives/2014/05/23/you-shouldnt-use-a...

TL:DR your work is not reproducible and we can't see what you did to get to your numbers. A million examples of why this is bad.

Also

> 1. Python script is good enough

You mean Python with pandas and numpy?

I use R which is also a great choice

> 2. Java/Scala is way to go.

For you but the vast majority of Data Scientist don't use either and their choice for people is not universal. Julia looks like a great new comer. I again mainly use R.

> 3 & 4 are good points.

Re: Your data fits in RAM

#58

Post by praseodym »

And if your data doesn't fit in a single server's RAM, just buy some more and run Apache Spark [1] on them. It's an in-memory computation engine that's really nice to program for: you don't have to worry about low-level clustering details (like MapReduce). And it's way (10-100x) faster than Hadoop.

[1] https://spark.apache.org

Re: Your data fits in RAM

#60

Post by threeseed »

Smerity wrote: post #36

It's probably worth extending "Your data fits in RAM" to "Your data doesn't fit in RAM, but it does fit on an SSD". So many problems will still work with quite reasonable performance when using an SSD instead. By using a single machine with an array of SSDs, you also avoid the complexity and overhead of distributed systems. My favourite realization of this: Frank McSherry shows how simplicity and a few optimisations…

Scalable graph systems have been around for many years and have never taken off.

Most businesses doing big data (like ours) often have multiple disparate data sources that at the start of the pipelines are ETLing into some EDW. Trying to consolidate them into a single integrated view is very difficult and time/resource intensive. Having billions of disconnected nodes in the graph would be very hard to reason with.

Post reply on HN