Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Your data fits in RAM

yourdatafitsinram.com

Post reply on HN
Print view View original

81–90 of 230 posts

Re: Your data fits in RAM

#81

Post by jules »

Anderkent wrote: post #63
JonnieCache wrote: post #21

Earlier quoted context omitted.

The subtext is that running a fancy distributed system is more exciting and beneficial for ones resume than simply buying a massive bloody server and putting postgres on it, and that people are making tech decisions on this basis.

This of course ignores that it's much easier to get your hands on a cluster of average machines than one massive bloody server, and all the non-performance-oriented benefits of running a cluster (availability etc.). Much easier to request a client provisions 20 of their standard machines, or get them from AWS. People don't like custom hardware, and for good reason.

Depends on what you mean by massive bloody server. You can get a server with a terabyte of RAM for a price that's insignificant compared to the cost developing software to run on a cluster.

Re: Your data fits in RAM

#83

Post by jacquesm »

e12e wrote: post #73
jacquesm wrote: post #45

Earlier quoted context omitted.

Take a graph, you could use an SQL database to store it and do your graph analysis using SQL, or, alternatively, you could convert your graph to an extremely compact in-memory format and then do your analysis on that. Much better efficiency for the same size problem, bonus: you can now analyze much larger graphs with the same hardware.

Or maybe a bit of both: http://stackoverflow.com/questions/27967093/how-to-aggregate... I appreciate you taking the time to answer -- and I get that there's a reason for why we have graph databases. But I really meant something more concrete, as in here's a real-world example that isn't feasible to do on machine X with postgresql, but easy(ish) with a proper graph structure/db -- rather than "not all data structures…

Ok, one more example: A German company holds a very large amount of profile data and wanted to search through it. On disk storage in the 100's of gigabytes. Smart encoding of the data and a clever search strategy allowed the identification of 'candidate' records for matches with fairly high accuracy, fetching the few records that matched and checking if they really were matches sped things up two orders of magnitude over their SQL based solution.

It's very much dependent on how frequently you update the data and whether or not (re)loading the data or updating your structure in memory can be done efficient or not to determine whether or not such an approach is useful or not but going from 'too long to wait for' to 'near instant' for the result of a query is a nice gain.

In the end 'programmer efficiency' versus 'program efficiency' is one trade-off and cost of the hardware to operate the solution on is another. Making those trade-offs and determining the optimum can be hard.

But a rule of thumb is that a solution built up out of generic building blocks will usually be slower, easier to set up, will use more power and will be more expensive to operate but cheaper to build initially than a custom solution that is more optimal over the longer term.

So for a one-off analysis such a custom solution would never fly, but if you need to run your queries many 100's of times per second and the power bill is something that worries you then a more optimal solution might be worth investing in.

Re: Your data fits in RAM

#84

Post by feld »

mosselman wrote: post #8
lukegb wrote: post #2

Inspired by https://twitter.com/garybernhardt/status/600783770925420546

Can someone explain in a bit more detail what this is about? Is the 'joke' that running data computation in RAM is faster than what? From disk?

People will build gigantic compute clusters with expensive storage backends when their entire dataset fits in memory.

If it fits in memory, it's going to be magnitudes faster to work with than on any other infrastructure you can build.

So the trick is, you take their "big data problem" and hand them a server where everything can be hot in memory and their problem no longer exists.

Re: Your data fits in RAM

#85

Post by vidarh »

acqq wrote: post #64
cornellphds wrote: post #51

Earlier quoted context omitted.

This is classic case of "Algorithm/Problem Selection" if your algorithm/problem is tailored to a task such as PageRank, surely a single threaded highly optimized code will beat a cluster designed for ETL tasks. In real organizations where there are multiple workflows/algorithms, distributed systems always win out. Systems like Hadoop take care of Administration, Redundancy, Monitoring and Scheduling in a manner that…

Ditto for computationally intensive work: if it is CPU dominated, more CPU's calculating in parallel will be of advantage, even if the data could fit some RAM. There's no a single simple answer, but sure, whenever less computers are enough, less should be used. The recent problem is, some people love "clouds" so much today that they push there the work that could really be done locally.

Part of the problem is that a lot of problems that are CPU dominated on a single system becomes IO dominated once you start distributing it at very low node counts without very careful attention to detail.

Re: Your data fits in RAM

#86

Post by nl »

studentrob wrote: post #78
threeseed wrote: post #65

Earlier quoted context omitted.

Spark is fast becoming the default tool for big data. The recent addition of SparkR in 1.4 means that now data scientists can leverage in memory data in the cluster that has been put there by output from either Scala or DW developers. Combine it with Tachyon ( http://tachyon-project.org ) and it's not hard to imagine petabytes of data all processed in memory.

Can you explain what Tachyon does that's different from what Spark already provides? I haven't used either Spark or Tachyon. I thought the Spark solution was to just put my dataset in memory. But the Tachyon page seems to say the same thing

There's a slide deck[1] that explains it rather well.

Basically, Tachyon acts as a distributed, reliable, in memory file system.

To generalise enormously, programs have problems sharing data in RAM. Tachyon lets you share data between (say) your Spark jobs and your Hadoop Map/Reduce jobs at RAM speed, even across machines (it understands data-locality, so will attempt to keep data close to where it is being used).

[1] http://www.cs.berkeley.edu/~haoyuan/talks/Tachyon_2014-10-16...

Re: Your data fits in RAM

#87

Post by lerouxb »

regularfry wrote: post #43
jacquesm wrote: post #32

Earlier quoted context omitted.

That really depends on your usecase. Not all analysis is 'one-shot' and not all businesses are free to upload their data into 'the cloud'.

Not only that, but uploading a 1TiB dataset implies a certain quality of connection which not all businesses want to take on...

As opposed to 10*100GiB that can be uploaded over a 56.6K modem, right? ;)

Re: Your data fits in RAM

#88

Post by baldfat »

vegabook wrote: post #67
baldfat wrote: post #55

Earlier quoted context omitted.

> 0. Spreadsheet is all you need. I HATE when people use Spreadsheets to do anything besides simple math. http://lemire.me/blog/archives/2014/05/23/you-shouldnt-use-a... TL:DR your work is not reproducible and we can't see what you did to get to your numbers. A million examples of why this is bad. Also > 1. Python script is good enough You mean Python with pandas and numpy? I use R which is also a great choice > 2. J…

Arguably Numpy/Pandas is just as performant as Scala/Java and it certainly beats R hands down when data becomes more than a say 10-20 gigabytes after which I find R slows to a crawl.

Untrue about the speed of R. R and Python are always around the same speed, but there are always other options specially with R, where there is always more than one way to do anything.

We have data.tables and dplyr which data.tables is maybe on average 50% faster and on some points multiple faster than Python [http://datascience.la/dplyr-and-a-very-basic-benchmark/]

Re: Your data fits in RAM

#89

Post by bakhy »

threeseed wrote: post #68
bakhy wrote: post #28

Earlier quoted context omitted.

Apart from what other commenters already said about the cost of software complexity, is there a variant of Amdahl's law that could be used here? 10x 100 GB servers working on a problem together will probably never be 10x faster than 1x 100 GB server. Perhaps just the increase in the order of magnitude of the distance information needs to travel is already sufficient to set some higher bounds... So you may need to buy…

But you're assuming that memory is the only thing that matters here. 10x100GB will have 10x the computing power of 1x1TB server.

Ah, yes. Thank you!

Although, that means the 10x setup must cost much more. I think the idea in the comments above was taking 10 cheaper, weaker servers and somehow coming out with roughly the same price...

Well, in any case, things just got too complicated :)

Re: Your data fits in RAM

#90

Post by studentrob »

nl wrote: post #86
studentrob wrote: post #78

Earlier quoted context omitted.

Can you explain what Tachyon does that's different from what Spark already provides? I haven't used either Spark or Tachyon. I thought the Spark solution was to just put my dataset in memory. But the Tachyon page seems to say the same thing

There's a slide deck[1] that explains it rather well. Basically, Tachyon acts as a distributed, reliable, in memory file system. To generalise enormously, programs have problems sharing data in RAM. Tachyon lets you share data between (say) your Spark jobs and your Hadoop Map/Reduce jobs at RAM speed, even across machines (it understands data-locality, so will attempt to keep data close to where it is being used). [1…

Neat, thanks for the link, the code examples towards the end make it clear that this is pretty simple to use.
Post reply on HN