Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Your data fits in RAM

yourdatafitsinram.com

Post reply on HN
Print view View original

211–220 of 230 posts

Re: Your data fits in RAM

#211

Post by vegabook »

mziel wrote: post #203
vegabook wrote: post #190

Earlier quoted context omitted.

> mm system.time(eigen(mm)) user system elapsed 5.26 0.00 5.25 IPy [1] >>> xx = np.random.rand(1000000).reshape(1000, 1000) IPy [2] >>> %timeit(np.linalg.eig(xx)) 1 loops, best of 3: 1.28 s per loop But where R really stinks is memory access: > system.time(for(x in 1:1000) for(y in 1:1000) mm[x, y] >> def do(): ...: for x in range(1000): ...: for y in range(1000): ...: xx[x, y] = 1 ...: IPy [10] >>> %timeit do() 10 l…

That's why you never ever grow lists with R. do.call('rbind',...) or even better data.table::rbindlist(). You can't blame R for being slow if you don't know how to write fast R code.

obviously I use do.call all day long because R is my primary weapon, but even if I say so myself, a happy R user, Python with Numpy is faster. I would invite you to show me a single instance where R is faster at bog-standard memory access, than Numpy. My example demonstrates exactly this. Can there be anything simpler than a matrix access? And if that's (8x) slower, everything built on this fundamental building block of all computing (accessing memory) will be slow too. It's R's primary weakness and everybody knows it. Let me make it abundantly clear:

  > xx  system.time(xx[]
If the very basics, namely changing stuff in memory, is so much slower, then the entire edifice built on it will be slower too, no matter how much you mess around with do.call. And to address the issue of (slow, but quickly expandable) Python lists, recall that all of data science in Python is built on Numpy so the above comparisons are fair.

Re: Your data fits in RAM

#212

Post by kragen »

falcolas wrote: post #70

Seems a bit naive, saying 2.1PB probably doesn't fit in ram, "but it could"... I get who this is aimed at, and why, but just saying that it fits in RAM isn't as useful as it could be. This is an opportunity to teach, not just snark.

Like, "To fit 2.1PB in RAM, you could spin up 9 r3.8xlarge EC2 instances for US$3.15 per hour"?

Re: Your data fits in RAM

#213

Post by jacquesm »

e12e wrote: post #154
jacquesm wrote: post #83

Earlier quoted context omitted.

Ok, one more example: A German company holds a very large amount of profile data and wanted to search through it. On disk storage in the 100's of gigabytes. Smart encoding of the data and a clever search strategy allowed the identification of 'candidate' records for matches with fairly high accuracy, fetching the few records that matched and checking if they really were matches sped things up two orders of magnitude…

> A German company holds a very large amount of profile data and wanted to search through it. On disk storage in the 100's of gigabytes. Smart encoding of the data and a clever search strategy allowed the identification of 'candidate' records for matches with fairly high accuracy, fetching the few records that matched and checking if they really were matches sped things up two orders of magnitude over their SQL based…

Tough crowd.

Re: Your data fits in RAM

#214

Post by jerven »

I think its wrong. It says 64 TB does not fit in RAM, but you can get 64TB machines from SGI as well 32 TB ones from Oracle.

The SGI one with up to 2048 cores are larger in their single system images than most people have in their clusters.

The benefit of these systems is not really the ease of programming but the speed of interconnect.

List price of the Oracle one was 3 million a few years ago. But most of that is actually in the high density dimms. These days I think the price must be lower, but I won't waste my Oracle sales contact time in figuring out what it is today. Of course it will still be expensive, it is an Oracle product after all.

However, an equivalent dell list price cluster of simple 1U boxes (512 6C/64GB ones!) will go for 1.5 million. The fact that to house 512 boxes i.e. 25 racks or so plus networking. Of course you do get 1/3rd more cores than the SGI one.

For many of us that are between the just use a single normal server and yet too small for the google solutions. These big memory solutions from Oracle and SGI can make sense even if they are not the first thing that comes to mind!

Re: Your data fits in RAM

#215

Post by jacquesm »

RogerL wrote: post #148
jacquesm wrote: post #52

Earlier quoted context omitted.

No, that's a very expensive server, and Dell will charge you a hefty premium for the memory. There are quite a few options that will cost you less than that (of course the maximum capacity will vary). It would be nice to see an article comparing all the high RAM machines side by side with specs and prices. The largest machine I have right now will hold 512G and was a run-of-the-mill machine, it was about $5K, I'd exp…

Can you point me to cheap 64GB LRDIMM Octal rank memory? Dell's prices for this seem to be the market rate, but maybe I don't know where to shop.

I can get Octal rank in bulk for ~$1K so that's $16 or thereabouts / G, still not bad for RAM that is obviously going to be sold in smaller quantities. Note that HP or Dell will probably not be happy if you use 3rd party RAM in their machines (if they didn't pull tricks to make sure only their own stuff works!).

(For contrast, that $1K if you'd spend it on HP branded RAM would not even get you four 16G dual rank units...).

Re: Your data fits in RAM

#216

Post by jacquesm »

RogerL wrote: post #146
jacquesm wrote: post #18

Earlier quoted context omitted.

Less than 6 machines with 1T each ;) (Assuming the 6 will operate only on local data and will never need to communicate in which case you may well end up with more than 6). But seriously: the price of RAM for servers is now ~10$ / G.

The price of 6TB is more in the range of 1/2 million. 64GB LRDIMM Octal rank is running $4.5K.

You really don't know where to shop ;)

5 seconds of googling gets you:

http://www.allhdd.com/index.php?target=products&mode=search&...

No idea if that will work in that particular server but technically there is no reason why it should not. You can establish a relationship with a distributor to get better prices than those that you can find listed. Those 'call for quotes' things can have two meanings: you're about to be screwed or 'we will give you a better price if you promise to not divulge that we did that'.

Re: Your data fits in RAM

#217

Post by SwellJoe »

vidarh wrote: post #180
SwellJoe wrote: post #141

So, let's say my system is currently backed by MySQL or PostgreSQL, and that is not fungible. How would one move that data into RAM, including writes? And, how would one maintain some level of safety in the event of a crash? i.e. I don't really care if I lose X amount of time worth of data (say, five minutes), but I do care that when I reboot the system, the database comes back from disk into RAM in a consistent stat…

If performance is a problem with a 1.4GB database, your system is either extremely low end, or you have some seriously un-optimised queries or database architecture and/or should take a long hard look at what caching you are not doing that you should be (including in the form of materialised views) It simply is not your overall disk IO capacity that is the performance problem with a dataset that small. At least not f…

So, my question wasn't about whether the system in question is shitty. I know that it's shitty. I also know that the CMS has less than optimal queries. That was also not my question.

"It simply is not your overall disk IO capacity that is the performance problem with a dataset that small."

But, it clearly, and measurably is; there's nothing to argue about there. That which comes from RAM (reads) is fast, that which waits on disk (writes) is slow. Writes take several seconds to complete, thus users wait several seconds for their comments and posts to save before being able to continue reading. That sucks, and is stupid and pointless, especially since it's not even all that important that we avoid data loss. A minute of data loss averages out to close enough to zero actual data loss, since crashes are so rare.

Re: Your data fits in RAM

#218

Post by vegabook »

pquerna wrote: post #197
vegabook wrote: post #138

Earlier quoted context omitted.

10k? Those sticks of RAM alone will cost you something like 75k USD. Then you'll need the processors, arguably 4 of the top of the line 18-core XEONs at 5000 USD each. Then you'll need to put it all together with software and a (properly cooled) rack, not to mention the terminal(s) to access it, plus the personnel to put this baby together for you. This box could easily cost you 150 grand.

Its not cost effective to use non E5-class Xeons, or go above 32GB DIMMs right now.... So you want a Dual-Processor setup, 16 DIMM slots, so 16x 32GB = 512GB w/ Dual Proc -- which you can do for about $10,000.

That's a very nice piece of kits for 10k I have to say. Thanks for the "sweetspot" price/perf advice. Seems like excellent value. I've had my heart on a badass mac pro but these specs put it to shame.
Post reply on HN