Wow, I wish I had the spare change for one of these beasts. I think I have enough NP-hard problems to fill any RAM to the brim :)
Your data fits in RAM
101–110 of 230 posts
Re: Your data fits in RAM
#102What is the point of this site other budget shaming?
Re: Your data fits in RAM
#103Damn. my data is 6597069766657 bytes. Apparently if it was 6597069766656 bytes it would have fitted in RAM.
Well, hate to break it to you, but you probably have some overhead associated with your data, like your operating system or structures related to processing your data.
Re: Your data fits in RAM
#104Earlier quoted context omitted.
Can someone explain in a bit more detail what this is about? Is the 'joke' that running data computation in RAM is faster than what? From disk?
The subtext is that running a fancy distributed system is more exciting and beneficial for ones resume than simply buying a massive bloody server and putting postgres on it, and that people are making tech decisions on this basis.
It may be more exciting, but don't those people know about the CAP theorem?
Re: Your data fits in RAM
#105More accurate title would be fit in RAM of single machine. Maybe some bonus category: 0. Spreadsheet is all you need. 1. Python script is good enough. 2. Java/Scala is way to go. 3. Need to manage memory (gc doesn't cut), some custom organization. 4. Actually needs a cluster.
> 0. Spreadsheet is all you need. I HATE when people use Spreadsheets to do anything besides simple math. http://lemire.me/blog/archives/2014/05/23/you-shouldnt-use-a... TL:DR your work is not reproducible and we can't see what you did to get to your numbers. A million examples of why this is bad. Also > 1. Python script is good enough You mean Python with pandas and numpy? I use R which is also a great choice > 2. J…
Sadly, I recall those arguments against spreadsheet computation being made in the early 1990s. People simply won't learn.
Ray Panko, University of Hawaii.
http://panko.shidler.hawaii.edu/SSR/
http://panko.shidler.hawaii.edu/SSR/
Goes back to 1993:
Re: Your data fits in RAM
#106Earlier quoted context omitted.
Complexity, sure. But cost? I thought a single 1 TB RAM server is more expensive than 10x 100 GB RAM servers. And many people don't want to deal with physical hardware. Dealing with physical hardware increases operational complexity too. They want to rent a virtual/cloud server. Which provider allows you to rent a virtual server with 1 TB RAM?
I took a look around for "high-ram" servers, and it seems one I can buy today, is HP ProLiant DL580 Gen9. With just 256 GB of ram, it clocks in at 540.995,- NOK (71.5k USD). It has 96 ram slots, and I can't seem to find anything bigger than 32 GB DDR4 RAM, and rounding the price up 96x32GB comes to roughly 672.000,- NOK (~90k USD). Adding that up (throwing away the puny ram installed), gets us to a little over double…
A Dell R920 with four E7-8880L v2 15 core processors (e.g. 60 real threads, 120 with HT), and 1024GB / 1TB of memory costs about $50,000 USD. To go to 1.5TB of memory pushes you to $60,000 USD.
Expensive in a relative to a low-end server or month of cloud usage, but that's an absurd amount of computational power.
Re: Your data fits in RAM
#107Seems a bit naive, saying 2.1PB probably doesn't fit in ram, "but it could"... I get who this is aimed at, and why, but just saying that it fits in RAM isn't as useful as it could be. This is an opportunity to teach, not just snark.
It wasn't intended to snark - I apologize if it was seen that way. I whipped this up super quick and perhaps should have expanded on my meaning.
Re: Your data fits in RAM
#108We build object stores... so, no it most definitely does not.
Re: Your data fits in RAM
#109Earlier quoted context omitted.
This of course ignores that it's much easier to get your hands on a cluster of average machines than one massive bloody server, and all the non-performance-oriented benefits of running a cluster (availability etc.). Much easier to request a client provisions 20 of their standard machines, or get them from AWS. People don't like custom hardware, and for good reason.
Depends on what you mean by massive bloody server. You can get a server with a terabyte of RAM for a price that's insignificant compared to the cost developing software to run on a cluster.
> You can get a server with a terabyte of RAM for a price that's insignificant compared to the cost developing software to run on a cluster.
This assumes that a) You're in the valley where average developer salary is $10k a month or more, b) You're a large company paying developers that salary.
There are lots of other places where a) Developers are cheaper, or b) You're a cash strapped startup whose developers are the founder(s) working for free.
Re: Your data fits in RAM
#110600 PiB
"No, it probably doesn't fit in RAM (but it might)."
Well, well, well.