Sorry for the simple question but if you store it in ram what is the strategy for when the server is turned off?
Your data fits in RAM
111–120 of 230 posts
Re: Your data fits in RAM
#112Earlier quoted context omitted.
This of course ignores that it's much easier to get your hands on a cluster of average machines than one massive bloody server, and all the non-performance-oriented benefits of running a cluster (availability etc.). Much easier to request a client provisions 20 of their standard machines, or get them from AWS. People don't like custom hardware, and for good reason.
Amazon offers some bloody huge servers... 32 core, 256GB RAM, and 48TB HDD space. d2.8x large
and fwiw, costs at amazon will scale linearly with resources. the 1 beefy box with 256GB RAM box costs about as much as 16 boxes with 16GB of RAM each.
Re: Your data fits in RAM
#113Even if your data doesn't fit in RAM... and even if it does... when you're developing, you should be using a sample of your data that fits into RAM.
Re: Your data fits in RAM
#114Brilliant!
Re: Your data fits in RAM
#115Sorry for the simple question but if you store it in ram what is the strategy for when the server is turned off?
This all depends on what the data is used for. You may need to persist the data to disk on write even if all your data is in RAM.
Re: Your data fits in RAM
#116Earlier quoted context omitted.
This is classic case of "Algorithm/Problem Selection" if your algorithm/problem is tailored to a task such as PageRank, surely a single threaded highly optimized code will beat a cluster designed for ETL tasks. In real organizations where there are multiple workflows/algorithms, distributed systems always win out. Systems like Hadoop take care of Administration, Redundancy, Monitoring and Scheduling in a manner that…
Ditto for computationally intensive work: if it is CPU dominated, more CPU's calculating in parallel will be of advantage, even if the data could fit some RAM. There's no a single simple answer, but sure, whenever less computers are enough, less should be used. The recent problem is, some people love "clouds" so much today that they push there the work that could really be done locally.
> There's no a single simple answer, but sure, whenever less computers are enough, less should be used.
This - if you're using this website as your sole calculation then something has gone seriously wrong.
It was more intended to provoke discussion and thinking around overengineering things which can easily be done with, say, awk or a few lines of .
If you have large CPU requirements then sure, use Hadoop/Spark/some other distributed architecture. If you have a >6TB dataset but you only need 1GB of it at a time, then, well, maybe you still only need one computer.
Re: Your data fits in RAM
#117Sorry for the simple question but if you store it in ram what is the strategy for when the server is turned off?
You turn it back on, and load it back from the hard drive.
Re: Your data fits in RAM
#118Earlier quoted context omitted.
This is classic case of "Algorithm/Problem Selection" if your algorithm/problem is tailored to a task such as PageRank, surely a single threaded highly optimized code will beat a cluster designed for ETL tasks. In real organizations where there are multiple workflows/algorithms, distributed systems always win out. Systems like Hadoop take care of Administration, Redundancy, Monitoring and Scheduling in a manner that…
Ditto for computationally intensive work: if it is CPU dominated, more CPU's calculating in parallel will be of advantage, even if the data could fit some RAM. There's no a single simple answer, but sure, whenever less computers are enough, less should be used. The recent problem is, some people love "clouds" so much today that they push there the work that could really be done locally.
The problem is getting 100% of the work to be parallel. If I have to do that last 10% in one machine there isn't much point to more than 10 machines.
Re: Your data fits in RAM
#119Sorry for the simple question but if you store it in ram what is the strategy for when the server is turned off?
[deleted]
Re: Your data fits in RAM
#120Earlier quoted context omitted.
Depends on what you mean by massive bloody server. You can get a server with a terabyte of RAM for a price that's insignificant compared to the cost developing software to run on a cluster.
> You can get a server with a terabyte of RAM for a price that's insignificant compared to the cost developing software to run on a cluster. This assumes that a) You're in the valley where average developer salary is $10k a month or more, b) You're a large company paying developers that salary. There are lots of other places where a) Developers are cheaper, or b) You're a cash strapped startup whose developers are th…
Comparison still holds, because if you buy a cluster with X amount of RAM the price will be roughly the same as a single server with X amount of RAM. Except that for some large X there won't be any off the shelf servers you can buy with that amount of RAM (let's say 2000GB), but lets be honest here, 99% of companies needs are under that X especially if we're talking about startups.