41–50 of 230 posts
Earlier quoted context omitted.
Can someone explain in a bit more detail what this is about? Is the 'joke' that running data computation in RAM is faster than what? From disk?
See also: https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html
In my opinion the correct answer is 255Gb. (i.e. AWS r3X8 High Memory instances ). While one can purchase servers with larger memory most likely you will run into limitation on number of cores. Also note that there is at least some overhead in processing data, so you would need at least 2X the size of raw data. Finally while its a good thing to tweet, joke about and make fun of buzzword while trying to appear smart.…
That's assuming that everything needs to be 'high availability' and buying two of everything is a must. This is definitely not always the case. In plenty of situations buying a single item and simply repairing it when it breaks is a perfectly good strategy.
Earlier quoted context omitted.
Isn't this what the cloud is for? Surely you just rent an instance of the computing power you need for an hour or two then upload the data, a script and wait.
That really depends on your usecase. Not all analysis is 'one-shot' and not all businesses are free to upload their data into 'the cloud'.
It's not as easy as just buying more RAM. You'll have to pay more attention to how you make use of the various caches in between your CPU and RAM.
Earlier quoted context omitted.
For some problems SQL would already be way too much overhead.
Could you give a concrete example? If your working set is small, say 1TB -- and so fits in RAM -- for what kind of problems would using SQL be so much of an overhead that you need a different approach? And what would that approach be? I suppose you could have a massive set of linear equations that you might be able to fit into 1TB of RAM, but would be difficult to work with as tables in Postgres?
Earlier quoted context omitted.
10 servers with 100G will use a lot more power and will require distribution of your algorithm right along with your dataset, so instead of 10 server you will probably end up with a pretty high multiple of 10.
Isn't this what the cloud is for? Surely you just rent an instance of the computing power you need for an hour or two then upload the data, a script and wait.
Although we can theoretically handle up to 2^64 bytes of RAM (16 exabytes), the practical limit is much lower. I think someone on Wikipedia said it's somewhere around 8TB, but I imagine the performance of random access into 8TB RAM is much worse than a motherboard designed for up to 32GB RAM. It's not as easy as just buying more RAM. You'll have to pay more attention to how you make use of the various caches in betwe…
Earlier quoted context omitted.
What does 6TB of RAM go for these days?
Less than 6 machines with 1T each ;) (Assuming the 6 will operate only on local data and will never need to communicate in which case you may well end up with more than 6). But seriously: the price of RAM for servers is now ~10$ / G.
Earlier quoted context omitted.
See also: https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html
You should post that if it hasn't been posted already, it's a much better way to make the case than the current link.
I originally saw it on HN, but almost two years ago. Old comments: https://news.ycombinator.com/item?id=6398650
If you don't have money, you can't. Very few people can afford it.