Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scalability, but at what cost? (2015)

frankmcsherry.org

Post reply on HN
Print view View original

61–70 of 80 posts

Re: Scalability, but at what cost? (2015)

#61

Post by bane »

ben_jones wrote: post #8
brianwawok wrote: post #3

Earlier quoted context omitted.

This is kind of the same argument as microservices. We could write 30 microservices deployed on 30 docker images with load balancing and FT and all that magic for a basic webapp... Or we could just write a pretty fast webserver and do it with 1 server. (Or if it is stateless, do it with a few for still a lot less work than a giant microservice cluster). I think in the last year or so microservices have become a littl…

People forget that the two methods of scaling are HORIZONTAL and VERTICAL. They think: "I can just put some micro-services behind HA proxy and boom, more capacity!". And then they forget if they had just modified that one query and tweaked that one for-loop they could've had that same capacity without launching six new servers with all kinds of potential for the wiring to go down and cause downtime. Plus the dev time…

I usually think of scaling in 3 methods: horizontal (add more machines), vertical (get a bigger machine) and inward (improve your algorithms).

The largest performance gains I've seen are most often of the inward type -- like multi-thousands of percent, just by rethinking the code or the approach.

Bonus, improving inward scaling multiplies investments in horizontal and vertical scaling.

Re: Scalability, but at what cost? (2015)

#62

Post by sijoe »

justinsaccount wrote: post #58
sijoe wrote: post #45

Earlier quoted context omitted.

[old HPC guy here, you kids get off my lawn!] ... well, then there is the issue of distributing that 2TB data set. I'll get to the Amdahl's law issue in a moment. This is a non-trivial problem. Ok, it is trivial, but its serial in most cases. Unless you start out with a completely distributed data set. And allocate permanent space on those 1000 nodes. So the data has to move once. And you can amortize that across all…

> 1000 nodes, 2TB of data, assume standard crappy cloud network connection, use a 1GbE connection per node. The serial portion of this computation is the data distribution. I find your statements confusing. The whole point of things like hadoop is that the data is already distributed and the data storage nodes are also computational nodes. So there is no data distribution that takes 1/4 day or even 50-100 seconds. It…

Heh (on the jobs paused bits) ... most modern shared file systems are HA (or nearly HA) except when people build them cheaply. And then you get an effective RAID0.

I was pointing out that if you are doing the analytics at AWS or similar on-demand scenario (a common pattern I see people trying/using and eventually rejecting), you have a serial data motion step to distribute data to your data lake before processing. Then you extract your results, decommission all those servers. Rinse, and repeat.

The point is, that for ephermal compute/storage scenarios, you have a set of poorly architected resources tied together in a way that pretty much guarantees you have a large (dominant) serial step before anything goes in parallel.

What we advocate (and have been doing so for more than a decade), are far more capable building blocks of storage+compute. So if you are going to build a system to process a large amount of data, instead of buying 1000 nodes and managing them, buy 20-50 far more capable nodes at a small fraction of the price, and get the same/better performance.

It also doesn't take 0 seconds. There is a distribution/management overhead (very much non-zero), as well as data motion overhead for results (depending upon the nature of the query). When you subdivide the problems finely enough, the query management overhead actually dominates the computation (which was another aspect of the point I made, but it wasn't explicit on my part).

So we are fine with building hadoop/spark/kdb+ systems ... we just want them built out of units that can move 10-20GB/s between storage and processing. Which lets you hit 50-100s per TB of processed data. Which gives you a fighting chance at dealing with PB scale problems in reasonable time frames, which a number of our users have.

Re: Scalability, but at what cost? (2015)

#63

Post by ap22213 »

I think the missed point is that Spark is very easy. I can get an average Java or Python developer trained up on it in less than a day. The python shell is very simple to use out of the box. And, it's incredibly convenient to be able to either run locally or on a huge cluster. I can use the same code to easily process batch jobs from 1 MiB to 100 TiB. In my mind, it's just a cost savings. Developer time is expensive, and it's hard to find great developers. Hardware is cheap.

No way am I a scalability expert, and I really don't have time to be one. I started using Spark when I had to sort 10 TiB on disk, and it scored the highest on sorting performance. I struggled with implementing a fast disk sort quickly, and I gave Spark a whirl, and it fixed my problem, fast. Since then, I've found it useful in a lot of other ways.

Re: Scalability, but at what cost? (2015)

#64

Post by wstrange »

FooBarWidget wrote: post #43
joslin01 wrote: post #33

Earlier quoted context omitted.

I talked to a CTO once that said he brought his RoR fleet down from 60 servers to 6-8 by switching to Scala.

I was once involved in a large-scale government project that rewrote a Java app to RoR. They went from 50 servers to 10. It has probably got more to do with the rewrite and the new architecture than whatever language it was written in.

Or the fact that new servers are an order of magnitude faster than the ones they replaced.

Re: Scalability, but at what cost? (2015)

#65

Post by Joeri »

Bromskloss wrote: post #54
sigusr1 wrote: post #25

Earlier quoted context omitted.

If you haven't seen stackoverflows server architecture posts you'd probably enjoy them: https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar... I still think containers are great, and I really like the abstractions kubernetes provides, but if I ever had enough traffic to worry about scaling, I envision running a small cluster of very powerful computers rather than 100s of weak ones.

Huh? Why are they running Windows?

Because they thought it was the better choice for them. You cannot argue with the result.

https://www.quora.com/Why-does-StackExchange-StackOverflow-u...

Re: Scalability, but at what cost? (2015)

#66

Post by Drdrdrq »

wstrange wrote: post #64
FooBarWidget wrote: post #43

Earlier quoted context omitted.

I was once involved in a large-scale government project that rewrote a Java app to RoR. They went from 50 servers to 10. It has probably got more to do with the rewrite and the new architecture than whatever language it was written in.

Or the fact that new servers are an order of magnitude faster than the ones they replaced.

Order of magnitude faster? I doubt that. Unless the old ones were really old / slow to begin with. But really, we are both speculating without knowing the facts.

Re: Scalability, but at what cost? (2015)

#67

Post by e12e »

sijoe wrote: post #60
e12e wrote: post #57

Earlier quoted context omitted.

Sounds like you are doing the opposite of what joyent are doing: they basically pair (relevant parts of) data with the program (to process that part). And the reduce/aggregate over the result (that's my takeaway from joyent's marketing, anyway). Which company do you work for? (unless it's a secret for some reason)

Actually Joyent's manta is quite similar in concept to what we've been doing for years (before manta came out). The idea is to build very capable systems and aggregate them. Not a bunch of fairly low end units (like typical AWS/etc). Our argument is that if you are going to build a high performance computing infrastructure, you ought to build it in an architecturally useful manner. The cost to do so is marginally mor…

Thank you for clarifying. Re-reading your first comment, in light of your second, I see that that's indeed what you were saying in the first place. But apparently that's not quite what I read :-)

Re: Scalability, but at what cost? (2015)

#68

Post by colin_mccabe »

Big data systems aren't about squeezing every drop of performance out of the hardware. They're about being able to scale your solution up effortlessly. Having a uniform programming model so that you don't have to rewrite your solution N times when you go from a small test dataset to production data, to 1000x the data. It's also about using a system that other people use, so that you can actually hire admins and service companies with experience in maintaining your solution.

Re: Scalability, but at what cost? (2015)

#69

Post by collyw »

matthewmacleod wrote: post #37
joslin01 wrote: post #33

Earlier quoted context omitted.

I talked to a CTO once that said he brought his RoR fleet down from 60 servers to 6-8 by switching to Scala.

When he said "switching to Scala" I reckon he probably meant "rewriting our platform." It's very difficult to do a comparison like this in practice, because switching languages inherently involves a rewrite of a platform.

I am curious how much a rewrite staying in Ruby could have saved. Its always easier to write code when you have a good specification, and a working application would fulfill that role well.

Re: Scalability, but at what cost? (2015)

#70

Post by Eridrus »

Eugr wrote: post #48

I agree with author that [in most cases] you don't need distributed processing for your algorithms. But sometimes you do, and when you do need it you have to understand that there is no silver bullet. Creating a distributed system is very difficult, even when using platforms like Spark. Not all algorithms can be scaled easily or scaled at all, and not all algorithms in Spark MLLib or GraphX are actually designed to b…

NLLib is ostensibly open source, why not improve it? Was your final solution too specialized?
Post reply on HN