Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scalability, but at what cost? (2015)

frankmcsherry.org

Post reply on HN
Print view View original

31–40 of 80 posts

Re: Scalability, but at what cost? (2015)

#31

Post by ChuckMcM »

JoachimSchipper wrote: post #26
Joeri wrote: post #17

Earlier quoted context omitted.

For a 2 TB dataset you could also pay supermicro 50k to get a 40 core 3 TB RAM monster that can keep that whole dataset in RAM. At 50 GB / sec throughput that would keep your query roundtrip time at somewhere around the minute mark. Not quite 3 seconds, but then not quite a thousand nodes either. Of course, rebooting that machine would be awkward. Still, I think the general rule applies that if you can buy a server t…

Idle question: is 50 GB/s a reasonable throughput to expect for such a monster? DDR4 has a peak transfer rate of ~12.8-19.2 GB/s per stick (per https://en.wikipedia.org/wiki/DDR4_SDRAM ), so I'd expect quite a bit more bandwidth for predictable accesses - are you using some useful rule-of-thumb, or just unduly pessimistic? ;-)

At NetApp when we were doing scaling analysis in the early 2000's it became clear the memory bandwidth was limited more by the transaction rate of the memory controller than it was the actual available bandwidth of the memory subsystem.

That is because a memory transaction involves "opening" a page, and then "doing the operation", which can be one to several hundred locations long. "Pointer chasing", code that reads in a structure, then deferences a pointer to another structure, then derefences that pointer to still another stucture, Etc. was really hard on the memory subsystem. It burned a lot of memory ops reading relatively small chunks of memory.

Its a great topic in systems architecture and there are a number of papers on it.

Re: Scalability, but at what cost? (2015)

#32

Post by Joeri »

sigusr1 wrote: post #25
brianwawok wrote: post #3

Earlier quoted context omitted.

This is kind of the same argument as microservices. We could write 30 microservices deployed on 30 docker images with load balancing and FT and all that magic for a basic webapp... Or we could just write a pretty fast webserver and do it with 1 server. (Or if it is stateless, do it with a few for still a lot less work than a giant microservice cluster). I think in the last year or so microservices have become a littl…

If you haven't seen stackoverflows server architecture posts you'd probably enjoy them: https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar... I still think containers are great, and I really like the abstractions kubernetes provides, but if I ever had enough traffic to worry about scaling, I envision running a small cluster of very powerful computers rather than 100s of weak ones.

So basically, through careful engineering, they can run stackoverflow.com, one of the top-ten sites in the world, on a single web server and a single database server. Beefy machines for sure, but it kind of takes the air out of the web-scale hype balloon.

Re: Scalability, but at what cost? (2015)

#33

Post by joslin01 »

brianwawok wrote: post #30
aianus wrote: post #16

Earlier quoted context omitted.

Vertical scaling requires hiring good engineers instead of mediocre ones (additional cost of $100,000s per year across the team). Horizontal scaling in comparison is much, much cheaper for your average CRUD app.

Maybe, maybe not. For example choosing Java over Ruby would give you 2-10x better perf per server... and I am not sure that Java devs cost any more or less than Ruby devs. Now we can get into an argument about developer productivity and all that.. but form a purely "i want to run 10x more users per server"... something like Java / Ruby gets you a long ways.

I talked to a CTO once that said he brought his RoR fleet down from 60 servers to 6-8 by switching to Scala.

Re: Scalability, but at what cost? (2015)

#34

Post by scott_s »

ChuckMcM wrote: post #10

I like the analysis, basically it says "hey you don't have big data" :-) but that requires a bit more explanation. The only advantage of clustered systems like Spark, Hadoop, and others is aggregate bandwidth to disk and memory. We know that because Amdahl's law tells us that parallelizing something invariably adds overhead. So from a systems perspective that overhead has to be "paid for" by some other improvement, a…

Amdahl's law is not about overhead - it is true that parallelizing a computation usually has some cost, and you have to make sure what you're parallelizing is a heavy-enough computation that you still win in the presence of that cost. But that's not what Amdahl's law is about.

It is about the limitation of overall performance improvement when improving the performance of a component. If, say, you improve the performance of a piece of your application that only accounts for 10% of the running time, then you're limited by a 10% total performance improvement. This matters in a parallel context because if you have heavily parallelized your application, and it has actually improved performance, you're likely to eventually be limited by the performance of the sequential parts of your application. See https://en.wikipedia.org/wiki/Amdahl%27s_law ; although the last line of the introduction reads like nonsense.

Re: Scalability, but at what cost? (2015)

#35

Post by fat0wl »

ChuckMcM wrote: post #10

I like the analysis, basically it says "hey you don't have big data" :-) but that requires a bit more explanation. The only advantage of clustered systems like Spark, Hadoop, and others is aggregate bandwidth to disk and memory. We know that because Amdahl's law tells us that parallelizing something invariably adds overhead. So from a systems perspective that overhead has to be "paid for" by some other improvement, a…

This is true but keep in mind that you have to set up a pipeline for collecting the data into HDFS (Storm? batch loading?) & you have to pay for the machines.

So while your analysis is valid, there are more "costs" at play like developer time, cluster maintenance, hardware. I like to play with Spark's ML libraries but am wary about designing projects specifically around them because of this overhead, especially when trying to distribute some API/tech that you'd like others to use.

Not trying to be a downer, I actually wish the choice to go distributed was more of a no-brainer, hah. Would love for some APIs to emerge that could be used locally/distributed transparently without actually having to run a dummy cluster & data migration to run locally.

Re: Scalability, but at what cost? (2015)

#36

Post by btilly »

I have long offered the following advice.

If you have code that is not able to run any more in a scripting language, and it is not embarrassingly parallel, you have two choices.

1. Move to something like C++, and optimize the heck out of it. You will gain something like 1-2 orders of magnitude in performance and then hit a wall.

2. Move to a distributed architecture. You immediately lose 1-2 orders of magnitude in performance, but then can scale essentially forever.

If you expect your distributed system to need less than 100 machines, you should seriously consider option #1.

Re: Scalability, but at what cost? (2015)

#37

Post by matthewmacleod »

joslin01 wrote: post #33
brianwawok wrote: post #30

Earlier quoted context omitted.

Maybe, maybe not. For example choosing Java over Ruby would give you 2-10x better perf per server... and I am not sure that Java devs cost any more or less than Ruby devs. Now we can get into an argument about developer productivity and all that.. but form a purely "i want to run 10x more users per server"... something like Java / Ruby gets you a long ways.

I talked to a CTO once that said he brought his RoR fleet down from 60 servers to 6-8 by switching to Scala.

When he said "switching to Scala" I reckon he probably meant "rewriting our platform."

It's very difficult to do a comparison like this in practice, because switching languages inherently involves a rewrite of a platform.

Re: Scalability, but at what cost? (2015)

#38

Post by aianus »

brianwawok wrote: post #30
aianus wrote: post #16

Earlier quoted context omitted.

Vertical scaling requires hiring good engineers instead of mediocre ones (additional cost of $100,000s per year across the team). Horizontal scaling in comparison is much, much cheaper for your average CRUD app.

Maybe, maybe not. For example choosing Java over Ruby would give you 2-10x better perf per server... and I am not sure that Java devs cost any more or less than Ruby devs. Now we can get into an argument about developer productivity and all that.. but form a purely "i want to run 10x more users per server"... something like Java / Ruby gets you a long ways.

> I am not sure that Java devs cost any more or less than Ruby devs

I don't know about that, how many bootcamps are pumping out Java programmers instead of Python/Ruby programmers?

Re: Scalability, but at what cost? (2015)

#39

Post by chubot »

btilly wrote: post #36

I have long offered the following advice. If you have code that is not able to run any more in a scripting language, and it is not embarrassingly parallel, you have two choices. 1. Move to something like C++, and optimize the heck out of it. You will gain something like 1-2 orders of magnitude in performance and then hit a wall. 2. Move to a distributed architecture. You immediately lose 1-2 orders of magnitude in pe…

I would call those options #2 and #3. Option #1 is:

1. Parallelize on a single machine. Most scripting languages have single-threaded bottlenecks (Python, Ruby, node.js, etc.), so this means using multiple processes. xargs -P goes a long way. The coding changes are essentially a subset of what you need to distribute your program anyway.

32x or 64x speedup is nothing to sneeze at on a modern machine. The difference between 5 minutes and 5 hours usually solves your problem, practically speaking. And this means you don't have to touch every line of code, as you would if you were doing a C++ rewrite.

But also don't forget that you can often rewrite 10% of your code in C++ and keep the other 90% in Python, and get a 10x speedup. This requires fairly deep understanding of both your program and of the Python/C interface. It helps to adopt a data flow style so you are not crossing the boundary a lot. And make sure you release the GIL, and consider starting threads in C++ rather than in Python, etc.

I've also optimized an R program in C++ and gotten 125x speedup -- and that's single threaded C++; multithreaded would be another 2 orders of magnitude!!! But it also involved fixing a bunch of R performance errors, so don't underestimate that too.

In practice, any program which you actually care enough about to rewrite for speed usually has some application-level performance bugs -- i.e. slowness unrelated to the slowness of the underlying language platform.

People scale out to many machines because they don't want to rewrite every line of code. But the first step is to "scale out" by using all the cores on a single machine. In some sense, this is the best of both worlds, because you are incurring neither the overhead of distribution (serialization and networking) nor the complexity of distributed error handling.

Re: Scalability, but at what cost? (2015)

#40

Post by brianwawok »

aianus wrote: post #38
brianwawok wrote: post #30

Earlier quoted context omitted.

Maybe, maybe not. For example choosing Java over Ruby would give you 2-10x better perf per server... and I am not sure that Java devs cost any more or less than Ruby devs. Now we can get into an argument about developer productivity and all that.. but form a purely "i want to run 10x more users per server"... something like Java / Ruby gets you a long ways.

> I am not sure that Java devs cost any more or less than Ruby devs I don't know about that, how many bootcamps are pumping out Java programmers instead of Python/Ruby programmers?

Well in Chicago some of the colleges are pumping out Java Devs...

I guess if you are comparing a 4 year college Java dev to a 6 week bootcamp Ruby dev you have something, but that seems weird...

Post reply on HN