Live data from Hacker News

Ask HN: How to learn how to scale software?

news.ycombinator.com

1–10 of 20 posts

Ask HN: How to learn how to scale software?

#1
I'm a mid-level software engineer, specialized in back-end programming. I'm getting my hands on Elixir currently, and I've read a few interesting blog posts (notably the one from Discord[1]) about its use case.

I've read the book called "Designing Data-Intensive applications" but it didn't scratch the same itch. I'd like to know how all of you who are writing software for millions of people scale your software. I'm guessing a lot of it comes from theory (e.g. blog articles, textbooks, courses) since it's not easily practicable (at least in my own company I don't need to scale for millions of users).

I don't know where to start so I'm hoping some of you can help

[1] https://discord.com/blog/using-rust-to-scale-elixir-for-11-million-concurrent-users

Re: Ask HN: How to learn how to scale software?

#2
First of all, the fact that you're asking this question puts you ahead of most engineers that I know. There's a well known saying that goes something like "Make it work, make it work well, then make it fast."

One of the simplest ways to think about scale is to think in terms of speed. This is a very very gross oversimplification and glosses over a lot of really important concepts, but at its core, you can say "if it's fast enough, it'll scale."

In a very simple mathematical sense, consider the idea that you have a single-instance, single-threaded application with no concurrency. If a request takes 1000ms to run, then you can do, at most, 1 request per second. If the request takes 100ms, you can do 10 requests per second. If it takes 10ms, you can do 100 request per second, and if it takes 1ms, you can do 1000 requests per second.

See? Speed is throughput is scale.

But that is, obviously, an oversimplification of the problem. Real applications are multi-threaded, multi-instance, and offer concurrency. So now the problem is identifying your bottlenecks and fixing them. But again, at its core, the main idea is speed. How can you make things as fast as possible?

(Note: There is a need to consider concurrency and parallelism, plus certain data stores have inherent speed limitations that may need to be overcome, and those things can offset poor speed, but the simplest path to scalability is speed and optimizing throughput.)

The analogy I like to use is the grocery store. Imagine you own a grocery store, and you want to make as much money as possible. Well, the best way to do that is to make sure your customers can get their food and check out as fast as possible. That means making sure the food is easy to find (i.e., read access is fast!), that they don't have to wait to check out (i.e., queue depth is low), and that checking out is fast (i.e., writes are fast). The faster your customers can walk in the door and back out again, the more customers you can sustain over a period of time.

On the other hand, if your customers take too long to find their groceries, or they spend too long waiting in line, or they have to write checks instead of swiping a smart phone, then you wind up with a backlog. And the larger the backlog, the longer it takes for money to hit your bank account.

So in this sense, time is literally money. The faster they can get through your system, the better.

I mentioned three different ways of thinking about speed: reads, writes, and queue depth.

Keeping with our grocery store analogy, consider how to improve each of those things. How do you make sure your customers can find what they're looking for as fast as possible? You "index" things. You put signs on the aisle, you organize your content in a way that is intuitive and puts related things near each other. If you want spaghetti, the pasta and the sauce and the parmesan cheese are all right next to each other. If you want breakfast, the eggs and milk and cinnamon rolls are right next to each other. In and out.

Similarly, your data needs to be organized smartly so that the user can get in and out as fast as possible. In a database, this means optimizing data structures, adding indices, and optimizing queries. Reduce expensive queries, keep cheap fast queries. Find ways to cache hot data. Make it easy to find what you need.

For writes, how do you speed up writes? One way is to make things asynchronous. Throw things that can be eventually consistent into queues and let an asynchronous job handle it outside the normal flow. The customer experiences minimal latency, and you've introduced concurrency to keep the data flowing while the customer is doing something else. This is, in part, why those little screens at the checkout counter ask you so many questions. They're distracting you while the cashier is scanning your groceries.

Queue depth optimization is important as well. If the queue gets really long at the grocery store, how do you improve that? You add more cashiers! The more cashiers you have, the more concurrent customers you can handle. But does it make sense to have 1 cashier per customer? Probably not. Now you've overscaled and you're spending too much money.

As you can see, this is a complex operation, and again, my analogy is overly simplified and very dumb, but I hope this gives you a decent idea of how to visualize a scalability problem.

I'm not familiar with Elixir, but frankly the concepts should translate to any language, although the details my vary.

My suggestion? Learn how to do profiling, identify bottlenecks, and target the biggest bang for your buck. The big risk here is micro-optimization, so fight for changes that give you order of magnitude improvements. Saving 50 microseconds isn't worth your time, but shaving off 1500 milliseconds almost certainly is.

Best of luck.

Re: Ask HN: How to learn how to scale software?

#3
Skilles like that are only really learned by doing it.

There are no shortcuts, sorry. No amount of synthetic experiments, simulations, theoretical examples will give you the actual hands on knowledge.

For sure, books etc. are useful to gain some basic understanding.

But beyond that - you should set more practical and specific goals that would lead you to scaling challenges naturally.

For example - build your own web crawler etc.

Or just set a goal to join a company with demanding products.

Re: Ask HN: How to learn how to scale software?

#6
post #2

First of all, the fact that you're asking this question puts you ahead of most engineers that I know. There's a well known saying that goes something like "Make it work, make it work well, then make it fast." One of the simplest ways to think about scale is to think in terms of speed. This is a very very gross oversimplification and glosses over a lot of really important concepts, but at its core, you can say "if it'…

As someone working in an extremely small scale world I don’t get opportunities to tackle scaling problems, so this was a very nice read. Thanks!

Re: Ask HN: How to learn how to scale software?

#7
What the contemporary dev environment teaches about scale is that it barely matters until you hit certain thresholds, and then it suddenly dominates everything.

A corollary of this is that as your program becomes more complex, relatively more of the code ceases to matter to performance, because its purpose is to configure a certain inner loop, and the inner loop becomes the only real bottleneck.

So, when coding for very small systems(retro or embedded) the whole program architecture matters, but at hyper scale, the problem is seen mostly in terms of efficiently distributing the load, and only deeply optimizing the program in very specific places.

A good starting point for thinking about this is a system like BitTorrent: The ideal torrent experience will most likely use multiple peers, and the performance needs to remain consistent as the number of peers increases and the load becomes more complex to distribute. But it's not really about the local performance so much as it is maintaining the overall network conditions - if every peer is doing "OK" at serving and retrieving files that's better than a very good experience served inconsistently.

Re: Ask HN: How to learn how to scale software?

#9

Skilles like that are only really learned by doing it. There are no shortcuts, sorry. No amount of synthetic experiments, simulations, theoretical examples will give you the actual hands on knowledge. For sure, books etc. are useful to gain some basic understanding. But beyond that - you should set more practical and specific goals that would lead you to scaling challenges naturally. For example - build your own web…

> Or just set a goal to join a company with demanding products.

Thie highlights a related but slightly different problem: it's important to learn enough to be able to talk about design choices for optimization, as these companies will throw you into such an interview as part of the hiring process.

Re: Ask HN: How to learn how to scale software?

#10
Another way to think about it is to be aware of and keep in mind all the various limitations of your implementation.

If you have a program that reads some stuff into memory, then transforms it, and then spits it back out (which is basically every program)... how can it still work if the data set is bigger than the amount of memory? What if it's bigger than the address space? (figure out how to split the data up and work on segments then combine the results)

Using ints or longs or whatever as an identifier? What if we have more things to identify than UINT_MAX or ULONG MAX? What if we need to allocate identifiers so quickly, and in so many different places, that contention for the next available number becomes a bottleneck? (use uuids or random identifiers)

A lot of people write software with the assumption that there is an unlimited amount of memory, CPU, etc., and usually that works fine, until it doesn't and it blows up. Scalability is about being aware of all the finite limitations, and having strategies for dealing with them instead of just hoping you don't get close to them.

Post reply on HN