Ask HN: How to learn web scalability?

Hey fellow hackers,

I am a programmer, but most of my work doesn't involve the web. The most I have done with web design is an EC2 instance running a LAMP stack. I am working on a project right now, but my concern is that I am limited by my 1 server experience? I know the gist of scaling, (traffic routing, memcaching, sharding) but I don't really know how to go about setting it up. How do you even predict if you need to scale like that? Can scaling be done in a modular fashion? I want to learn about web scale :) Any pointers in the right direction would be greatly appreciated. If you could mention the full stack solution and not just one or two technologies that would be good.

Thanks in advance!

Udacity's Web Application Engineering course is pretty good.

http://www.udacity.com/view#Course/cs253/CourseRev/apr2012/U...

Particularly, Unit 6 and Unit 7 talk about scaling.

The course uses GAE, but the concepts apply everywhere. The professor is Steve Huffman who has practical experience scaling Reddit and now Hipmunk.

The best way to learn web scalability is to work at companies which have serious scaling problems. They tend to both a) develop a lot of organic knowledge about scaling, including by pushing the state of the art forward and b) they almost definitionally have more money than God, and as a consequence can spend copious money on hiring and training.

One thing companies with serious scaling problems will do is send you to places like e.g. JavaOne (and many, many more specialized conferences besides), where in the presentations you'll listen to e.g. LinkedIn talk about how to make writes immediately consistent to the originating user and then fan them out to other users over the course of the next few minutes, and what this requirement does to a three-tier architecture.

There's also a lot online. High Scalability will get mentioned, and I tend to keep my ears to the ground for interesting conferences and then see if they post videos and/or slide decks (SlideShare is wonderful for these).

OK, that's the answer you want. Here's the answer you need: you are overwhelmingly unlikely to have scaling problems. Single servers are big-n-beefy things, even single VPSes. Depending on what you're doing with it, a single EC2 instance running a non-optimized LAMP stack can easily get you to hundreds of thousands of users served, and/or millions of dollars of revenue. The questions of a) where do I get hundreds of thousands of users? or b) how do I convince businesses to pay $100k a month for my software? are of much more immediate and pressing concern to you than how you would take that very successful business to 10x or 100x its hypothetical size.

(Much like "I'm concerned my experience with managing a household budget will not prepare me for the challenges posed by a 9 figure bank account after I exit. How will I ever cope? Where can I learn about the tax challenges and portfolio allocation strategies of the newly wealthy?" the obvious answer sounds like "You'll hire someone to do that for you. Now, get closer to exiting than 'Having no product and no users' and worry about being a hundred-millionaire some other day.")

Until or unless you don't face scaling problems you won't e able to learn it. Luckily my project's had scaling problems in its core. Every design discussion we had was always having some "scaling caution" because we know we will be making a technical liability of we had to just solve the business problem. Essentialy, you need such projects or projects thats would be anticipating scaling issues in long run would be right projects to work for. Every learning you carry from projects and companies on scaling never fits to another scaling problem. In other words, you just can't use some tools to solve the problem. For example, just by knowing to use HBase or Cassandara or scalable technologies you would never be able to call you as a "scaling expert". It is much more beyond tools knowledge like trying to understand cost around a scalable idea, maintainability etc.,. I just scratched the surface, I can talk another twenty pages.

Another way to answer your question would be - think how FB might have implemented "people you may know" widget. Do they need to traverse thru 900m users to check whether you are related them in some way? Is that possible to do so everytime page gets rendered? Certainly not. There is some other way to do this. So taking up such questions and trying to find answers helps to get motivated on getting a feel of how scaling could be done? May be trying to make such POC would lead you understand technologies in problem specific approach.

Just my thoughts. Can't go back and review my typing as iPhone and HN doesn't go well. Will edit this for more clarity when am near to desk.

Learn by doing. And don't worry too much about it until the need comes. Luckily, the problem itself implies you're doing well, so the effort of dealing with it when the time comes will pay for itself.