What does it take to run a web app with 5K – 10K users?

That server runs happily as a single servo on http://modulus.io with absolutely no need for intervention on my part.

The rest of the application has similar requirements. I have one micro-equivalent server running the front-end, one the api and one the thumbnail generation. In general, this requires no hand-holding by me.

If your site is not processing or memory-intensive it should be feasible to scale to 10k users with a single $5/month instance on DigitalOcean or an equivalent level server on Heroku or Modulus or GCE.

Good luck attracting your first 5k users!

1: https://github.com/filearts/plunker_run/)

However, the biggest piece to scaling your application is the automation of everything you possibly can so that you can scale when you need to. You're going to be in a bit of pain if you need to scale everything manually.

Here's a few things I automate using Jenkins:

* Creation of web application servers(whether it be Puppet, Chef or Ansible, etc) make sure you can bring up a new node quickly and scale your app layer horizontally. Ideally automate the addition of this node to the LB.

* Data store backup/restores to all staging environments on a schedule(tests backups/restores) this is done using some custom code and the Backup gem. This way your dev team has access to an env that closely resembles prod and can resolve current prod bugs.

* External security scans using NMap (again using custom scripts). The jenkins job will fail if output is not as it expects. This way if we change a layer of our infrastructure we can know if something is exposed and shouldn't be.

* Static code analysis using Brakeman

Information you're going to need to scale your infra:

* Metrics on each one of your hosts. Use DataDog if you can afford it, integrates with all major systems and technologies. Great tool.

* Log collection via something like Logstash or Loggly and being able to visualize your application and web logs.

* Application response time measurements using something like NewRelic or building your own using StatsD and tracking the heck out of your application actions

Last but not least, have a plan for failure. While you're laying in bed at night, ask yourself these question:

* What would happen if the DB went poof? Can I restore it? How much data will I lose from the last backup? Will I know when this happens?

* What would happen if you're now being scanned by 30 ips from the netherlands, all of which are submitting garbage data into your forms. Are you protected against this? How will the added load effect your app layer? Do you have a way to automate the responses to those requests so as to deny them? This is a case of when, not if. Be ready.

* What would happen if my site gets put on Digg(lol)?

There's no magic bullet here. It's just practice, failure and learning from yours and others mistakes.

Good luck!

E.g. you have a script that does backups. You log the script's output, but one day something fails and the script is no longer executed.

Some form of dead man's handle is needed; the only way I can think of is to set up a monitoring service to check your log store for these entries every X hours.

Any alternatives?