Heroku. I just pay for the privilege of not thinking (almost) about such issues.
Also services like platform.sh and others.
Ask HN: How do you make sure your servers are up as a single founder?
71–80 of 233 posts
Re: Ask HN: How do you make sure your servers are up as a single founder?
#72I guess it really depends on what AWS services you use. There are companies that can manage your AWS 24/7 for a fee.
Other options includes using another service that offers 24/7 uptime. Obviously you pay more for that.
Re: Ask HN: How do you make sure your servers are up as a single founder?
#73I build my projects on Google App Engine and it has been stable and reliable without much administration. The platform is not without its challenges, especially with the Gen 2 rollout, but no issues related to administration/interruption. PaaS could be a good place to explore...
seconding this, if you don't need lots of resources it makes sense. I pair it with Amazon cloudfront and so far, almost one year in with zero problems. By far the biggest win for me is the peace of mind
Re: Ask HN: How do you make sure your servers are up as a single founder?
#74Most of the suggestions here is suggesting ways of restarting services when they go down, which is a good start, but that doesn't actually solve the issue I hit last night... My system integrates with an external system and what happened is this external system started sending me unexpected data, which my system wasn't able to handle, because I didn't expect it so never thought to test for it -- the issue was that I…
Always treat third-party systems like they're full of nitroglycerin. Double check all response codes, expect the unexpected, degrade gracefully when it hits the fan. You're always better off serving up a nice 500 error page than spinning forever or returning a false positive to users. And make sure you have a clear SLA with them and can escalate/mitigate/compensate when they don't fulfill it.
Re: Ask HN: How do you make sure your servers are up as a single founder?
#75I use uptime robot http://uptimerobot.com for monitoring, they have a free plan or paid if you want faster checks. If it's truly critical to have no down time then you probably need to build that resilience in to your architecture.
+1 for UptimeRobot. Learned about it right here on HN:
Re: Ask HN: How do you make sure your servers are up as a single founder?
#76FWIW - if you just want to make sure your services are up - consider:
1) pagerduty.com or uptimerobot.com for remote monitoring to make sure you site(s) are up (and get alerts when they're not).
2) Datadog or New Relic if you want deeper monitoring (application performance, database performance, diagnostics/debugging.
3) Rollbar.com (site doesn't seem to respond) for site performance/errors.
4) Roll your own with Prometheus (https://prometheus.io/, or Nagios (https://www.nagios.org/)/IcingA. Or... strangely - I still use MRTG for a few perf monitoring things: https://oss.oetiker.ch/mrtg/
5) If you want to monitor the status of deploys/builds - I love integrating CI/CD systems with Slack - very helpful.
Hope that helps - I've spent a lot of my career monitoring things, and have this mantra that I need to know about services down before customers call to tell me same.
(a lot of these have free tiers)
Re: Ask HN: How do you make sure your servers are up as a single founder?
#77Ive struggled with this for years. AWS is not foolproof and with environments for web, Android, amd ios availability gremlins have killed much of my spirit despite users proclaiming how they've been looking like a service like mine for yrs.
Docker, elastic beanstalk, SNS, and the hidden world of AWS instance performance are all a PITA. Oh yea, certs...
Welcome help as well.
Re: Ask HN: How do you make sure your servers are up as a single founder?
#78I am a solo founder of a website monitoring SaaS [0]. Theoretically, my uptime should be higher than that of my customers'. Here are a few things that I found helpful in the course of running my business:
* Redundancy. If you process background jobs, have multiple workers listening on the same queues (preferably in different regions or availability zones). Run multiple web servers and put them behind a load balancer. If you use AWS RDS or Heroku Postgres, use Multi-AZ deployment. Be mindful of your costs though, because they can skyrocket fast.
* Minimize moving parts (e.g. databases, servers, etc..). If possible, separate your marketing site from your web app. Prefer static sites over dynamic ones.
* Don't deploy at least 2 hours before you go to sleep (or leave your desk). 2 hours is usually enough to spot botched deploys.
* Try to use managed services as much as possible. As a solo founder, you probably have better things to focus on. As I mentioned before, keep an eye on your costs.
* Write unit/integration/system tests. Have a good coverage, but don't beat yourself up for not having 100%.
* Monitor your infrastructure and set up alerts. Whenever my logs match a predefined regex pattern (e.g "fatal" OR "exception" OR "error"), I get notified immediately. To be sure that alerts reach you, route them to multiple channels (e.g. email, SMS, Slack, etc..). Obviously, I'm biased here.
I'm not gonna lie, these things make me anxious, even to this day (it used to be worse). I take my laptop everywhere I go and make sure that my phone is always charged.
Re: Ask HN: How do you make sure your servers are up as a single founder?
#79Most of the suggestions here is suggesting ways of restarting services when they go down, which is a good start, but that doesn't actually solve the issue I hit last night... My system integrates with an external system and what happened is this external system started sending me unexpected data, which my system wasn't able to handle, because I didn't expect it so never thought to test for it -- the issue was that I…
I used to run a service solo that processed data from multiple external sources and as you said you need to program defensively when dealing with input.
I handled it with a pipeline that did the following, 1. validation, 2. transform data if needed, 3. load the data. If validation failed, the data would get "quarantined" and I would get an email notification, or slack notification if urgent.
I don't generally write a lot of unit tests, but all my validation and transformation logic would have 100% coverage because things always change and you need to make sure future updates never break the system
Re: Ask HN: How do you make sure your servers are up as a single founder?
#80Back when I was working on everything myself, I deployed everything through AWS Lambda and API Gateway, with all my static assets on S3 and CloudFront. I had exactly zero infrastructure issues over the course of two years and never dealt with security patches, SSH'ing, etc. If I were doing billions of requests, it may not have been the most cost effective, but it helped me scale without worrying about typical devops…
How do you keep things fast?
Lambda functions can have cold starts that introduce latency. How do you manage that?
(From my small amount of experience - please prove me wrong.)