Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you make sure your servers are up as a single founder?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 233 posts

Re: Ask HN: How do you make sure your servers are up as a single founder?

#41

Post by kd5bjo »

dkersten wrote: post #14

Most of the suggestions here is suggesting ways of restarting services when they go down, which is a good start, but that doesn't actually solve the issue I hit last night... My system integrates with an external system and what happened is this external system started sending me unexpected data, which my system wasn't able to handle, because I didn't expect it so never thought to test for it -- the issue was that I…

For situations like this, it's extremely useful to have a human-review queue that the automated system can drop jobs into. It keeps the rest of the system running, and lets the support staff (you) fix the problem during normal business hours.

This is how we handled errors in our video archiving system at Justin.tv, and to my knowledge we never lost a single frame that made it to the broadcast servers. The raw bits were streamed to disk as they came in, and only got removed once the VOD servers had the final version— any errors would retry a few times and then get flagged for manual processing. We did have a few close calls where something broke the whole archiving system, and the broadcast servers came dangerously close to shutting down due to full disks, though.

Re: Ask HN: How do you make sure your servers are up as a single founder?

#43

Post by asdkhadsj »

nathan_f77 wrote: post #28

I'm working on FormAPI [1] as a solo founder. I started on Heroku, but Heroku was a bit unreliable and I had some random outages that I couldn't predict or control. (This was even while using dynos in their professional tier.) I also had a lot of free AWS credits, so I migrated to AWS. I didn't want to write all my terraform templates from scratch, so I spent a lot of time looking for something that already existed,…

Convox looks nice! My only issue is that it seems tied to AWS?

It would be nice to have a stateless tool abstract Terraform a bit, to let you use more providers as a basic PaaS.

Re: Ask HN: How do you make sure your servers are up as a single founder?

#44

Post by asdkhadsj »

ransom1538 wrote: post #32

1. Stay on AWS only. 2. Pay for a Business Support plan. https://aws.amazon.com/premiumsupport/pricing/ 3. Call business support about something "how do I restart my server" - so you know how to file a ticket, get a feel for how quick the response is and how it works. Do not over think this. EG: terraform templates

re: terraform, that's only part of the picture right? What do you recommend for provisioning? I assume Terraform + Packer? I looked into these a while back and they seemed good.

My only concern was that my target was very low-cost setups, and I wanted something like Packer but let me provision multiple images onto a single machine. Eg, if I just used Packer, as far as I could tell I would have to have 1 machine per image. It sounds odd, but I didn't want to pay $5*Services, especially when the number of users and load was very small. Being able to deploy in a PaaS fashion to something like Docker on the machine seemed best.

But then I was looking at Terraform + Packer + DockerSomething, and things went back to feeling less simple.

Re: Ask HN: How do you make sure your servers are up as a single founder?

#45

Post by Beltiras »

Redundancy. Failure should always be an option. Specific answers will depend on your stack. Nobody will be able to monitor and react like you will because all IT solutions are their own species of butterfly with their own intricacies. If uptime is really that important you might be at the stage where you need to take on an employee.

Re: Ask HN: How do you make sure your servers are up as a single founder?

#46

Post by gamache »

kevinyun wrote: post #39
nlg wrote: post #23

I agree in general with the responses encouraging better usage of managed platforms. I've run a SaaS app for a couple of years using a combination of AWS Elasticbeanstalk (Flask and Django) and AWS Lambda. Server resource related downtime has been minimal and recovery is quick/automated. Even hosting on Lambda you can run into issues without layers of redundancy (Lambda may be fine but a Route 53 outage would prevent…

We are considering Datadog, and nothing else seems to compare to them, but they seem extremely expensive. As a small startup/solo founder, did your implementation justify costs?

$15 per server per month isn't that extreme. I've used Datadog in production for 3+ years (starting from a headcount of 10) and can honestly say they've been worth every penny. Observability is irreplacable.

Re: Ask HN: How do you make sure your servers are up as a single founder?

#48

Post by bszupnick »

hellcow wrote: post #18

The only way to achieve high availability is to have redundancy of all things. Random things will go wrong that you can't predict. Boxes will die suddenly and without reason, even after months of working fine without changes, and always at the worst possible moment. Your system needs to be built to withstand that. I'll take the opposite approach of everyone here and recommend against serverless, kubernetes, and Herok…

As a solo bootstrapping technical founder, I am in LOVE with Heroku.

My product is single-tenant, which can be tough infrastructure-wise since each app/customer needs a cluster of servers and services (Postgres and RabbitMQ). The Heroku pipelines enables me to have a testing app, staging app, and when I want to push staging to production, I can push the code to all my single-tenant production apps with the push of a button.

In theory I could do this with Bitbucket or Gitlab CI/CD pipelines, but this enabled me to focus on app development instead of devops.

That's just my preference, of course.

Re: Ask HN: How do you make sure your servers are up as a single founder?

#49

Post by Clubber »

dkersten wrote: post #35
Clubber wrote: post #21

Earlier quoted context omitted.

Defensive programming (or I call it fail-safe programming) is a must for any type of service / daemon. I employ a healthy dose of exception trapping and logging, and I get an email whenever it happens. People aren't perfect, but you can anticipate a lot of failures. It usually involves bad data as in your case. Each time you get bitten, change your code so it fails gracefully.

Agreed. Be as defensive as possible, trap all exceptions (this allowed me to identify the problem and fix it very quickly, but I still had to step in and fix it) and validate absolutely everything no matter how unlikely it seems. Also go over every system and ask "what if an unexpected error happens, will it take the system down? will it prevent other requests/tasks/users from working?"

Also, if you are running on linux, check out supervisor. I use it to keep my WebAPI's up. If it crashes, it starts it up again.

Re: Ask HN: How do you make sure your servers are up as a single founder?

#50

Post by scarface74 »

jasonkester wrote: post #36

I build my stuff on top of a stack that hardly ever goes down. All my SaaS products run on a Windows server, with SQL Server as a database and ASP.NET on IIS running the public sites. You can probably come up with a lot of uncharitable things to say about those technologies, but "flimsy" and "fragile" likely aren't in the list. As a result, when things go seriously wrong, the application pool will recycle itself and…

I agree. I’ve spent close to 20 years hosting on IIS and have never had downtown attributable to it - especially since the app pool will recycle. Most of the issues I’ve seen were caused by people who didn’t know how to use EF correctly or badly tuned databases/queries.

Even now, the only “Linux deployments” I do are either with Lambda or Fargate (Serverless Docker). I just don’t like managing servers. These days even my EC2 instances are disposable.

Post reply on HN