Ask HN: How do you make sure your servers are up as a single founder? • Hacker News Board

Re: Ask HN: How do you make sure your servers are up as a single founder?

#121

Post by CoolGuySteve » Wed, Nov 06, 2019, 9:19 PM UTC

I currently run a batch of trading servers solo. The trading system is a C++ process with an asynchronous logger that prints log levels and times. One of the issues with trading is that you're dependent on your datafeed and exchange connections working which is out of your control.

I use a python monitoring script that tails logs watching for ALERT level log lines and constant order activity combined with a cron watchjob to ensure the process is alive during trading hours. The exception handler in the monitoring script sends alerts if the script itself dies.

If there are any issues I use twilio to text me the exception text/log line. I also use AWS SES to email myself but getting gmail to permanently not block SES is a pain in the ass. By design Twilio + AWS SES are the only external dependencies I have for the monitoring system (too bad SES sucks).

On my phone I have Termius SSH setup so I can log in and check/fix things. I have a bunch of short aliases in my .profile on the trading server to do the most common stuff so that I can type them easily from my phone.

I also do all my work through a compressed SSH tmux including editing and compiling code. So if things get hairy I can pair my phone with my laptop, attach to the tmux right where I left off, and fix things over even a 3G connection.

This compressed SSH trick is a huge quality of life improvement compared to previous finance jobs I've worked where they use Windows + Citrix/RDP just to launch a Putty session into a Linux machine. It's almost like finance IT has never actually had to fix anything while away from work.

Re: Ask HN: How do you make sure your servers are up as a single founder?

#122

Post by randlet » Wed, Nov 06, 2019, 9:22 PM UTC

jmstfv wrote: post #78

I am a solo founder of a website monitoring SaaS [0]. Theoretically, my uptime should be higher than that of my customers'. Here are a few things that I found helpful in the course of running my business: * Redundancy. If you process background jobs, have multiple workers listening on the same queues (preferably in different regions or availability zones). Run multiple web servers and put them behind a load balancer.…

With regards to AWS RDS & EC2 multi-az deploys and cost...buy reserved instances! It is much cheaper if you can afford to pay up front for 1yr or 3 years at a time.

Re: Ask HN: How do you make sure your servers are up as a single founder?

#123

Post by janee » Wed, Nov 06, 2019, 9:22 PM UTC

Imo you will have to get outsourced on-call if your downtime tolerance is very very low.

Otherwise I'd suggest religiously documenting your outage root causes and contemplating hard what could've avoided that outcome.

Then lastly for monitoring on the cheap:

Sentry.io - alerts.

Opsgenie - on-call management.

Heroku+new relic - heartbeat & performance.

Tldr; Keep your stack small and nimble and try to learn from past outages

Re: Ask HN: How do you make sure your servers are up as a single founder?

#124

Post by jmstfv » Wed, Nov 06, 2019, 9:23 PM UTC

rurp wrote: post #112

jmstfv wrote: post #78

I am a solo founder of a website monitoring SaaS [0]. Theoretically, my uptime should be higher than that of my customers'. Here are a few things that I found helpful in the course of running my business: * Redundancy. If you process background jobs, have multiple workers listening on the same queues (preferably in different regions or availability zones). Run multiple web servers and put them behind a load balancer.…

This is a great post, thanks. On your last point, do you mind sharing what you use to monitor the logs and send out alerts?

I send my logs to Papertrail and they have a feature that will notify you if your logs match a predefined pattern. Some of the patterns that I use:

* Account deletion (DELETE FROM \"accounts\")

* New successful sign up (INSERT INTO "accounts")

* New signup attempt (Started POST "/signup") [0]

* Fatal Exception Error ("fatal" OR "exception" OR "error")

* Warning ("warn")

I do share my costs and tools publicly, in case if you're interested: https://tryhexadecimal.com/costs

[0] I got hit by a botnet attack couple weeks ago, so I keep a tab on their activity (https://news.ycombinator.com/item?id=21327416)

Re: Ask HN: How do you make sure your servers are up as a single founder?

#125

Post by jmstfv » Wed, Nov 06, 2019, 9:25 PM UTC

randlet wrote: post #122

jmstfv wrote: post #78

I am a solo founder of a website monitoring SaaS [0]. Theoretically, my uptime should be higher than that of my customers'. Here are a few things that I found helpful in the course of running my business: * Redundancy. If you process background jobs, have multiple workers listening on the same queues (preferably in different regions or availability zones). Run multiple web servers and put them behind a load balancer.…

With regards to AWS RDS & EC2 multi-az deploys and cost...buy reserved instances! It is much cheaper if you can afford to pay up front for 1yr or 3 years at a time.

I do consider that but I'm still early in this game. My AWS spending is $1.38/day on EC2 and $1.05/day on RDS, all covered by AWS credit: https://tryhexadecimal.com/costs

Re: Ask HN: How do you make sure your servers are up as a single founder?

#126

Post by apankrat » Wed, Nov 06, 2019, 9:30 PM UTC

krat0sprakhar wrote: post #114

jmstfv wrote: post #78

I am a solo founder of a website monitoring SaaS [0]. Theoretically, my uptime should be higher than that of my customers'. Here are a few things that I found helpful in the course of running my business: * Redundancy. If you process background jobs, have multiple workers listening on the same queues (preferably in different regions or availability zones). Run multiple web servers and put them behind a load balancer.…

Offtopic: Really love how clean and readable the website is! Great work!

Indeedly!

Very clean and simple, but not trivial or bare. Really good job, jmstfv.

Re: Ask HN: How do you make sure your servers are up as a single founder?

#127

Post by CoolGuySteve » Wed, Nov 06, 2019, 9:35 PM UTC

scarface74 wrote: post #34

hellcow wrote: post #18

The only way to achieve high availability is to have redundancy of all things. Random things will go wrong that you can't predict. Boxes will die suddenly and without reason, even after months of working fine without changes, and always at the worst possible moment. Your system needs to be built to withstand that. I'll take the opposite approach of everyone here and recommend against serverless, kubernetes, and Herok…

No you don’t need to understand your infra from the ground up - especially as a solo founder. You should offload as much of the grunt work as you can afford to so you can concentrate on your business domain. If something “goes wrong” or you don’t understand how to implement something with managed services, support is just a ticket and a live chat/phone call away. I can speak from personal experience that AWS business…

It depends on the service I think. I've had ECS errors that took AWS support days to figure out (turns out some permission quota thing was overriding some ECS thing).

All in all, I think maybe I might have to find some other batch processing system.

Re: Ask HN: How do you make sure your servers are up as a single founder?

#128

Post by threeseed » Wed, Nov 06, 2019, 9:38 PM UTC

randlet wrote: post #122

jmstfv wrote: post #78

I am a solo founder of a website monitoring SaaS [0]. Theoretically, my uptime should be higher than that of my customers'. Here are a few things that I found helpful in the course of running my business: * Redundancy. If you process background jobs, have multiple workers listening on the same queues (preferably in different regions or availability zones). Run multiple web servers and put them behind a load balancer.…

With regards to AWS RDS & EC2 multi-az deploys and cost...buy reserved instances! It is much cheaper if you can afford to pay up front for 1yr or 3 years at a time.

For EC2 you really should be using spot fleets and making sure your architecture can support instances coming and going.

Re: Ask HN: How do you make sure your servers are up as a single founder?

#129

Post by eps » Wed, Nov 06, 2019, 9:49 PM UTC

Another option is just to not tackle systems that require 24/7 uptime IF you are just one person. Instead, make an installable product or do a service that's not interactive or real-time.

I've been in the game for a while and every time I run across an idea for a service, there's always a question of whether I'd be OK with sleeping with a pager, remoting to the servers at 4 am on Saturday and generally be slaved to the business. The answer, upon some reflection, is inevitably No. This is the domain of teams.

Re: Ask HN: How do you make sure your servers are up as a single founder?

#130

Post by jillesvangurp » Wed, Nov 06, 2019, 9:50 PM UTC

Short answer: promising 5 nines of uptime is not a thing for startups. Downtime is going to happen and you are going to be asleep, drunk, or otherwise not fit for doing any emergency ops. It's not the end of the world. Happens to the best of us.

So given that, just do the right things to prevent things going down and get to a reasonable level of comfort.

I recently shut down the infrastructure for my (failed) startup. Some parts of that had been up and running for close to four years. We had some incidents over the years of course but nothing that impacted our business.

Simple things you can do: - CI & CD + deployment automation. This is an investment but having a reliable CI & CD pipeline means your deployments are automated and predictable. Easier if you do it from day 1. - Have good tests. Sounds obvious but you can't do CD without good tests. Writing good tests is a good skill to have. Many startups just wing it here and if you don't get the funding to rewrite your software it may kill your startup. - Have redundancy. I.e. two app servers instead of 1. Use availability zones. Have a sane DB that can survive a master outage. - Have backups (verified ones) and a well tested procedure & plan for restoring those. - Pick your favorite cloud provider and go for hosted solutions for infrastructure that you need rather than saving a few pennies hosting shit yourself on some cheap rack server. I.e. use Amazon RDS or equivalent and don't reinvent the wheels of configuring, deploying, monitoring, operating, and backing that up. Your time (even if you had some, which you don't) is worth more than the cost of several years of using that even if you only spend a few days on this. There's more to this stuff than apt-get install whatever and walking away. - make conservative/boring choices for infrastructure. I.e. use postgresql instead of some relatively obscure nosql thingy. They both might work. Postgresql is a lot less likely to not work and when that happens it's probably because of something you did. If you take risks with some parts, make a point of not taking risks with other parts. I.e. balance the risks. - When stuff goes wrong, learn from it and don't let it happen again. - Manage expectations for your users and customers. Don't promise them anything you can't deliver. Like 5 nines. When shit goes wrong be honest and open about it. - Have a battle plan for when the worst happens. What do you do if some hacker gets into your system or your data-center gets taken out by a comet or some other freak accident? Who do you call? What do you do? How would you find out? Hope for the best but definitely plan for the worst. When your servers are down, improvising is likely to cause more problems.