Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you make sure your servers are up as a single founder?

news.ycombinator.com

Post reply on HN
Print view View original

181–190 of 233 posts

Re: Ask HN: How do you make sure your servers are up as a single founder?

#181

Post by nlg »

kevinyun wrote: post #39
nlg wrote: post #23

I agree in general with the responses encouraging better usage of managed platforms. I've run a SaaS app for a couple of years using a combination of AWS Elasticbeanstalk (Flask and Django) and AWS Lambda. Server resource related downtime has been minimal and recovery is quick/automated. Even hosting on Lambda you can run into issues without layers of redundancy (Lambda may be fine but a Route 53 outage would prevent…

We are considering Datadog, and nothing else seems to compare to them, but they seem extremely expensive. As a small startup/solo founder, did your implementation justify costs?

Yes, it's been an incredibly valuable tool for me. I get the most out of the host monitoring and the logging / alerting. The APM is nice, but I'm not using it nearly as much as the other pieces.

Just getting logging centralized alone has saved me tons of time, which is in turn more time spent on the product. I've been able to use the log parsing to setup metrics that tell me when an outside integration is acting up and isolate which paths. Take a day to really learn how their logs work and you'll be able to generate metrics / advanced event alerting in no time.

I was hesitant to pay the premium, but the peace of mind has been worth it. You can piece the same thing together with open source tooling. But then you've got another thing to manage.

Re: Ask HN: How do you make sure your servers are up as a single founder?

#182

Post by drubenstein »

> Can I pay someone to monitor my AWS deploy and make sure it's healthy?

Yes. There are consulting shops that will do this, as will many of the monitoring tools listed in the thread (though these tools will not fix the problem for you). Broadly speaking, there is a cost associated with this, as well as the cost associated with your downtime. If the cost of your downtime (reputational risk, SLA credits, etc) outweighs the cost of hiring someone to cut your MTTR to 5 minutes (assuming you can playbook out all of the relevant scenarios) + provides some value in stress reduction, then you should do this. If you've been doing this a while, you can math it out. In what experience I've had though, an outside person is unlikely to be able to fix an "unknown unknown", they just won't know your environment as well as you will.

All that said, one hour of service interruption a year is still better than most.

Re: Ask HN: How do you make sure your servers are up as a single founder?

#183

Post by gspetr »

apankrat wrote: post #110

> I was able to sign in to the AWS console and resolve the issue Kids these days. I had a RAM stick fry in one of the physical machines sitting in a colo 1 hour drive away. Not die, but just start flipping bits here and there, triggering most bizarre alerts you can imagine. On the night of December 24th. Now, that was fun. --- To add --- If you are a single founder - expect downtime and expect it to be stressful. Inh…

Isn't this exactly the case where you could have avoided this hassle entirely had you shelled out some cash for ECC memory?

Re: Ask HN: How do you make sure your servers are up as a single founder?

#184

Post by vandershraaf »

My applications which are built using Laravel are deployed through Laravel Forge. There is definitely extra charge for it, but having Forge to simplify deployment really save my time especially in case of any issue.

For monitoring, I am using Stackdriver which has easy-to-use health check.

Re: Ask HN: How do you make sure your servers are up as a single founder?

#186

Post by taf2 »

gspetr wrote: post #183
apankrat wrote: post #110

> I was able to sign in to the AWS console and resolve the issue Kids these days. I had a RAM stick fry in one of the physical machines sitting in a colo 1 hour drive away. Not die, but just start flipping bits here and there, triggering most bizarre alerts you can imagine. On the night of December 24th. Now, that was fun. --- To add --- If you are a single founder - expect downtime and expect it to be stressful. Inh…

Isn't this exactly the case where you could have avoided this hassle entirely had you shelled out some cash for ECC memory?

Live and learn is what I think the take away of this story is all about... I had a server fail dec 25 mid morning. It caused failures in away I had thought about before because instead appearing completely dead it was alive enough to not let go of any tcp connections. For the critical component in question, I didn’t have the correct timeouts in place... so as the single operator I was fortunate that my wife was also my co founder and so was a bit more understanding.

Re: Ask HN: How do you make sure your servers are up as a single founder?

#187

Post by cagmz »

jmstfv wrote: post #78

I am a solo founder of a website monitoring SaaS [0]. Theoretically, my uptime should be higher than that of my customers'. Here are a few things that I found helpful in the course of running my business: * Redundancy. If you process background jobs, have multiple workers listening on the same queues (preferably in different regions or availability zones). Run multiple web servers and put them behind a load balancer.…

Just curious; what do you use Redis for? Thanks for your post!

Re: Ask HN: How do you make sure your servers are up as a single founder?

#189

Post by [deleted] »

taf2 wrote: post #186
gspetr wrote: post #183

Earlier quoted context omitted.

Isn't this exactly the case where you could have avoided this hassle entirely had you shelled out some cash for ECC memory?

Live and learn is what I think the take away of this story is all about... I had a server fail dec 25 mid morning. It caused failures in away I had thought about before because instead appearing completely dead it was alive enough to not let go of any tcp connections. For the critical component in question, I didn’t have the correct timeouts in place... so as the single operator I was fortunate that my wife was also…

[deleted]

Re: Ask HN: How do you make sure your servers are up as a single founder?

#190

Post by o-__-o »

jmstfv wrote: post #117
tnolet wrote: post #109

Earlier quoted context omitted.

This is is scary. This is exactly how I run my (also monitoring) SaaS and it shows that the OP has learned how to minimize risk and prepare for the worst. You are married to your mobile & laptop. I also shows there is no free lunch: you need to invest in redundancy.

From time to time I ponder about all the terrible things that could happen: what can go wrong will probably go wrong so it helps to be (kinda) prepared.

As a single owner (of also a monitoring SaaS) I am currently putting out a fire where my primary datacenter died. I have suffered data loss even with precautions I had in place, one server lost all filesystems which took out my git repos.

I have backups, I have clones. I've still been in partial outage for 4 days and will be fully up tomorrow when I literally drive my servers to a new DC. Surprisingly I have slept 8 hours every night and I'm not worried. I've been in contact with my customers and provided solutions to keep them alive. If they leave they're going to leave, nothing I can do. I am looking to make sure everything is built uniformly (the server that died was the last of an old build process) and invest in scaling to the cloud in a bit more efficient and orderly manner.

Post reply on HN