Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you make sure your servers are up as a single founder?

news.ycombinator.com

Post reply on HN
Print view View original

201–210 of 233 posts

Re: Ask HN: How do you make sure your servers are up as a single founder?

#201

Post by type0 »

fanf2 wrote: post #154
apankrat wrote: post #110

> I was able to sign in to the AWS console and resolve the issue Kids these days. I had a RAM stick fry in one of the physical machines sitting in a colo 1 hour drive away. Not die, but just start flipping bits here and there, triggering most bizarre alerts you can imagine. On the night of December 24th. Now, that was fun. --- To add --- If you are a single founder - expect downtime and expect it to be stressful. Inh…

That reminds me of the time we had a DIMM actually melt on the 22nd December http://fanf2.user.srcf.net/hermes/doc/misc/orange-fire/

Thank you for sharing, that's a real nightmare before Christmas story!

Re: Ask HN: How do you make sure your servers are up as a single founder?

#202

Post by type0 »

randlet wrote: post #168
__d wrote: post #141

You might also want to consider some additional risks that are often overlooked: Billing issues. What happens if the credit card you use to pay for everything gets hijacked, and you're trapped with a blocked card trying to clean it up but your bank is taking their sweet time and won't give you another card until it's sorted? ALWAYS have a backup credit card. DNS Registrar. There's a hard SPOF in the DNS, where your r…

> What happens if the credit card you use to pay for everything gets hijacked, and you're trapped with a blocked card trying to clean it up but your bank is taking their sweet time and won't give you another card until it's sorted? This happened to me this week. Luckily bank got me the new card within 2 business days, but still was a bit stressful and I burned a day getting my payment info updated everywhere.

> What happens if the credit card you use to pay for everything gets hijacked, and you're trapped with a blocked card trying to clean it up but your bank is taking their sweet time and won't give you another card until it's sorted?

It doesn't even have to get hijacked, we had our bank reissuing cards because of some bug on the old chip, the card number stayed the same and the expiration dates changed but Google Cloud blocked the paying accounts. Security is the top priority sure, but I don't want some proprietary algorithms decide whether or not our server will be up or not. The fix was to move away from Google, because bouncing on their support took to long.

Re: Ask HN: How do you make sure your servers are up as a single founder?

#204

Post by timc3 »

thrownaway954 wrote: post #93

I know this is going to be down-voted to nonexistence since everyone now-a-days wants to serverless, AWS and what not. personally i've always used either hosting.com or inmotionhosting.com. yes they are more expensive than AWS and what not, but the thing is, they both have a support staff 24/7/365. I called whenever i need and have someone remote into my server and fix whatever is wrong. furthermore, i can even have…

Sounds like a very sensible solution.

Re: Ask HN: How do you make sure your servers are up as a single founder?

#205

Post by dTal »

daniel_iversen wrote: post #152
jmstfv wrote: post #78

I am a solo founder of a website monitoring SaaS [0]. Theoretically, my uptime should be higher than that of my customers'. Here are a few things that I found helpful in the course of running my business: * Redundancy. If you process background jobs, have multiple workers listening on the same queues (preferably in different regions or availability zones). Run multiple web servers and put them behind a load balancer.…

> Monitor your infrastructure and set up alerts [..] "fatal" OR "exception" OR "error" I almost have the regex "fatal|invalid|unknown|error|except|critical|cannot" in muscle memory many years after having last had to type it - must have typed it thousands of times tailing and grepping logs :- )

That's a smell. If you have to type a mouthful like that more than twice, put it in a script.

Re: Ask HN: How do you make sure your servers are up as a single founder?

#206

Post by mister_hn »

tachion wrote: post #20
mister_hn wrote: post #13

yes you can do. Or try to automatize as much as possible: - add health check mechanisms - if health check is broken => restart service - if restart service doesn't help after X retry => redeploy previous state (if any available) Try to use Kubernetes or Docker Swarm if possible, combined with Terraform

Restarting the service and redeploying it should be absolutely the last resort and aren't really sound advice, mainly, because you are losing the invaluable crashed state of the system, that may be vital (sometimes logs are not enough) to discover _why_ the system crashed in the first place and then delivering a fix for that particular issue. Once that's done, you incorporate this into your infrastructure automation…

> Restarting the service and redeploying it should be absolutely the last resort and aren't really sound advice, mainly, because you are losing the invaluable crashed state of the system, that may be vital

I assume you have a separated logging mechanism, where all logs are collected, independently from the restarted service. Don't forget to log the state of your system as much as possible for post-mortem analysis

Re: Ask HN: How do you make sure your servers are up as a single founder?

#207

Post by heliodor »

seized wrote: post #195
aspectmin wrote: post #76

FWIW - if you just want to make sure your services are up - consider: 1) pagerduty.com or uptimerobot.com for remote monitoring to make sure you site(s) are up (and get alerts when they're not). 2) Datadog or New Relic if you want deeper monitoring (application performance, database performance, diagnostics/debugging. 3) Rollbar.com (site doesn't seem to respond) for site performance/errors. 4) Roll your own with Pro…

StatusCake and PingMonit are two more good ones in the same vein as UptimeRobot. All have free tiers for testing and small scale/homelab use too.

There are plenty of options and plenty free pricing tiers. I use multiple ones for redundancy and comparison. One of them ended up producing false alerts, surprisingly.

Re: Ask HN: How do you make sure your servers are up as a single founder?

#208

Post by byteshock »

kevinyun wrote: post #39
nlg wrote: post #23

I agree in general with the responses encouraging better usage of managed platforms. I've run a SaaS app for a couple of years using a combination of AWS Elasticbeanstalk (Flask and Django) and AWS Lambda. Server resource related downtime has been minimal and recovery is quick/automated. Even hosting on Lambda you can run into issues without layers of redundancy (Lambda may be fine but a Route 53 outage would prevent…

We are considering Datadog, and nothing else seems to compare to them, but they seem extremely expensive. As a small startup/solo founder, did your implementation justify costs?

Datadog is the best all around tool with APM and log monitoring built in. I highly recommend it as it will be substantially cheaper than paying for separate monitoring, apm, logs, etc. The support is also a lot more receptive towards small business owners unlike other platforms such as New Relic.

Also if you send custom stats to datadog such as login activity, you can use their Anomaly detection to find suspicious behavior.

Disclaimer: we use datadog at my company and have tried all the other popular options. Hands down datadog is the most feature rich and user friendly.

Re: Ask HN: How do you make sure your servers are up as a single founder?

#209

Post by byteshock »

heliodor wrote: post #180
kevinyun wrote: post #39

Earlier quoted context omitted.

We are considering Datadog, and nothing else seems to compare to them, but they seem extremely expensive. As a small startup/solo founder, did your implementation justify costs?

Plenty of people are happy using the industry-standard open-source tools: - metrics system (Prometheus, InfluxDB, Graphite, etc.) - dashboards (Grafana) - alerts (both the metrics system and Grafana can handle this) If you're not a big-data company, you can self-host instances of these products reliably. Disclosure: My startup, https://HostedMetrics.com , provides turnkey hosted versions of these software packages. I…

Your product doesn’t even come close to what datadog offers....

Re: Ask HN: How do you make sure your servers are up as a single founder?

#210

Post by [deleted] »

o-__-o wrote: post #190
jmstfv wrote: post #117

Earlier quoted context omitted.

From time to time I ponder about all the terrible things that could happen: what can go wrong will probably go wrong so it helps to be (kinda) prepared.

As a single owner (of also a monitoring SaaS) I am currently putting out a fire where my primary datacenter died. I have suffered data loss even with precautions I had in place, one server lost all filesystems which took out my git repos. I have backups, I have clones. I've still been in partial outage for 4 days and will be fully up tomorrow when I literally drive my servers to a new DC. Surprisingly I have slept 8…

[deleted]
Post reply on HN