Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you make sure your servers are up as a single founder?

news.ycombinator.com

Post reply on HN
Print view View original

211–220 of 233 posts

Re: Ask HN: How do you make sure your servers are up as a single founder?

#211

Post by sphix0r »

nh2 wrote: post #147

I rent dedicated servers at Hetzner. No cloud machines, no hosted cloud services for production beyond DNS. * 3 machines in separate data centers (equivalent of AWS AZs) for >= 30 EUR/month each. ECC RAM. * These machines are /very/ reliable. Uptime of > 300 days are common, reboots happen only for the relevant kernel updates. * Triple-redundancy Postgres synchronous replication with automatic failover (using Stolon)…

Very well-thought infra and nice metrics. What kind of application are you running if I may ask?

Re: Ask HN: How do you make sure your servers are up as a single founder?

#212

Post by nickthemagicman »

scarface74 wrote: post #101
nickthemagicman wrote: post #55

Earlier quoted context omitted.

Did you ever have any problems with the function concurrency limit?

It’s a soft limit. You can request a larger limit and get it increased within 30 minutes.

Awesome thanks! Sorry to bug but this is very interesting to me. Did you use a framework like Serverless?

Re: Ask HN: How do you make sure your servers are up as a single founder?

#213

Post by scarface74 »

nickthemagicman wrote: post #212
scarface74 wrote: post #101

Earlier quoted context omitted.

It’s a soft limit. You can request a larger limit and get it increased within 30 minutes.

Awesome thanks! Sorry to bug but this is very interesting to me. Did you use a framework like Serverless?

No. I’m of the belief that when you choose a platform, go all in. Given a choice between the Serverless platform and AWS’s Serverless Application Model, I would choose SAM.

I also work for a company that pays for the business support plan, they will help you to an extent with any weirdness in their own platform - but not with third party utilities.

Another advantage with using SAM, is that you can create and configure your lambda/API Gateway from the web console and then export your SAM/CloudFormation template.

All that being said, for APIs, I don’t use either. I would recommend using standard frameworks like C#/WebAPI, JS/Node, Python/Flask/Django instead and using the proxy integration libraries that AWS provides. It lets you develop/debug locally like you are accustom to and it gives you the optionality to move your APIs to Fargate (Serverless Docker) or EC2 instances without any code changes:

C#/Web API

https://aws.amazon.com/blogs/developer/deploy-an-existing-as...

Node:

https://github.com/awslabs/aws-serverless-express

Python/Flask

https://dev.to/apcelent/deploying-flask-on-aws-lambda-4k42

I’ve been told that there are similar frameworks for other languages but those are the three that I use.

If you have any other questions, feel free to email me. My address is in my profile. I’m not a consultant trying to sell anything....

Re: Ask HN: How do you make sure your servers are up as a single founder?

#214

Post by apankrat »

fanf2 wrote: post #154
apankrat wrote: post #110

> I was able to sign in to the AWS console and resolve the issue Kids these days. I had a RAM stick fry in one of the physical machines sitting in a colo 1 hour drive away. Not die, but just start flipping bits here and there, triggering most bizarre alerts you can imagine. On the night of December 24th. Now, that was fun. --- To add --- If you are a single founder - expect downtime and expect it to be stressful. Inh…

That reminds me of the time we had a DIMM actually melt on the 22nd December http://fanf2.user.srcf.net/hermes/doc/misc/orange-fire/

I hope you kept it ;)

Re: Ask HN: How do you make sure your servers are up as a single founder?

#215

Post by unoti »

daniel_iversen wrote: post #152
jmstfv wrote: post #78

I am a solo founder of a website monitoring SaaS [0]. Theoretically, my uptime should be higher than that of my customers'. Here are a few things that I found helpful in the course of running my business: * Redundancy. If you process background jobs, have multiple workers listening on the same queues (preferably in different regions or availability zones). Run multiple web servers and put them behind a load balancer.…

> Monitor your infrastructure and set up alerts [..] "fatal" OR "exception" OR "error" I almost have the regex "fatal|invalid|unknown|error|except|critical|cannot" in muscle memory many years after having last had to type it - must have typed it thousands of times tailing and grepping logs :- )

Instead of having a Regex that searches for error|critical|except it’s a good thing to have a log level in your logging infrastructure, so that you can query for example log level=2 and get all the bad things.

It takes a bit to work this into the code and infrastructure everywhere but it’s worth it.

Re: Ask HN: How do you make sure your servers are up as a single founder?

#216

Post by cagmz »

jmstfv wrote: post #199
cagmz wrote: post #187

Earlier quoted context omitted.

Just curious; what do you use Redis for? Thanks for your post!

Jobs (e.g. uptime checks, payment processing, sending emails) are added to the job queue (Redis) and background workers pick it up from there, and churn through them asynchronously. Most of the work happens in the background, so in my case, Redis is a critical piece of infrastructure. https://devcenter.heroku.com/articles/background-jobs-queuei...

Ah, of course! I saw background jobs and Redis but didn't make the connection at the time.

Re: Ask HN: How do you make sure your servers are up as a single founder?

#217

Post by bullen »

I made my own (every minute) monitor: http://monitor.rupy.se

I also warns me if the CPU load goes up over 80%.

For the first two years of going live I had this hardwired to my Pebble via real-time mail, but now I know my platform is robust; so I can choose worry about other things.

Re: Ask HN: How do you make sure your servers are up as a single founder?

#218

Post by ConnorLeet »

asadlionpk wrote: post #159

We are a very small team at https://codeinterview.io . We recently achieved a respectable level of reliability with a tiny team. Some things you should do: - Atleast have a pool of 2 instances (ideally per service) running under an auto-scaler or a managed K8s (GKE is best) with LB in front. May also want to explore EBS and google cloud run. If you can use them, use them! - Uptime alerts. pingdom (or newrelic alerts)…

One thing that has helped me a lot with monitoring is custom application-level metrics.

If you have a good idea of the usage patterns of your service, create metrics backed by the patterns. This can help you find things that CPU/Memory will hide.

Re: Ask HN: How do you make sure your servers are up as a single founder?

#219

Post by waxzce »

Very simple: you need to get higher level of service, instead of paying for servers you need to pay for up-time. It's what PaaS, managed service or serverless does : manage server for you, at scale. To have something online you need: - servers - VM/OS management - Scalability system - monitoring (hardware, OS, applicative & functional) - action on monitoring and escalation management - update every weeks - observability

That's what we provide at Clever Cloud BTW https://www.clever-cloud.com/

Re: Ask HN: How do you make sure your servers are up as a single founder?

#220

Post by wstuartcl »

tootie wrote: post #74
dkersten wrote: post #14

Most of the suggestions here is suggesting ways of restarting services when they go down, which is a good start, but that doesn't actually solve the issue I hit last night... My system integrates with an external system and what happened is this external system started sending me unexpected data, which my system wasn't able to handle, because I didn't expect it so never thought to test for it -- the issue was that I…

Always treat third-party systems like they're full of nitroglycerin. Double check all response codes, expect the unexpected, degrade gracefully when it hits the fan. You're always better off serving up a nice 500 error page than spinning forever or returning a false positive to users. And make sure you have a clear SLA with them and can escalate/mitigate/compensate when they don't fulfill it.

This. Write guards like the external integration is an active malicious adversary -- because when they change api, go down, have their own issues they may as well be attacking your integration.
Post reply on HN