Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using AWS Lambda to call and text you when your servers are down

thisdata.com

Post reply on HN
Print view View original

41–50 of 55 posts

Re: Using AWS Lambda to call and text you when your servers are down

#41

Post by teddyh »

snom380 wrote: post #40
teddyh wrote: post #39

Earlier quoted context omitted.

We’ve had issues with Pingdom at work. We don’t use them ourselves, but we host web sites, and some customer of ours used Pingdom to monitor their web site hosted on our servers. The customer would complain to us about downtime reported by Pingdom, but we would read the logs and find everything OK, with multiple successful accesses from other people during the time which Pingdom reported our customer’s site as being…

Doesn't services like Pingdom support multiple ping locations? If all of those fail, there's a very high chance there's an actual problem, if not with your server then with your (ISPs) connectivity.

If our ISP was down, we would not have had successful accesses from other people at the same time. If some transit ISP was down somewhere between us and Pingdom, well, that’s the Internet for you, eh? Regardless, Pingdom would report us as down, even though we weren’t at fault.

Re: Using AWS Lambda to call and text you when your servers are down

#42

Post by cddotdotslash »

robinson-wall wrote: post #36
cddotdotslash wrote: post #26

Earlier quoted context omitted.

Of course it can go down, and you can have CloudWatch alerts to alert you about that. But so can your Nagios server sending pings go down or the fancy SaaS you signed up for.

Did you just suggest using a third AWS service to let you know if the second AWS service monitoring your first AWS service goes down?

Yes, because they're different services running on different architecture and distributed differently. I challenge you to find one time in the past five years where CloudWatch was down at the same time as other services. Even if you can, I'm sure your custom built Nagios server in your datacenter has gone down as many or more times too.

But my bigger point here is that you're essentially asking "well how do you monitor your monitor?" At which point up the chain do you have enough? Also, I think the original post was simply a demo of what is possible. Yet whenever someone posts something, people go in the comments to belittle it. "Yeah, you built a monitoring solution... Well what happens if that goes down?"

Which is a legitimate question. But obviously if your production service is that critical to your business, you won't be monitoring it with a service that costs $0.0000002 per execution.

Re: Using AWS Lambda to call and text you when your servers are down

#43

Post by dalore »

teddyh wrote: post #41
snom380 wrote: post #40

Earlier quoted context omitted.

Doesn't services like Pingdom support multiple ping locations? If all of those fail, there's a very high chance there's an actual problem, if not with your server then with your (ISPs) connectivity.

If our ISP was down, we would not have had successful accesses from other people at the same time. If some transit ISP was down somewhere between us and Pingdom, well, that’s the Internet for you, eh? Regardless, Pingdom would report us as down, even though we weren’t at fault.

Yes you were down for some of your users. If that's ok for you that's fine. But if I were you I would be calling my ISP and trying to sort out why customers from location X can't access but customers from location Y can.

If you're providing a service to your users, and they say that the service is down using pingdom, you should be looking into, not just saying "Works on my machine".

Re: Using AWS Lambda to call and text you when your servers are down

#44

Post by falcolas »

cddotdotslash wrote: post #42
robinson-wall wrote: post #36

Earlier quoted context omitted.

Did you just suggest using a third AWS service to let you know if the second AWS service monitoring your first AWS service goes down?

Yes, because they're different services running on different architecture and distributed differently. I challenge you to find one time in the past five years where CloudWatch was down at the same time as other services. Even if you can, I'm sure your custom built Nagios server in your datacenter has gone down as many or more times too. But my bigger point here is that you're essentially asking "well how do you monit…

> they're different services running on different architecture and distributed differently

I think you underestimate the interdependency of services in AWS. Historically, if there were problems with S3 or EBS in us-east-1, you could expect the entire API to be flaky, and things like autoscaling to fail. These have been better distributed, but failures still cascade.

> I think the original post was simply a demo of what is possible

No, it wasn't a demo, it was an actual production issue. No alarms, no error logs, no way to tell it wasn't working other than someone noticing the queues were getting larger and contacting AWS.

> people go in the comments to belittle it

Only because the original project projects AWS Lambda as "the solution" for such problems, not realizing that it is just as fallible a solution as everything else.

> Well what happens if that goes down?

The solution to this is well known - two monitoring systems in physically separate locations that monitor each other as well as mission critical systems. Nagios, Icinga, and a dozen other well-tested solutions work remarkably well for these roles, yet people keep writing "new" solutions over and over and over.

> But obviously if your production service is that critical to your business, you won't be monitoring it with [this] service

Then what's it's value, other than as an intellectual exercise?

Re: Using AWS Lambda to call and text you when your servers are down

#45

Post by falcolas »

thisone wrote: post #31
falcolas wrote: post #3

And I'm sure Lambda will never go down. Right? Right?? (It has. Completely and silently stopped processing against Kinesis queues for a few hours recently. Guess what AWS Step is built on?)

don't worry, the status was green the entire time...

Yes, it was indeed green the entire time. Of course, AWS is almost always green, so long as something is up...

Re: Using AWS Lambda to call and text you when your servers are down

#46

Post by chinathrow »

teddyh wrote: post #41
snom380 wrote: post #40

Earlier quoted context omitted.

Doesn't services like Pingdom support multiple ping locations? If all of those fail, there's a very high chance there's an actual problem, if not with your server then with your (ISPs) connectivity.

If our ISP was down, we would not have had successful accesses from other people at the same time. If some transit ISP was down somewhere between us and Pingdom, well, that’s the Internet for you, eh? Regardless, Pingdom would report us as down, even though we weren’t at fault.

We've seen the same with Pingdom. ISPs are (mostly) multi-homed and Pingdom might use just one (affected) route to the ISP in question and then fail badly.

If Pingdom can't get to your site, it's highly likely your users can't either.

Re: Using AWS Lambda to call and text you when your servers are down

#47

Post by teddyh »

dalore wrote: post #43
teddyh wrote: post #41

Earlier quoted context omitted.

If our ISP was down, we would not have had successful accesses from other people at the same time. If some transit ISP was down somewhere between us and Pingdom, well, that’s the Internet for you, eh? Regardless, Pingdom would report us as down, even though we weren’t at fault.

Yes you were down for some of your users. If that's ok for you that's fine. But if I were you I would be calling my ISP and trying to sort out why customers from location X can't access but customers from location Y can. If you're providing a service to your users, and they say that the service is down using pingdom, you should be looking into, not just saying "Works on my machine".

Why should we be the ones to look into it? It was a random intermittent short-duration fault in the middle if the Internet, at some unknown place on the then-current path between us and Pingdom. Why should not Pingdom be at least equally as obligated to look into it? After all, they’re the ones actually using the failing connection, in order to monitor our and others’ services. But no, Pingdom simply report us as being down, and leave the hard part to us; i.e. the part where we have to explain to our customers that the Pingdom report is actually provably incorrect.

I mean, what qualifies as “being up”? If some random link in the middle of the Internet goes down, and you suddenly, for 30 seconds, are unreachable for the few hundred people going through that exact link because it happens to be the best path between those people and your server, can they claim that you have failed to provide adequate uptime? If such a fault happens, are you then responsible to troubleshoot it? I say no. The Internet is the ISP’s responsibility, and the only faults actually meaningful to report to your ISP are the repeatable or long-lasting ones. Small stuff like this is not worth anybody’s time (except ISPs) to go digging into.

Re: Using AWS Lambda to call and text you when your servers are down

#48

Post by cddotdotslash »

falcolas wrote: post #44
cddotdotslash wrote: post #42

Earlier quoted context omitted.

Yes, because they're different services running on different architecture and distributed differently. I challenge you to find one time in the past five years where CloudWatch was down at the same time as other services. Even if you can, I'm sure your custom built Nagios server in your datacenter has gone down as many or more times too. But my bigger point here is that you're essentially asking "well how do you monit…

> they're different services running on different architecture and distributed differently I think you underestimate the interdependency of services in AWS. Historically, if there were problems with S3 or EBS in us-east-1, you could expect the entire API to be flaky, and things like autoscaling to fail. These have been better distributed, but failures still cascade. > I think the original post was simply a demo of wh…

Launch two Lambda functions, heck, 8 Lambda functions, one in each AWS region that supports it. They all monitor one another, plus run your checks. Next, are you going to say all 8 regions will go down at once?

The whole setup will still cost $0/month.

> The solution to this is well known - two monitoring systems in physically separate locations that monitor each other as well as mission critical systems. Nagios, Icinga, and a dozen other well-tested solutions work remarkably well for these roles, yet people keep writing "new" solutions over and over and over.

Because not everyone needs heavy solutions to do something simple. Side projects, small sites, etc. And some people enjoy implementing old use cases using new technology. When Go was rising in popularity, half the posts on the front page were re-implementing fairly common features in Go.

Even if you're not going to implement this yourself, there can still be some value for other readers.

Re: Using AWS Lambda to call and text you when your servers are down

#49

Post by teddyh »

chinathrow wrote: post #46
teddyh wrote: post #41

Earlier quoted context omitted.

If our ISP was down, we would not have had successful accesses from other people at the same time. If some transit ISP was down somewhere between us and Pingdom, well, that’s the Internet for you, eh? Regardless, Pingdom would report us as down, even though we weren’t at fault.

We've seen the same with Pingdom. ISPs are (mostly) multi-homed and Pingdom might use just one (affected) route to the ISP in question and then fail badly. If Pingdom can't get to your site, it's highly likely your users can't either.

That was not the case with us; Pingdom would report short outages, like a few seconds here, a couple of minutes there, and only a handful of occurrences for the whole report duration (IIRC).

Re: Using AWS Lambda to call and text you when your servers are down

#50

Post by falcolas »

cddotdotslash wrote: post #48
falcolas wrote: post #44

Earlier quoted context omitted.

> they're different services running on different architecture and distributed differently I think you underestimate the interdependency of services in AWS. Historically, if there were problems with S3 or EBS in us-east-1, you could expect the entire API to be flaky, and things like autoscaling to fail. These have been better distributed, but failures still cascade. > I think the original post was simply a demo of wh…

Launch two Lambda functions, heck, 8 Lambda functions, one in each AWS region that supports it. They all monitor one another, plus run your checks. Next, are you going to say all 8 regions will go down at once? The whole setup will still cost $0/month. > The solution to this is well known - two monitoring systems in physically separate locations that monitor each other as well as mission critical systems. Nagios, Ici…

> are you going to say all 8 regions will go down at once

I hope not. But then it's not just Lambda triggered by cloudwatch alarms anymore. You'd probably have to set up something to ensure that Lambda, when called via cloudwatch alarms, is being triggered properly. Useful, but suddenly a lot more complicated.

> The whole setup will still cost $0/month.

Unlikely. A small amount, but certainly not 0. Especially when you start adding Lambda heartbeats.

> And some people enjoy implementing old use cases using new technology.

Which is fine; call it an experiment, call it exploration, I have no problem with that. It's frustrating to see such a stripped down article treating it like it's going to be the one, without reasonable discussions about how it could fail. There are a minimum of three failure points in this system alone, with no discussion on how to compensate for them.

Post reply on HN