And I'm sure Lambda will never go down. Right? Right?? (It has. Completely and silently stopped processing against Kinesis queues for a few hours recently. Guess what AWS Step is built on?)
don't worry, the status was green the entire time...
Using AWS Lambda to call and text you when your servers are down
31–40 of 55 posts
Re: Using AWS Lambda to call and text you when your servers are down
#32From a engineering point of view this is really cool, but as an ex-sysadmin I feel that I need to reiterate and emphasize something that is alluded to in the second paragraph. Too many things can go wrong and you are all around better off outsourcing this to something like Pingdom. You don't have sufficient levels of reliability, you aren't dual homed across twilio and another phone system. Maybe the cause of your ou…
> Now your site and your monitoring is down. Much better to outsource to people who obsess over doing this right and making sure they are properly redundant. You make valid points about redundancy and levels of reliability but keep in mind that even Pingdom can go down: http://royal.pingdom.com/2016/10/24/ddos-attack-affects-ping...
With your own solution you will likely encounter the same problems that pingdom faced including this one. The benefit of a service like pingdom is that they already solved those problems for you or if they haven't you don't have to waste time solving them yourself. It's not very efficient if everyone solves the same problems over and over again.
Re: Using AWS Lambda to call and text you when your servers are down
#33I've used montastic.com for years. 2min setup to fire and forget.
Re: Using AWS Lambda to call and text you when your servers are down
#34I use https://aremysitesup.com/ and I've found it really helpful as it one of the few inexpensive services I've found that will CALL me if things are down. SMS is nice, but I use the do-not-disturb feature on my phone in the evenings, and at least on iOS, the only way to punch through that is with a call from a number on my favorites list. This meets that need very well and I've found the service to be quite spot on alerting me (both when I had one instance of things hitting the fan, but also during scheduled maintenance). I'd highly recommend.
Re: Using AWS Lambda to call and text you when your servers are down
#35From a engineering point of view this is really cool, but as an ex-sysadmin I feel that I need to reiterate and emphasize something that is alluded to in the second paragraph. Too many things can go wrong and you are all around better off outsourcing this to something like Pingdom. You don't have sufficient levels of reliability, you aren't dual homed across twilio and another phone system. Maybe the cause of your ou…
> Now your site and your monitoring is down. Much better to outsource to people who obsess over doing this right and making sure they are properly redundant. You make valid points about redundancy and levels of reliability but keep in mind that even Pingdom can go down: http://royal.pingdom.com/2016/10/24/ddos-attack-affects-ping...
Use 2 or more providers. Signing up takes a minute or two and there are free alternatives
Re: Using AWS Lambda to call and text you when your servers are down
#36And I'm sure Lambda will never go down. Right? Right?? (It has. Completely and silently stopped processing against Kinesis queues for a few hours recently. Guess what AWS Step is built on?)
Of course it can go down, and you can have CloudWatch alerts to alert you about that. But so can your Nagios server sending pings go down or the fancy SaaS you signed up for.
Did you just suggest using a third AWS service to let you know if the second AWS service monitoring your first AWS service goes down?
Re: Using AWS Lambda to call and text you when your servers are down
#37I don't know if everyone knows this but you can make texts using email. Most providers have SMTP gateways for SMS services. Verizon runs @vtext.com
I did this for non-critical ops, it's just a foward to my phone's email address, really simple.
The problem is that not every company is reliable or even has that email
Re: Using AWS Lambda to call and text you when your servers are down
#38I wrote Apex Ping ( https://apex.sh/ping/ ) for those who want more features and/or don't want to waste the time to save a few bucks :D.
Thanks for this. I would use it but you only do HTTP requests, right? It would be great if you could also do a true ICMP ping (like the name suggests!)
Re: Using AWS Lambda to call and text you when your servers are down
#39From a engineering point of view this is really cool, but as an ex-sysadmin I feel that I need to reiterate and emphasize something that is alluded to in the second paragraph. Too many things can go wrong and you are all around better off outsourcing this to something like Pingdom. You don't have sufficient levels of reliability, you aren't dual homed across twilio and another phone system. Maybe the cause of your ou…
We’ve had issues with Pingdom at work. We don’t use them ourselves, but we host web sites, and some customer of ours used Pingdom to monitor their web site hosted on our servers. The customer would complain to us about downtime reported by Pingdom, but we would read the logs and find everything OK, with multiple successful accesses from other people during the time which Pingdom reported our customer’s site as being down. A huge pain.
Re: Using AWS Lambda to call and text you when your servers are down
#40From a engineering point of view this is really cool, but as an ex-sysadmin I feel that I need to reiterate and emphasize something that is alluded to in the second paragraph. Too many things can go wrong and you are all around better off outsourcing this to something like Pingdom. You don't have sufficient levels of reliability, you aren't dual homed across twilio and another phone system. Maybe the cause of your ou…
We’ve had issues with Pingdom at work. We don’t use them ourselves, but we host web sites, and some customer of ours used Pingdom to monitor their web site hosted on our servers. The customer would complain to us about downtime reported by Pingdom, but we would read the logs and find everything OK, with multiple successful accesses from other people during the time which Pingdom reported our customer’s site as being…
Doesn't services like Pingdom support multiple ping locations? If all of those fail, there's a very high chance there's an actual problem, if not with your server then with your (ISPs) connectivity.