Company was hosting cardiac patient monitoring on EC2 • Hacker News Board

Re: Company was hosting cardiac patient monitoring on EC2

#41

Post by gregburek » Sat, Apr 23, 2011, 8:28 PM UTC

teilo wrote: post #35

Elastic Load Balancer, anyone? I have no sympathy for this company. They deserve to get sued for this. Yes, for something life threatening like this, EC2 is a bad idea, but they didn't even bother to take advantage of the geographic redundancy, much less something so basic as having backup AMIs ready in another AZ in your current region. Of course, a whole lot of companies are learning this now.

Unfortunately, this last AWS problem affected multiple AZs in the us-east region. The OP very well may have had an alternate AZ failover plan, but like Quora, Sencha, Reddit, FourSquare and Heroku, they probably kept it region specific.

As for backing up to multiple regions, I can imagine them thinking that sending everything over the public internet as being a bad idea. However, not having a multi-region/multi-provider failover plan was a worse one.

Re: Company was hosting cardiac patient monitoring on EC2

#42

Post by kinofcain » Sat, Apr 23, 2011, 8:33 PM UTC

teilo wrote: post #35

Elastic Load Balancer, anyone? I have no sympathy for this company. They deserve to get sued for this. Yes, for something life threatening like this, EC2 is a bad idea, but they didn't even bother to take advantage of the geographic redundancy, much less something so basic as having backup AMIs ready in another AZ in your current region. Of course, a whole lot of companies are learning this now.

ELB doesn't work across regions, the failure at EC2 affected multiple Availability zones in the Virginia Region. ELB might have helped, but it's likely that some people had all their "availabiliy" zones knocked out.

In fact, it's looking like all the automatic-failovers from people hosting on those data centers compounded the problem when everyone's scripts tried to recover at the same time – what was referred to in a previous article as a "bank run".

But yeah, if you're doing heart monitoring, you need to have your servers replicated across a wide geographic area.

Re: Company was hosting cardiac patient monitoring on EC2

#43

Post by zaidf » Sat, Apr 23, 2011, 8:38 PM UTC

byoung2 wrote: post #23

We are a monitoring company and are monitoring hundreds of cardiac patients at home. If this is a serious post and there really is a service that monitors cardiac patients from home, first you would have to deal with the unreliability of in-home broadband connections. What if a router needs to be reset, or the DSL goes out? Even if you ignore that, you would expect that any server setup will go down at some time. It…

I don't think it's so much about trust as alternatives and best practices. Ideally you should not trust any system when running mission critical apps and always have multiple levels of backups and monitoring including humans on call.

Re: Company was hosting cardiac patient monitoring on EC2

#44

Post by tybris » Sat, Apr 23, 2011, 8:50 PM UTC

Those people should be prosecuted for utter incompetence.

Re: Company was hosting cardiac patient monitoring on EC2

#45

Post by forgotAgain » Sat, Apr 23, 2011, 8:52 PM UTC

This is too incredible to be true. But if it is true then I would expect the company in question to be out of business quite soon. A company that put's a life critical system on AWS shows a total disregard for their patients well being.

Re: Company was hosting cardiac patient monitoring on EC2

#46

Post by leon_ » Sat, Apr 23, 2011, 8:57 PM UTC

lawnchair_larry wrote: post #21

st0p wrote: post #3

That's not even stupid, that is moronic.

Why? Who else would you suggest for providing this type of service? Given the information we had a week ago, I can see how Amazon may have looked like the best solution for HA. Their site never goes down, after all.

So what? Even if God himself would promise 100% uptime I'd better have 2 or 3 fallbacks for such a service.

Re: Company was hosting cardiac patient monitoring on EC2

#47

Post by troels » Sat, Apr 23, 2011, 8:58 PM UTC

csomar wrote: post #25

Isn't this a good start-up idea? A start-up that backup your website and deploy it to another hosting in case the one you are using is down.

How about a host-abstraction layer, so that you can deal with different providers through a uniform interface?

Re: Company was hosting cardiac patient monitoring on EC2

#48

Post by muyuu » Sat, Apr 23, 2011, 8:59 PM UTC

ceejayoz wrote: post #30

chrisjsmith wrote: post #24

Earlier quoted context omitted.

Cardiac monitoring systems DO NOT have downtime believe it or not.

Bullshit. Leads come off, equipment malfunctions, patients fidget, and they definitely don't have redundant stuff on each patient.

I'm going to guess grandparent means that they don't include "downtime" in the contract. Which would be outrageous, BTW.

Re: Company was hosting cardiac patient monitoring on EC2

#49

Post by pstack » Sat, Apr 23, 2011, 9:02 PM UTC

kierank wrote: post #18

blantonl wrote: post #2

Actual quote in the Amazon support forums from this company: " This not just some social network website issue, but a serious threat to peoples lives! "

I have a feeling (and I really really hope) this is a joke because of that sentence.

I tend to suspect this entire thing, but as I've been working with enterprise teams across the board for over a decade from universities to companies that specialize in space exploration and governments and hospitals, I have discovered that there is sometimes a distinct lack of common sense and that often people with little or no skill end up in positions where they can make drastic decisions. I just have to believe that in this case, were it real, someone would have stopped the plan to be cloud-only before it reached that point.

If this is real, it's almost sickening on two levels. The first and most obvious is the risk the company exposed clients to in their deployment. Second, the lack of communication to paying customers from Amazon. I'm a fan of Amazon and probably spend more money there than at any other company on the planet and am disappointed whenever I see a business that puts up a giant wall between their paying customers and the support they need.

On the other hand, that comment leads us to a great idea for a startup: Social ECG Monitoring with a dash of Gamification. I even got all the flashy keywords in there!

Re: Company was hosting cardiac patient monitoring on EC2

#50

Post by zheng » Sat, Apr 23, 2011, 9:11 PM UTC

sliverstorm wrote: post #19

zaidf wrote: post #15

This is where you wish you had a data center address and a cabinet number where you could pop in, grab your servers and move the heck out to another DC. I wonder if this will get more companies to maintain a non-cloud, vanilla setup on some dedicated or collocated boxes. May be we should have an annual day to bring down our primary servers and see how the backups do. The idea shouldn't be to confirm if everything wor…

You know, that gets me thinking- wouldn't it be pretty damn slick if Amazon offered a way to plug your own hardware into the greater EC2 cluster remotely, so that if Big EC2 goes down, you still have copies of your machines running locally? Maybe you could even timeshare your machines out for some other instances for a discount on your service, similar to how you can send solar power back up the power grid for an ele…

Could someone who knows better comment on the feasibility of this? It sounds like a really good idea, but I'd imagine it's a bit more difficult than it sounds.