Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

news.softpedia.com

Post reply on HN
Print view View original

151–160 of 211 posts

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#151

Post by stevetrewick »

From the linked report :

Based upon the available information, the cause for the reported event was due to the customer not following instructions concerning the installation of anti-virus software; therefore, there is no indication that the reported event was related to product malfunction or defect

I beg to differ. I'd consider a momentary loss of file I/O due to lock contention causing a machine to require a reboot a shocking defect in - say - a word processor (which, notably, do not have this problem). That this risk is apparently known and the vendor's sole mitigation is to document a 'Don't do that then' is absolutely 100% an indication of a product defect, even in the absence of an actual occurrence.

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#152

Post by billforsternz »

"A critical medical equipment crashed during a heart procedure due to a timely scan triggered by the antivirus software installed on the PC to which the said device was sending data for logging and monitoring."

That should be untimely. The opposite of timely.

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#153

Post by stevetrewick »

sillysaurus3 wrote: post #83
Avernar wrote: post #77

"Merge says the antivirus froze access to crucial data acquired during the heart catheterization. Unable to access real-time data, the app crashed spectacularly. The company claims that they included proper instructions in their documentation, advising companies to whitelist Merge Hemo's folders in order to prevent crashes from happening, so it seems that the whole incident was nothing more than an oversight on the m…

Software shouldn't necessarily try to account for errors in that manner. Usually, the most graceful thing to do is to exit cleanly. For example, if there is a massive amount of data, it has to be stored on disk. It's too large to keep in memory. And if the point of the program is to transform that data in real time, then it has to have access to the disk. The antivirus basically unplugged the disk. What can it do to…

Software should absolutely try to account for these kinds of errors and routinely does. When was the last time you saw a word processor or spreadsheet bork a machine so hard it needed to be restarted just because an AV scan kicked in? ReadFile and ReadFileEx both hand you a specific error if some part of the file you are trying to read is locked by another process because it's hardly rare. 'Halt and catch fire' is not generally considered a proportionate response. I've no idea where you're getting this 'unplugged the disk' thing from, AV software does not work this way.

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#154

Post by wpietri »

Avernar wrote: post #107
sillysaurus3 wrote: post #100

Earlier quoted context omitted.

In that scenario, your surgeon would see the program suddenly freeze. The program likely looks like this: data acquisition -> transformation -> display transformation on monitor. If the transformation step fails, the monitor will end up displaying (a) nothing, (b) random data, or (c) the most recent image. None of these help the surgeon continue surgery. It's the same as a crash. If your environment fails, there's no…

> In that scenario, your surgeon would see the program suddenly freeze. Only if the programmer or his management were incompetent. The display routine should be running on a separate thread than the processing code. No whole program freeze should occur. As for displaying random data, why would the programmer want to do this? Either display nothing or the last readings WITH a message that it's not real time. It's not…

Definitely. Each component should do its best to keep on keeping on. The display program should keep displaying something, even it's just the most recent data with a big "connection lost" warning. The device should ring-buffer the data and upon reconnection the screen should show as much as possible. The OS should have a strong opinion that the surgery app is very important, and that should the app fail, it should be restarted instantly.

Moreover, this is the kind of thing that should come up in robustness testing. Things should get bumped and wiggled. They should get unplugged and turned off. If the software is really going to run on random Windows boxes, then it should be tested on random Windows boxes. (At which point somebody will hopefully say, "Wow, this sucks, let's make it an appliance.")

No matter what happens, it shouldn't result in a "mysterious crash right in the middle of a heart procedure when the screen went black and doctors had to reboot their computer".

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#155

Post by technion »

iask wrote: post #50

There are a couple of things here from my POV, first - I would replace the head of their IT and any senior IT staff - who seem to look for the quickest-then-cheap solutions. Dumb ducks who don't spend the extra time understanding the importance of the infrastructure and the software they install. And also replace the service vendor, if they have one. I've seen this happen time and again, where companies have some 3rd…

    I would replace the head of their IT and any senior IT staff
It's a very good bet the senior IT team were following orders from somewhere else in the chain here.

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#156

Post by chris_wot »

scruple wrote: post #150
emcrazyone wrote: post #142

Earlier quoted context omitted.

I'm just curious. I work in the automotive sector and develop hardware and software using components that are advertised as functionally safe. I use harden RTOS from vendors who claim their RTOSes are in medical devices as well as military systems. One such system is Disti ( http://www.disti.com/ ) In the automotive field, our software is MISRA compliant, static analysis is done (Klockworks - http://www.klocwork.com/…

> I find it amazingly short sited that antivirus software is even allowed on a medical device to begin with. Well... Then you should consider yourself blessed to have never had to deal with the bureaucracy of a hospital IT department and administrative staff. Who owns the medical device? Who paid for it? If it's a glorified Windows machine and it's attaching itself to a hospitals WiFi network... Who has to use this m…

Sounds like an awesome job with good engineers but neglectful and irresponsible management.

If you are only making these warnings verbally, you might want to consider emailing your immediate manager with a list of concerns. Make it as neutral as possible and ask for guidance on how they want to address the issues. But if it's on the mail server, it will be good for discovery if the worst happens, and frankly given lives are at stake you probably need to show, in writing, that you were attempting to have the issues addressed.

Who knows? That might actually get traction. Might even save someone's life!

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#157

Post by chris_wot »

scruple wrote: post #139
sathackr wrote: post #135

Earlier quoted context omitted.

I'm with you...I can see exactly how this can happen. Unfortunately the only thing that can solve the apathetic board and executive management problem(who only see dollar signs) is the actuality, or realistic possibility, of significant financial loss, or loss of their personal freedom(prison) due to the negligence of the system. And a $10 Mil fine for a fault in something that you make $100 Mil off of is not signifi…

I've thought about this a lot. I've had private conversations with the CEO which lead me to believe that their apathy is a, if not the, primary driver in this situation, at least within the company. Ultimately, they are the single individual who can force these changes in the departments. As things stand today, as far as I can tell, the CEO and the rest of the executive team got theirs and that's that. Anything extra…

I can't help responding again. If you have tangible evidence of neglect or regulatory non-compliance, or even risks that are known about but not being dealt with by management - have you considered compiling this material and and reporting it to the FDA?

But as I've said before - I really hope you have written down your concerns to someone in management. If it gets to the point where negligence takes out the company, there's going to be an attempt to make someone a scapegoat. Depending on your role in the company you don't want to be held personally liable for the incompetence and ruthlessness of management...

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#158

Post by chris_wot »

Bluestrike2 wrote: post #134
sillysaurus3 wrote: post #98

Earlier quoted context omitted.

During a surgery, the program doesn't have the luxury of showing a screen that says "No telemetry available." Such a program would be considered equally unreliable. Worse, it would lead to confusion: "Why is the telemetry unavailable? What does 'Error Code 2931' mean?" A spectacular crash immediately led to pinpointing the problem: The antivirus. If the program's sole purpose is to transform a massive amount of data…

But the spectacular crash didn't pinpoint the problem. That came afterwards, when the manufacturer was able to look into the crash. In a situation like this, confusion is all but inevitable. As a developer, the goal should be to minimize that confusion to the greatest extent possible. A blank screen and crash introduces another step to the process as people wonder "what's going on?" instead of "shit, it threw an erro…

I wonder what surgeons think about software engineers? Except for open heart surgery, they don't normally do their fixes by stopping and starting the thing they are repairing...

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#159

Post by Joof »

Avernar wrote: post #77

"Merge says the antivirus froze access to crucial data acquired during the heart catheterization. Unable to access real-time data, the app crashed spectacularly. The company claims that they included proper instructions in their documentation, advising companies to whitelist Merge Hemo's folders in order to prevent crashes from happening, so it seems that the whole incident was nothing more than an oversight on the m…

Agreed. This is what happens when software fails catastrophically. People die and we lose confidence in software that could save lives if it worked properly.

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#160

Post by stevetrewick »

mappu wrote: post #113
Avernar wrote: post #77

"Merge says the antivirus froze access to crucial data acquired during the heart catheterization. Unable to access real-time data, the app crashed spectacularly. The company claims that they included proper instructions in their documentation, advising companies to whitelist Merge Hemo's folders in order to prevent crashes from happening, so it seems that the whole incident was nothing more than an oversight on the m…

Better to crash (and restart quickly into a known state) than to enter a rare, untested code path.

File I/O error handlers should not be a rare untested code path.
Post reply on HN