Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

news.softpedia.com

Post reply on HN
Print view View original

141–150 of 211 posts

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#141

Post by sathackr »

scruple wrote: post #139
sathackr wrote: post #135

Earlier quoted context omitted.

I'm with you...I can see exactly how this can happen. Unfortunately the only thing that can solve the apathetic board and executive management problem(who only see dollar signs) is the actuality, or realistic possibility, of significant financial loss, or loss of their personal freedom(prison) due to the negligence of the system. And a $10 Mil fine for a fault in something that you make $100 Mil off of is not signifi…

I've thought about this a lot. I've had private conversations with the CEO which lead me to believe that their apathy is a, if not the, primary driver in this situation, at least within the company. Ultimately, they are the single individual who can force these changes in the departments. As things stand today, as far as I can tell, the CEO and the rest of the executive team got theirs and that's that. Anything extra…

Along the same lines as your 'where the fuck is the FDA' comment -- I've worked in Financial and Healthcare systems on and off for about the last 10 years.

I have seen SSAE16 audited companies that haven't patched anything in years. FDIC examined institutions with ATM machines still running OS/2 Warp(actually probably more secure than the ones running XP, with no updates installed. Ever.)

I once found the management interface of a SAN with a public IP address directly on the device, no firewall rules of any sort, and the device still had the default username/password. It hadn't been patched or rebooted in over 2 years.

More shocking is that a review of the logs didn't show any successful unauthorized logins. Of course, they could have cleaned up after themselves, but further investigation was outside the scope of my engagement(They didn't want to know. They were happy to present that, despite the oversight, there was no indication that PHI had been accessed by unauthorized people. Their conclusion, not mine.)

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#142

Post by emcrazyone »

scruple wrote: post #132
sathackr wrote: post #89

Earlier quoted context omitted.

Exactly this. As I was reading the article I hoped to find this exact point in the HN comments. The fault lies in the bad software. It could have been the indexing service, online defrag, automatic updates, or any of the other various background processes windows runs. If it is critical software, it should be designed in a way to not fail when something non-critical malfunctions, and even the critical pieces should b…

I work for a medical devices company and I just want to say: We, specifically a few of us on the engineering staff, bring this sort of shit up constantly. I go hoarse having the same conversations over and over and over again about robustness in the face of failure, resiliency, redundancy, etc... The truth is that we're beholden to a board and an executive management team that, quite simply, doesn't give a fuck about…

I'm just curious. I work in the automotive sector and develop hardware and software using components that are advertised as functionally safe. I use harden RTOS from vendors who claim their RTOSes are in medical devices as well as military systems.

One such system is Disti (http://www.disti.com/)

In the automotive field, our software is MISRA compliant, static analysis is done (Klockworks - http://www.klocwork.com/) and we follow a very strict set of guidelines outlined in ISO25119 and ISO26262 for the construction and agricultural markets. Think self driving tractors and combines. For example: A tractor traveling down a field with a combine following it a few rows over separating and chopping things into a catcher all done with one person driving.

This shit can't happen where I work. Every component on our circuit boards has a MTTFd of 40 years. Hardware watchdogs can kill the system if software goes awry.

Software is written to readiness level called SRL-1, SRL-2, etc... Unit tests, peer reviews, etc... Functional safety in medical devices is covered under 510(K) (http://www.fda.gov/MedicalDevices/ProductsandMedicalProcedur...)

I find it amazingly short sited that antivirus software is even allowed on a medical device to begin with. I can't even imagine how this system even passed the easiest audit for software readiness.

How is it you "go hoarse having the same conversations?" Do you not have to meet FDA compliance criteria? Are you in the US?

I didn't read the whole article so I'm assuming this happened in the US. For me, we sell autonomous vehicles in the European markets where functional safety seems to be a bit more aggressive there right now for vehicles. Not sure about medical devices.

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#143

Post by combatentropy »

> they included proper instructions in their documentation, advising companies to whitelist Merge Hemo's folders in order to prevent crashes from happening, so it seems that the whole incident was nothing more than an oversight on the medical unit's side.

And the hospital included full instructions to the software company on how to properly perform a heart transplant, so they were baffled why the programmer just let his teammate die of heart failure.

Come on, this kind of stuff should be a zero-configuration hardware-based black box, with its own buttons, screen, etc. --- not something that needs to be (or even can be) connected to something outside the vendor's total control.

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#144

Post by qb45 »

steven2012 wrote: post #18

Antivirus scans are one of those things added on IT checklists to cover their ass whenever something wrong happens. But it rarely is useful. It only causes problems. We've seen so many issues related to virus scans throughout the years it's crazy. What's better is to lock down the servers with only minimal access. I haven't used virus scan on my main desktop for over 10 years because I don't click on weird emails and…

> I haven't used virus scan on my main desktop for over 10 years because I don't click on weird emails and I don't go to sketchy websites ever.

Haha, I used to be like that as a teen in Windows 9x era until one day I ran tcpdump on the router ;)

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#145

Post by dchichkov »

state wrote: post #123
dchichkov wrote: post #120

Let me surprise you, with the code quality that sometimes is running in what is actually 'life-critical' software. Back in the nineties, I wrote a nice piece of some 300kb of C code, for DOS/x86. It was a complete software package, controlling medical equipment that was testing speed of blood coagulation. These tests are crucial in the patient post-operation recovery. This piece of C code had some hardware control co…

I understand why you probably don't want to put it up, but boy would it be fun to look at the code you're describing.

I probably will put it up. It's a nice inspirational story for teens out here. Doubt there'd be any repercussions, no one cares about some random code on GitHub. And the equipment is hopefully taken out of service years ago, it was more than 20 years back. I wish I knew how long it had been used, but there've been only about 10-20 units sold, I think.

I vaguely remember adding extra features for a year or so (like adding support for HP laserjet printer). But one of the founders of the company (on the business side) had some health problems, and I guess that had played role in very small number of units sold. The only feedback that I've had, is pretty much that my father took me to a lab once, that had a unit deployed, for a support call. And I've seen some real printouts with patient names, from the unit. The lab assistant seemed to be happy with the device. I remember them showing me some blood plasma and teaching me to count cells, during lab tour ;)

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#146

Post by noonespecial »

scruple wrote: post #139
sathackr wrote: post #135

Earlier quoted context omitted.

I'm with you...I can see exactly how this can happen. Unfortunately the only thing that can solve the apathetic board and executive management problem(who only see dollar signs) is the actuality, or realistic possibility, of significant financial loss, or loss of their personal freedom(prison) due to the negligence of the system. And a $10 Mil fine for a fault in something that you make $100 Mil off of is not signifi…

I've thought about this a lot. I've had private conversations with the CEO which lead me to believe that their apathy is a, if not the, primary driver in this situation, at least within the company. Ultimately, they are the single individual who can force these changes in the departments. As things stand today, as far as I can tell, the CEO and the rest of the executive team got theirs and that's that. Anything extra…

>Where the fuck is the FDA? We send them paperwork and they are happy.

When regulation becomes more about permission than proficiency, you'll get corruption instead of competence.

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#148

Post by PhasmaFelis »

sillysaurus3 wrote: post #95
wpietri wrote: post #94

Earlier quoted context omitted.

I totally disagree. Sure, usually the most graceful thing to do is exit and hope a human fixes it. But that's usual because the usual condition is that sudden failure is NBD and a human is right there to screw with it. That's becoming less common, though. When software was mostly something running on a PC doing some boring office task, reliability didn't matter. But as software is running our airplanes, our cars, our…

It wasn't a rhetorical question. What could this program possibly do to recover? If the CPU fails, no one would say the program was unreliable. In this case, the disk failed, because the antivirus unplugged it. Was the program unreliable?

Normal software does not crash the entire OS because the antivirus was looking at a file it wanted. We expect better of text editors, for God's sake. There is absolutely no excuse for a life-critical system to fail to meet that rock-bottom standard.

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#149

Post by Avernar »

Bluestrike2 wrote: post #136
Avernar wrote: post #102

Earlier quoted context omitted.

Yes it does! Showing "no telemetry available" is exactly what it should do. Crashing = unreliable. Reporting an error condition = reliable. Immediately? Took them 5 minutes to reboot the computer. The scan of the folder would take seconds let alone minutes. Pinpointing the problem is secondary. Not killing the patient is primary. > If the program's sole purpose is to transform a massive amount of data in real time, i…

I agree with you, but the flat line might not be the best example because that has a very specific meaning (asystole) that doctors will take certain actions based on without necessarily trying to verify it manually when time is already critical. You should never be able to confuse an error message for anything else.

> You should never be able to confuse an error message for anything else.

Exactly. Which is why "Can't read file sensors.dat" is way better than just crashing. Crashing is one of the worst error messages you can get because you don't know what happened.

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#150

Post by scruple »

emcrazyone wrote: post #142
scruple wrote: post #132

Earlier quoted context omitted.

I work for a medical devices company and I just want to say: We, specifically a few of us on the engineering staff, bring this sort of shit up constantly. I go hoarse having the same conversations over and over and over again about robustness in the face of failure, resiliency, redundancy, etc... The truth is that we're beholden to a board and an executive management team that, quite simply, doesn't give a fuck about…

I'm just curious. I work in the automotive sector and develop hardware and software using components that are advertised as functionally safe. I use harden RTOS from vendors who claim their RTOSes are in medical devices as well as military systems. One such system is Disti ( http://www.disti.com/ ) In the automotive field, our software is MISRA compliant, static analysis is done (Klockworks - http://www.klocwork.com/…

> I find it amazingly short sited that antivirus software is even allowed on a medical device to begin with.

Well... Then you should consider yourself blessed to have never had to deal with the bureaucracy of a hospital IT department and administrative staff.

Who owns the medical device? Who paid for it? If it's a glorified Windows machine and it's attaching itself to a hospitals WiFi network... Who has to use this machine? Physicians, surgeons, anesthesiologists, radiologists, other specialists, nurses, staff? All of them need to be trained on it's usage, no doubt. They don't get that training in schooling. Who provides it? This and a million other things stack up. So, well, I mean it can start to make sense how these things end up with random AV software installed on them, right?

> How is it you "go hoarse having the same conversations?" Do you not have to meet FDA compliance criteria? Are you in the US?

Yes, we are. Yes, we do "have to meet FDA compliance." I can't define "have to meet" and I work here. Of course, I'm just an engineer. We have legal, executive, and other staff for those matters. I'm sorry, I'm not trying to be an asshole... I'm just trying to be honest about where I find myself in this situation.

Post reply on HN