Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan • Hacker News Board

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#101

Post by symtos » Sat, May 07, 2016, 10:50 PM UTC

Blackthorn wrote: post #93

symtos wrote: post #90

Earlier quoted context omitted.

using software with known problems in order to avoid potential problems from an upgrade does not seem like a non-bad decision

Is the medical device working right now? Yes. Could, upon upgrading, the device stop working, possibly in a subtle way that might kill somebody? Yes. The approval process for medical devices is rightfully difficult. Software upgrades, even if they seem trivial, should not be a backdoor process of bypassing testing and approval.

...and could software deployed to the device by some random who just exploited some well-known security flaw that never got patched, kill people?

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#102

Post by Avernar » Sat, May 07, 2016, 10:52 PM UTC

sillysaurus3 wrote: post #98

Avernar wrote: post #96

Earlier quoted context omitted.

> When the disk is plugged back in, it should be able to restart without any problems. But I think that's a different kind of resiliency than what you're referring to. Yes and no. I was referring to restarting internally when the error condition went away but restarting the app and waiting for telemetry to return can be a valid solution. Think of your torrent software. If you crank your firewall to block it while it'…

During a surgery, the program doesn't have the luxury of showing a screen that says "No telemetry available." Such a program would be considered equally unreliable. Worse, it would lead to confusion: "Why is the telemetry unavailable? What does 'Error Code 2931' mean?" A spectacular crash immediately led to pinpointing the problem: The antivirus. If the program's sole purpose is to transform a massive amount of data…

Yes it does! Showing "no telemetry available" is exactly what it should do. Crashing = unreliable. Reporting an error condition = reliable.

Immediately? Took them 5 minutes to reboot the computer. The scan of the folder would take seconds let alone minutes. Pinpointing the problem is secondary. Not killing the patient is primary.

> If the program's sole purpose is to transform a massive amount of data in real time, it must have disk access by definition. It can't not have disk access.

And that is the mind set the programmers of the software had. You have to take care of error conditions. The processing can't have no disk access but no disk access can occur temporarily or permanently. What can you do? Pause the processing part of your program. Or make the processing part treat "no data" as valid input and display something else.

Imagine taking that viewpoint with an ECG machine: This machine displays a heart rate waveform. So it must have a heart rate input. If there is no heart rate we'll just crash requiring a 5 minute reboot.

Hell no! Draw a straight line and set off a buzzer!

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#103

Post by saganus » Sat, May 07, 2016, 10:54 PM UTC

Wtf?

"The antivirus was configured to scan for viruses every hour, and the scan started right in the middle of the procedure."

Who configures an antiviurs for an hourly scan on a doctor's computer?

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#104

Post by dlp211 » Sat, May 07, 2016, 10:55 PM UTC

zxcvcxz wrote: post #79

dlp211 wrote: post #71

Earlier quoted context omitted.

Get out of here with that nonsense. You may not be a fan of paid software, but the Windows Kernel is just as good as any FOSS kernel today in regards to stability. The Server and embedded SKUs also come with a ton of the extraneous stuff removed that one would ever be worried about. The issue here is that someone decided that a machine that should only ever be connected to an air gapped network needed anti-virus soft…

> the Windows Kernel is just as good as any FOSS kernel today in regards to stability. No it's really not. I've used Windows and Linux a lot and in 10 years of using Linux I've only twice had a kernel panic while using non-experimental software. On Windows I've had countless BSODs.

I'm guessing that your use cases for Windows and FOSS kernels are different, but I'll counter your anecdote with some of my own.

Almost every single BSOD today is a hardware/driver problem. After Server 2003, MSFT got serious about kernel stability. I haven't had a BSOD from a kernel problem since Vista.

Secondly, I also run multiple Server 2012 R2 servers with server hardware, and have never had a BSOD with any of those machines.

Finally, Microsoft runs Azure on Win Server, if it wasn't stable, it's cloud platform would be in serious trouble.

Disclaimer: I still work for Microsoft

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#105

Post by dantiberian » Sat, May 07, 2016, 11:04 PM UTC

DyslexicAtheist wrote: post #92

Amezarak wrote: post #47

Earlier quoted context omitted.

> When you evaluate the choice of Windows you have to acknowledge that it brings with it the vulnerability of viruses and so the necessity of anti-virus software. No matter what OS you choose it is vulnerable to viruses. You and I will agree that the odds are your Windows system is much more at risk by at least an order of magnitude. But the IA people who demanded that this system run antivirus are just as likely to…

> You and I will agree that the odds are your Windows system is much more at risk by at least an order of magnitude it is a common misconception that Windows is still worse a platform than Linux when it comes to security. Not trolling ... I'm using Linux since '96 and built my life and career on it. Opinion of some people in infosec circles (@thegrugq @csoghoian ...) is that Windows no longer lags behind: https://gru…

There's two separate things that are conflated here:

* Is Windows security equal to or greater than Linux and OS X (Probably yes)

* Are the overwhelming majority of viruses written to target Windows systems (yes)

I really enjoyed this article on security economics which goes into this in more depth: http://tidbits.com/article/15939

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#106

Post by jcrawfordor » Sat, May 07, 2016, 11:04 PM UTC

raverbashing wrote: post #15

11thEarlOfMar wrote: post #2

/rant/ I can't tell you how many times we've chased down field problems that ultimately were the result of antivirus scans. It's been so bad, that one of the first questions we now ask when we get a tool-down report is "is there antivirus running and what is the configuration?" Bringing Windows into the architecture of any type of capital equipment control system is a bane. A scourge. I mean to say, it really is a mi…

This This machines costs hundreds of thousands of dollars. There should be no excuse for using Windows. None. I would not be surprised if the "antivirus" thing was some PHB requirement

I think a lot of people are misreading this article. It appears that the device is standalone and runs some embedded OS, but as a feature logs data to the doctor's existing computer. So it's not so much that they built the tool on windows as it is that they built the tool on the operating system they expected doctor's to be using.

I still think this was a poor decision, but it's a different kind of poor decision

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#107

Post by Avernar » Sat, May 07, 2016, 11:08 PM UTC

sillysaurus3 wrote: post #100

Avernar wrote: post #99

Earlier quoted context omitted.

The disk did not fail. An I/O operation failed. The first is a permanent condition, the second can be permanent or transient. Big difference. In Linux a signal can cause an I/O to fail. In Windows it's antivirus and other background tasks can cause I/O to fail. What can it do to recover? Retry the I/O operation! It should keep trying until the operator tells it to stop.

In that scenario, your surgeon would see the program suddenly freeze. The program likely looks like this: data acquisition -> transformation -> display transformation on monitor. If the transformation step fails, the monitor will end up displaying (a) nothing, (b) random data, or (c) the most recent image. None of these help the surgeon continue surgery. It's the same as a crash. If your environment fails, there's no…

> In that scenario, your surgeon would see the program suddenly freeze.

Only if the programmer or his management were incompetent. The display routine should be running on a separate thread than the processing code. No whole program freeze should occur.

As for displaying random data, why would the programmer want to do this? Either display nothing or the last readings WITH a message that it's not real time.

It's not the same as a crash! A crash requires 5 minutes minimum guaranteed. Restarting instantly after telemetry returns can happen under a second in the best case which can be the difference between a live and dead patient.

> If your environment fails, there's nothing you can do to recover. Planes aren't designed to survive the loss of a wing. Why is this case any different?

There are different kinds of failure. Permanent and transient. Following the permanent procedure for a transient case can be fatal.

Take your airplane example. Loss of a wing is permanent. That would be like the CPU failing or an external cable being cut.

But your engines shutting down can be permanent or transient. Just like disk I/O failing. You'd use the transient procedure in this case. Keep trying to restart the engines. If they restart, great! You've just saved the plane.

Same with the disk I/O. The programmer should keep trying to restart the I/O. If it comes back, great! You've just saved the patient.

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#108

Post by ChuckMcM » Sat, May 07, 2016, 11:10 PM UTC

Amezarak wrote: post #47

ChuckMcM wrote: post #42

Earlier quoted context omitted.

I think you have it backwards (no disrespect intended). When you evaluate the choice of Windows you have to acknowledge that it brings with it the vulnerability of viruses and so the necessity of anti-virus software. Either you own that decision, and as part of your support your tool provides the necessary antivirus and you also insure through testing configuration management that its configured appropriately, or you…

> When you evaluate the choice of Windows you have to acknowledge that it brings with it the vulnerability of viruses and so the necessity of anti-virus software. No matter what OS you choose it is vulnerable to viruses. You and I will agree that the odds are your Windows system is much more at risk by at least an order of magnitude. But the IA people who demanded that this system run antivirus are just as likely to…

   > No matter what OS you choose it is vulnerable to
   > viruses.

It is a bit more nuanced than that. While it is true that absolute security is generally deemed impossible, if you use a widely deployed operating system in your device there are both a number of actors trying to compromise it for different reasons, and a number of examples that can be acquired for testing different exploits.

By writing just enough "OS" to achieve your goals in an embedded system, and then designing a clean access API through which you cannot affect the underlying code (no "here download this new firmware" call) you can avoid that particular threat vector.

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#109

Post by jcrawfordor » Sat, May 07, 2016, 11:22 PM UTC

tjohns wrote: post #86

zxcvcxz wrote: post #79

Earlier quoted context omitted.

> the Windows Kernel is just as good as any FOSS kernel today in regards to stability. No it's really not. I've used Windows and Linux a lot and in 10 years of using Linux I've only twice had a kernel panic while using non-experimental software. On Windows I've had countless BSODs.

To be fair (and I'm really not a Windows fan), many of the BSODs on modern versions Windows can be attributed to shoddy third-party drivers. And as a point of comparison, I've had tons of kernel panics on MacOS over the years, for various reasons. Sometimes defective hardware, sometimes odd software interactions, and sometimes for reasons I can't explain. Windows has a lot of problems. The kernel itself is not really…

I had a friend whose Macbook Pro started kernel panicking every time he exited emacs, completely reliably. A rather bizarre problem that persisted for months on his device, through some updates, but we couldn't replicate on another machine. Some kind of weird hardware issue? Corrupted software somewhere not detected by integrity checks? No idea. But these days, kernel panics and BSODs tend to originate from odd edge-case problems like that.

(And yes, the joke "but why would you ever exit emacs?" was said many times)

Re: Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan

#110

Post by jcrawfordor » Sat, May 07, 2016, 11:26 PM UTC

steven2012 wrote: post #18

Antivirus scans are one of those things added on IT checklists to cover their ass whenever something wrong happens. But it rarely is useful. It only causes problems. We've seen so many issues related to virus scans throughout the years it's crazy. What's better is to lock down the servers with only minimal access. I haven't used virus scan on my main desktop for over 10 years because I don't click on weird emails and…

Yeah, you don't click on weird emails and don't go to sketchy websites. Try managing IT security for an enterprise of 10,000 employees. A/V will save your ass hundreds of times every single day.

Computer professionals rarely understand the use case for A/V precisely because they are not the use case. In most all applications, A/V serves first as a safeguard against stupid user behavior, and only second as a safeguard against more advanced penetration (and in the latter case, one with only rare success). I'd bet that the #1 way enterprises are getting breached is still malicious email attachments, that's certainly true in my experience.