“I Want to Know What Code Is Running Inside My Body” • Hacker News Board

Re: “I Want to Know What Code Is Running Inside My Body”

#91

Post by Illniyar » Tue, Aug 09, 2016, 5:43 AM UTC

radicality wrote: post #82

How about when I'm flying an airplane; I'm also putting my life in the hands of people that wrote the code that controls it and I have to trust that the plane won't shut itself down mid-flight because of faulty code. Should a similar argument be made here?

I think the difference between these two are 2-fold - first is ownership and second is personal.

A pacemaker is something you bought and owned, when flying in a plane you are buying a service, this is similar to earlier discussions about being able to change your car's software under DMCA etc...

A pacemaker is also personal, in that it's something that only you have and for your specific pacemaker the only affected party is you. A specific flight has hundreds of affected people which makes the burden of responsibility (for lack of a better word) shared between many people.

Re: “I Want to Know What Code Is Running Inside My Body”

#92

Post by jordigh » Tue, Aug 09, 2016, 5:57 AM UTC

seehafer wrote: post #90

jordigh wrote: post #78

Earlier quoted context omitted.

I know. Nope. The FDA probably doesn't even know what source code is. They have vague regulations on how medical devices should be tested, which by tradition has been interpreted in a particular way to mean certain kinds of documents have to be prepared. There are auditors that check that those documents are written. Nobody checks that what the documents say about the software is in fact true because neither those wr…

As someone who has been on the receiving end of several FDA audits, I would really like to know on what basis you're saying all of this, because everything in my 10 years of experience of doing this is contrary to what you've said.

Maybe we worked in different fields? I only did 3 years, only received one FDA audit, but many other audits from pharmaceutical companies. I was in the more diagnostic side, not therapeutic, although we did have some safety checks where we had to quickly raise an alarm if our analysis showed a potential medical emergency.

Did the FDA actually check your software or just your documentation? Did the auditors very carefully grill you or just languidly ticked off boxes on a form? Did you find the process of writing documentation instrumental in ensuring that your software was carefully tested? Did you ensure that your tests were reproducible and comprehensive?

None of these things were done very carefully in our case. My superiors were very insistent on the documentation and were quite proud of the quality of our software but they mostly never had any interaction with it and had no real idea of what we did for software validation. They were more concerned with making sure signatures and dates were correct and that our documentation didn't make us look bad.

I think the audits must look more impressive when you're in charge, but as the one actually writing the software I was thinking... that's it? The FDA doesn't really give a damn about what I really worked on, do they?

Re: “I Want to Know What Code Is Running Inside My Body”

#93

Post by ddeck » Tue, Aug 09, 2016, 5:58 AM UTC

radicality wrote: post #82

How about when I'm flying an airplane; I'm also putting my life in the hands of people that wrote the code that controls it and I have to trust that the plane won't shut itself down mid-flight because of faulty code. Should a similar argument be made here?

>Should a similar argument be made here?

Yes, and it has been. See "DO-178C, Software Considerations in Airborne Systems and Equipment Certification" [1] for example.

Also:

https://en.wikipedia.org/wiki/Avionics_software

[1] https://en.wikipedia.org/wiki/DO-178C

Re: “I Want to Know What Code Is Running Inside My Body”

#94

Post by jsmthrowaway » Tue, Aug 09, 2016, 6:10 AM UTC

gravypod wrote: post #33

JoshTriplett wrote: post #23

Earlier quoted context omitted.

It's also self-modifying code that bootstraps its own compiler.

Yea it's got some cool genetic algorithms.

One could frame the entire computing industry as a distributed genetic algorithm, executed by the real computers in order to understand themselves and the environment around them. One could further posit that we don't really have a good handle on the right fitness function yet.

(I realize this sounds like a low-effort joke, but think about it for a second.)

Re: “I Want to Know What Code Is Running Inside My Body”

#95

Post by visarga » Tue, Aug 09, 2016, 6:16 AM UTC

radicality wrote: post #82

How about when I'm flying an airplane; I'm also putting my life in the hands of people that wrote the code that controls it and I have to trust that the plane won't shut itself down mid-flight because of faulty code. Should a similar argument be made here?

> when I'm flying an airplane; I'm also putting my life in the hands of people that wrote the code

Yes, but you don't have to. You're not gonna die if you don't put your life in their hands. But with pacemakers and such, you have to get one or you die. Then, you depend on the manufacturer.

Re: “I Want to Know What Code Is Running Inside My Body”

#96

Post by visarga » Tue, Aug 09, 2016, 6:22 AM UTC

yostrovs wrote: post #77

jordigh wrote: post #61

Earlier quoted context omitted.

Y'know, back in the early 2000's and the days of Slashdot, it was quite common to find people who advocated for free software everywhere. Now we find people who like yourself have to specify that the radical position that all software should be free is something worthy of serious consideration. That they're not joking or trying to be deliberately provocative. What happened to us? Why did we go from boasting about ins…

Because Linux was supposed to become a great thing, but instead it remained a paradise for geeks to do what they think is best. Software built to make money, on the other hand, was built to improve things like ease of use, aesthetics, and buyer's happiness, because that's what buyers were looking for. The open-source people never really cared about the dumb people and lay folks, the ignoramuses that didn't care to le…

I was under the impression that Mac OS, iOS, Android and recently Windows all run "Linux" under the hood nowadays. Not to mention the presence of Linux in the cloud, which is essential for the functioning of most websites, apps and mobile devices. Am I wrong?

Basically, it's easier to tell where there is no Linux than the opposite. That surely doesn't sound like such a failure. It's as if all the other OSes and systems are front-ends for Linux subsystems.

Re: “I Want to Know What Code Is Running Inside My Body”

#97

Post by TickleSteve » Tue, Aug 09, 2016, 6:23 AM UTC

cooper12 wrote: post #75

yolesaber wrote: post #64

I saw a talk about medical device security (or lack thereof) at the Eleventh Hope a few weekends ago. Very scary. They started off with a story about patients in a hospital who became horribly addicted to morphine because they were able to hack the machine from resources found online ( http://www.massdevice.com/hospital-patient-hacks-his-own-mor... ). Go on Shodan and search for medical devices and terminology (e.g.…

The sad part is that the companies will use this security by obscurity argument against open sourcing.

contrary to popular opinion....

Obscurity is good practice as one layer of a layered defence system.

See "Defence in Depth" https://en.wikipedia.org/wiki/Defense_in_depth_(computing)

"Defense in depth is originally a military strategy that seeks to delay rather than prevent the advance of an attacker by yielding space to buy time".

We have to acknowledge that no system is perfect, there will always be holes, therefore a good approach is to layer up the imperfect systems which delays the attacker.

Obscurity is one of those layers, a system will always be more secure if you have to find it first.

Re: “I Want to Know What Code Is Running Inside My Body”

#98

Post by nitrogen » Tue, Aug 09, 2016, 6:33 AM UTC

TickleSteve wrote: post #97

cooper12 wrote: post #75

Earlier quoted context omitted.

The sad part is that the companies will use this security by obscurity argument against open sourcing.

contrary to popular opinion.... Obscurity is good practice as one layer of a layered defence system. See "Defence in Depth" https://en.wikipedia.org/wiki/Defense_in_depth_(computing) "Defense in depth is originally a military strategy that seeks to delay rather than prevent the advance of an attacker by yielding space to buy time". We have to acknowledge that no system is perfect, there will always be holes, therefor…

True, but history has demonstrated countless times that closed source code doesn't provide near enough obscurity to deter hackers, and automated fuzzing tools make it even easier.

Re: “I Want to Know What Code Is Running Inside My Body”

#99

Post by TickleSteve » Tue, Aug 09, 2016, 6:38 AM UTC

You most certainly don't want people to be able to modify safety critical code within a pacemaker.

What most developers don't realise is the level of engineering strictness that goes into anything safety-related. The rules and regulations related to anything that affects the human body is in a different league than what most developers are familiar with.

What is a problem here, is that the design (not the code) apparently did not take into account any messaging security, relying on obscurity as its only defence.

If the code was open-sourced, don't expect to find lots of buffer overflow attack vectors, or simple things like that. Its the design of the system as a whole at fault, and that is already open.

Medical devices such as these are not black boxes to the people that certify them, everything is open to them, source included. Having worked in that sort of area, I trust the systems that are in place.

Re: “I Want to Know What Code Is Running Inside My Body”

#100

Post by grogenaut » Tue, Aug 09, 2016, 6:45 AM UTC

function_seven wrote: post #69

tdicola wrote: post #50

Earlier quoted context omitted.

Does the FDA have the knowledge and experts to really understand if the firmware is good or bad though?

That's a tough one to answer. The cynic in me says probably not. That they're so focused on pharmaceuticals and "analog" medical devices that they haven't developed those capabilities. But I also know that the FDA is a massive organization, and there's no reason they couldn't hire for this specific purpose. But then the cynic says that government pay grades may not be up to snuff. See the HCA rollout and subsequent r…

There have been medical devices external to the body for a long time. Therac 25 (1982) is world wide web (1989). Airplanes were first flight controlled by computer in 1958, and commercially in the concorde in 1969.

Think about that for a minute. There has been an official government review process for safety of computer controlled airplanes since the 70s, and a good decade for medical devices (likely earlier) 10 years earlier than the WWW existed.

I think it's kind of funny that you'd think that you were doing it better than production life critical systems THAT CAN ACTUALLY KILL PEOPLE that have been around at least ten if not 30 years longer than the tech you likely (sorry assumption) base your career around.