Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“I Want to Know What Code Is Running Inside My Body”

backchannel.com

Post reply on HN
Print view View original

131–140 of 217 posts

Re: “I Want to Know What Code Is Running Inside My Body”

#131

Post by munificent »

TickleSteve wrote: post #115
adrianN wrote: post #108

Earlier quoted context omitted.

Industrial control systems are also safety critical systems that have to adhere to very similar regulations as medical systems. Yet, when you ask hackers in that field you'll quickly learn that they have terrible code and abysmal security. Rules and regulations do very little to improve code quality and security, imho. Most things in these regulations are either best practices that any software engineer does (write t…

industrial control != safety critical. "industrial control" covers both certified and non-certified code, you will have to be more specific than that for the purposes of this conversation. Quite often, cases such as the pacemaker are flaws in system design , not code . (e.g. no secure messaging, probably due to the lack of awareness when it was designed). That is not to say that safety-critical code is perfect... jus…

> That is not to say that safety-critical code is perfect... just that it has a lot more rigour and inspection involved than run-of-the-mill website code.

I had assumed that as well until all of the horror stories around Toyota's firmware came to light.

https://en.wikipedia.org/wiki/2009%E2%80%9311_Toyota_vehicle...

Re: “I Want to Know What Code Is Running Inside My Body”

#132

Post by michaelt »

TickleSteve wrote: post #99

You most certainly don't want people to be able to modify safety critical code within a pacemaker. What most developers don't realise is the level of engineering strictness that goes into anything safety-related. The rules and regulations related to anything that affects the human body is in a different league than what most developers are familiar with. What is a problem here, is that the design (not the code) appar…

  rules and regulations [...] in a different league than what
  most developers are familiar with. [...] the design (not the
  code) apparently did not take into account any messaging
  security, relying on obscurity as its only defence.
Imagine you were building a suspension bridge to the highest safety standards, and you had people with microscopes manually inspect every grain of sand and cement that went into the foundations. But the suspension cables were made of old washing lines.

You would be achieving a very high and a very low standard of inspection at the same time.

Some people would say standards of inspection are only as strong as their weakest link; that the whole claim of such inspection is to prove the absence of such weak links; and that a standard that has approved such a bridge has, by so doing, shown the claimed proof of safety is a useless joke.

Other people would say the inspection is a checklist of common errors, rather than a proof of the safety of every aspect of the system; that proof of perfect safety was never its claim or goal; and that this merely shows there should be some extra boxes on the checklist, which was in any case a constantly evolving document.

Re: “I Want to Know What Code Is Running Inside My Body”

#133

Post by throwanem »

TickleSteve wrote: post #112
j-pb wrote: post #111

Earlier quoted context omitted.

Software certification is a joke, and mostly consists of checking off a few boxes on workflow and development methods. Smart electricity meters are safety critical and I know students writing on it in their summer jobs. Airplanes are safety critical and they accidentally bridged the on plane wifi into the fly by wire system. Unless this stuff is written in COQ and formally verified, I won't trust it a bit.

Certification most certainly isn't a joke. It involves a hell of a lot of work, enough to double or triple the software portion of a project. Whatever certification you have been involved with may have been a joke. Safety-critical certification most certainly is not. BTW: I'm s/w architect on a set of smart meters. And they most certainly are not safety critical. (IMO They should be, but thats a separate issue). ...a…

Not Wifi, but IIRC a guy called Chris Roberts was able to leverage a wireline connection to an underseat IFE controller into FADEC control on multiple aircraft. Last I heard of the case, he might be going to prison for it, so apparently at least some people find him credible.

Similarly, witness recent revelations about car hackability - I forget if it was Blackhat or DEFCON where a couple of guys demoed a fully remote attack through the entertainment system's cell network access that culminated in the ability to override the steering, throttle, and brake controls.

Maybe these are not what is meant by "safety critical" in your use of the phrase. If that's so, I think it would clarify matters greatly for you to define what you mean by it that's more specific than "can kill people if it goes wrong".

Re: “I Want to Know What Code Is Running Inside My Body”

#134

Post by throwanem »

grogenaut wrote: post #100
function_seven wrote: post #69

Earlier quoted context omitted.

That's a tough one to answer. The cynic in me says probably not. That they're so focused on pharmaceuticals and "analog" medical devices that they haven't developed those capabilities. But I also know that the FDA is a massive organization, and there's no reason they couldn't hire for this specific purpose. But then the cynic says that government pay grades may not be up to snuff. See the HCA rollout and subsequent r…

There have been medical devices external to the body for a long time. Therac 25 (1982) is world wide web (1989). Airplanes were first flight controlled by computer in 1958, and commercially in the concorde in 1969. Think about that for a minute. There has been an official government review process for safety of computer controlled airplanes since the 70s, and a good decade for medical devices (likely earlier) 10 year…

Uh, did you really just cite Therac 25 in favor of safety review? You do know that was the one that had a bug which slipped past review and killed some people, right?

Your overall point is well taken, but maybe put a little more thought into the examples you pick to support it...

Re: “I Want to Know What Code Is Running Inside My Body”

#135

Post by xenophonf »

TickleSteve wrote: post #112
j-pb wrote: post #111

Earlier quoted context omitted.

Software certification is a joke, and mostly consists of checking off a few boxes on workflow and development methods. Smart electricity meters are safety critical and I know students writing on it in their summer jobs. Airplanes are safety critical and they accidentally bridged the on plane wifi into the fly by wire system. Unless this stuff is written in COQ and formally verified, I won't trust it a bit.

Certification most certainly isn't a joke. It involves a hell of a lot of work, enough to double or triple the software portion of a project. Whatever certification you have been involved with may have been a joke. Safety-critical certification most certainly is not. BTW: I'm s/w architect on a set of smart meters. And they most certainly are not safety critical. (IMO They should be, but thats a separate issue). ...a…

I'm not sure, but your correspondent may be referring to Hugo Teso's 2013 presentation for Hack in the Box, http://conference.hitb.org/hitbsecconf2013ams/hugo-teso/, or possibly to design revisions of the Boeing 777.

Re: “I Want to Know What Code Is Running Inside My Body”

#136

Post by throwanem »

ben_jones wrote: post #55
beambot wrote: post #28

Earlier quoted context omitted.

With or without a warrant...? EDIT: Not sure what's up w/ the downvotes. There are well-established ways for regulatory agencies (whether FDA, FCC, etc) to obtain firmware for devices -- and it almost always involves a warrant under extraneous circumstances -vs- proactively receiving proprietary code.

What if they had to subpoena a tomato grown in a field next to the toxic waste dump? What if they opted not to examine that tomato because they didn't have the resources to issue, process, and support, the lengthy bureaucratic process involved in such things?

To extend the metaphor, if the farm that grew the tomato is selling them to people to eat, I don't see that a subpoena is or should be required. But it breaks down anyway, because a tomato's source code is right there in it, with no opaque binary blobs to worry about trying to decompile.

Re: “I Want to Know What Code Is Running Inside My Body”

#137

Post by belorn »

TickleSteve wrote: post #97
cooper12 wrote: post #75

Earlier quoted context omitted.

The sad part is that the companies will use this security by obscurity argument against open sourcing.

contrary to popular opinion.... Obscurity is good practice as one layer of a layered defence system. See "Defence in Depth" https://en.wikipedia.org/wiki/Defense_in_depth_(computing) "Defense in depth is originally a military strategy that seeks to delay rather than prevent the advance of an attacker by yielding space to buy time". We have to acknowledge that no system is perfect, there will always be holes, therefor…

Security by obscurity is in practice almost always a bad idea.

1# security by obscurity gives a false sense of security. Under no circumstance should obscurity be used as a deciding factor behind a management decision.

2# security by obscurity cost money and time, and should only be used when all real form of security measures has been implemented. Even the military are currently not always implementing multi-token authentication, ipsec and selinux. Instead of trusting that the medical deceive is safe behind two layers, a static password and a secret port, add a certificate and implement challenge and response.

3# the priority to implement security by obscurity should be far lower than all the real security technologies. When reading security reports by pen testers, its important to understand the difference between a verified code injection vulnerability and a system information disclosure. Fixing a remotely code injection bug is much more important than hiding the fact that a system is running a up-to-date stable version of Debian, yet many security guides and reports for pen-testing tools rarely priorities.

4# security by obscurity often has real cost in support, brittleness of the system, and debugging. There is still in 2016 firewalls that will permanent block any ip address that has sent a icmp package to them. The amount of work employees are spending to unblock customers that accidentally end up in the block list could be spent on making sure that the system is just that more perfect against the serious attackers who can afford to spend $50 to access a botnet for a few hours.

Re: “I Want to Know What Code Is Running Inside My Body”

#138

Post by throwanem »

Illniyar wrote: post #91
radicality wrote: post #82

How about when I'm flying an airplane; I'm also putting my life in the hands of people that wrote the code that controls it and I have to trust that the plane won't shut itself down mid-flight because of faulty code. Should a similar argument be made here?

I think the difference between these two are 2-fold - first is ownership and second is personal. A pacemaker is something you bought and owned, when flying in a plane you are buying a service, this is similar to earlier discussions about being able to change your car's software under DMCA etc... A pacemaker is also personal, in that it's something that only you have and for your specific pacemaker the only affected p…

> A pacemaker is something you bought and owned

Sure about that? How much did you pay for, and how much did your insurance cover? What's the proportion look like?

Before you downvote, note: I don't like this argument. I find it downright horrifying. But I can't imagine no one will ever make it in a serious way, so it bears considering how to respond.

Re: “I Want to Know What Code Is Running Inside My Body”

#139

Post by TickleSteve »

michaelt wrote: post #132
TickleSteve wrote: post #99

You most certainly don't want people to be able to modify safety critical code within a pacemaker. What most developers don't realise is the level of engineering strictness that goes into anything safety-related. The rules and regulations related to anything that affects the human body is in a different league than what most developers are familiar with. What is a problem here, is that the design (not the code) appar…

rules and regulations [...] in a different league than what most developers are familiar with. [...] the design (not the code) apparently did not take into account any messaging security, relying on obscurity as its only defence. Imagine you were building a suspension bridge to the highest safety standards, and you had people with microscopes manually inspect every grain of sand and cement that went into the foundati…

Inspection in this case is a proof that the design you're certifying is fit-for-purpose.

So in your example, it would show that the loading and stresses on your washing lines were sufficiently low to meet the safety margin of the bridge.

Certification of safety-related and medical devices is not a check-box exercise, it looks at the dynamic and static behaviour of your system as a whole (not just the software).

Re: “I Want to Know What Code Is Running Inside My Body”

#140

Post by TickleSteve »

belorn wrote: post #137
TickleSteve wrote: post #97

Earlier quoted context omitted.

contrary to popular opinion.... Obscurity is good practice as one layer of a layered defence system. See "Defence in Depth" https://en.wikipedia.org/wiki/Defense_in_depth_(computing) "Defense in depth is originally a military strategy that seeks to delay rather than prevent the advance of an attacker by yielding space to buy time". We have to acknowledge that no system is perfect, there will always be holes, therefor…

Security by obscurity is in practice almost always a bad idea. 1# security by obscurity gives a false sense of security. Under no circumstance should obscurity be used as a deciding factor behind a management decision. 2# security by obscurity cost money and time, and should only be used when all real form of security measures has been implemented. Even the military are currently not always implementing multi-token a…

I agree with your numbered points, but not the conclusion that its always a bad idea.

Its common sense that I can't pick a lock if I cant find the lock.

This says nothing about the quality of the lock or what is behind the lock.

Post reply on HN