Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Microsoft President: We Need a Hippocratic Oath for Software Engineers

capitalandgrowth.org

Post reply on HN
Print view View original

371–380 of 544 posts

Re: Microsoft President: We Need a Hippocratic Oath for Software Engineers

#371

Post by stoneman24 »

leeoniya wrote: post #145
ezoe wrote: post #113

Earlier quoted context omitted.

The problem is, software can be really complex it's possible to make it so no one programmer can understand the whole picture. The tasks can be divided so the individual programmers are given order like "do X in condition of Y" which itself seems harmless and lawful, but combining them lead to malicious behaviours.

this is precisely the problem. few physicists knew they were all collectively building a nuclear bomb because the big picture was well compartmentalized. feynman only found out because he picked the locks of his colleagues to piece it all together.

Just finished “Atomic” by Tim Baggot. A history of the development of the bomb. From my reading, lots of the physicists knew. They pushed forward as they didn’t know the extent of the german project (who had stalled due to lack of resources and a belief that while possible, a bomb wasn’t practical due to a need for a large amount of material). The safe-cracking was for amusement.

Re: Microsoft President: We Need a Hippocratic Oath for Software Engineers

#373

Post by TeaDrunk »

imtringued wrote: post #358
nicoburns wrote: post #272

Earlier quoted context omitted.

> Software engineers are accountable to their bosses before their users, no matter how high minded we like to pretend to be. Isn't that what a hipppocratic oath would solve. They'd be accountable to the oath before their bosses, and that would give them reasonable grounds to refuse unethical work.

How do you guarantee that the boss will cooperate with the software engineers and tell them about all the unethical business practices that are currently happening?

Eventually it would have to be told to a CTO, president or Vice President it some technological product, software architect, etc. Software Engineers generally need to understand the use case of a product to be able to develop something effective. It would be exceptionally difficult to convince someone to code something that creates fake profiles of people who don’t use your site and no one have any idea that that’s what’s happening.

Re: Microsoft President: We Need a Hippocratic Oath for Software Engineers

#374

Post by bumby »

imtringued wrote: post #329
skissane wrote: post #118

Earlier quoted context omitted.

> Said another way, engineers now need to be technical and legal experts in the business domain? Consider something like the 737 MAX debacle – did the programmers writing the MCAS code actually have enough aviation domain knowledge and understanding of where the component fit in the overall system to realise it was a threat to people's lives? I don't know, but my guess is the most likely answer is "No".

From my limited information the MCAS code was primarily causing problems in association of incorrect readings by damaged sensors. Of course one could argue that this is an engineering failure because the MCAS failed to account for wrong sensor input but when you consider the legal implications of a MCAS fallback there is actually not much that can be done on the software side. The MCAS is an optional component that r…

>actually not much that can be done on the software side.

This isn’t exactly true. There are mitigations (both software and non-software) that are expected to be done depending on hazard analysis. One of the items discovered is Boeing mischaracterized the MCAS hazard (it should have had a “catastrophic” hazard class). In addition, they didn’t appear to follow their own process for dual inputs required even for the lower severity class assigned. The “optional” part of MCAS was the secondary sensor reading into the software

Re: Microsoft President: We Need a Hippocratic Oath for Software Engineers

#376

Post by adaml_623 »

alacombe wrote: post #251
hyko wrote: post #147

It should be noted that no ethically-trained software engineer would ever consent to write a DestroyBaghdad procedure. Basic professional ethics would instead require him to write a DestroyCity procedure, to which Baghdad could be given as a parameter. -Nathaniel Borenstein

static void destroyBaghdad() { destroyCity("Baghdad"); } Here, it's done.

20% success rate there.

https://geotargit.com/called.php?qcity=Baghdad

Re: Microsoft President: We Need a Hippocratic Oath for Software Engineers

#377

Post by TeaDrunk »

Cthulhu_ wrote: post #362
madrox wrote: post #8

I’m not a fan of comparisons to the Hippocratic oath. The greatest risk to AI ethics is not the ethics of software engineers but the ethics of the software engineering process. By the time tasks are handed to engineers, most of the ethical decisions have been made by product managers, designers, and business stakeholders who are focused on their own goals. Software engineers are accountable to their bosses before the…

Pretty much. Those that take the oath would quickly see themselves not hired for certain positions, in favor of those that didn't take the oath.

This assumes all of the vast majority of engineers are easily replaceable. Given that several tech companies prefer the chance of not hiring a good fit over the chance of needing to fire someone, I think the cost to a company if a significant subset of its engineers declined unethical work (especially engineers who at this point are high enough to have a good idea of the broad scope of the software) would make it difficult to perform what you suggest.

Re: Microsoft President: We Need a Hippocratic Oath for Software Engineers

#380

Post by dTal »

imtringued wrote: post #329
skissane wrote: post #118

Earlier quoted context omitted.

> Said another way, engineers now need to be technical and legal experts in the business domain? Consider something like the 737 MAX debacle – did the programmers writing the MCAS code actually have enough aviation domain knowledge and understanding of where the component fit in the overall system to realise it was a threat to people's lives? I don't know, but my guess is the most likely answer is "No".

From my limited information the MCAS code was primarily causing problems in association of incorrect readings by damaged sensors. Of course one could argue that this is an engineering failure because the MCAS failed to account for wrong sensor input but when you consider the legal implications of a MCAS fallback there is actually not much that can be done on the software side. The MCAS is an optional component that r…

>The MCAS is an optional component that reduces certification and training costs

No. The MCAS was a "necessary" component for pitch stability - without it, a 737 MAX in a pitch-up attitude would, in the absence of correcting inputs, pitch up further and further until it stalled. Without it, the airframe is uncertifiable, full stop.

Post reply on HN