31–40 of 89 posts
And even in the case of bridges and the like, they do have a tendency to collapse to earthquakes, wars, and the like. Given enough time, most everything has bugs.
I think we all can agree that programmers almost never begin a project with the intention of causing harm. All the examples cited involved situations where immense pressure applied by higher-up impel a programmer to knuckle-under and agree to such approaches.
The solution that is offered seems very specific to now - we take the fall guy who caved in to one sort of incentive and we put an opposite incentive on him to force a different behavior (btw without removing the first incentive). How fucked-up is that?
Obviously, the better, saner solution is removing the existing perverse incentives, giving software engineers more leverage in decisions, punishing high-ups if they don't give software engineers autonomy to make decisions. And heck, execute the CEOs when the bridges collapse. The bucks should stop with them, right?
When people bring up a code of ethics, actuaries seem to come up a lot because they can potentially lose their license for mis-evaluating risks, but anecdotally what I've heard is that in reality if your boss asks you to sign off on something, and you refuse they'll find someone else to sign off on it and just label you as not a team player. At which point you've harmed your career and had no real impact. I bring thi…
That's true. Is there a value to refusing to do work, even though it might not be able to stop something from being built in spite of it? If you know, a priori, refusing to work on something you consider to be unethical will have /zero/ effect on the overall project, but negatively affect you, is it then ethical to do the work, or are you required to fall on your sword?
1) Listen to all competing arguments. Make sure you are making a sound decision and your opposition is genuine and not based on a personal bias.
2) Make it explicit that you are are saying no in your role as a professional, and that this is not just a casual "no". Ethics require engineers to make decisions only in areas of competency, so make it clear that you are competent to make this decision, and your decision derives directly from your competency.
3) Write it down. The document should cover 2) above, and set down the defensible reasons why assent is being refused, addressing counter arguments. Consider that the arguments you write down could well be tested in court, and you will be cross examined on them. The document/report should be of the same level as if you were a consultant to your employer.
4) Sign and submit the document (keeping a copy for yourself). You need to be open about your opposition. Keep it at a professional level. Hopefully you will get some respect, as your document should make it clear that you have legitimate concerns and are not just being difficult.
5) Be prepared for the consequences. Your opposition is now in writing and impossible to ignore. No sane boss is going to ignore it, as the legal consequences of doing so are too great if you turn out to be right. You will have heat applied to you, as the alternatives are to either address your concerns or to get you to retract, and the latter option is cheaper.
6) If the final decision is against you, either live with the decision or resign. If the issue is of sufficient gravity your only option may be to resign, whether that be for ethical reasons or your own legal protection. (Yes your Honour, my work killed 100 people, and I was fully aware of it as shown by this document that I wrote.)
If you end up resigning, then you're probably making the right decision anyway, as the events leading up to your resignation will have shown your employer to be a dud.
Granted, reality won't be as cut and dried as above, and will involve a lot of anguish.
Earlier quoted context omitted.
This is simply false. True systemic quality will lower costs, speed production, and increase value all at once. The reason is that, beyond a certain organizational complexity, we fail at creating the systems necessary to create the positive feedback loops necessary for quality, low cost, and fast execution to flourish naturally. Instead, we live in dysfunctional organizations where the tradeoffs prevail, and we choos…
You've stated that it's not more expensive. But everyone in the industry knows otherwise. Unless you can produce some evidence to the contrary, I'm going to insist it's you who is incorrect here.
When looking at increasing quality (essentially, scope) in isolation of systems and the environment in which that quality is created, then the only way to increase quality is to increase cost. That much is true, but it's also incomplete.
When looking at and eventually optimizing the whole system—from the customer and their needs, to the leadership of the company, to the clarity of purpose, to values and culture, to the intrinsic motivation of your employees, to methods of management, to your learning and education processes, to your processes of production, to your communication links, to your output and how it's measured, and how each is controlled and understood in great detail—you can and will improve quality much more than you could by simply increasing cost in isolation—and at the same time, you'll decrease your overall costs in the long term, you'll speed up production, and you'll increase sales and customer satisfaction. The end result is a better product, for lower cost, with more predictability, proud happy employees, and faster to boot. That's what Quality really means.
Looking at quality in terms of cost alone is the naive mistake. Look at the whole system.
And for some evidence, look at Japan circa 1948. These exact methods of systemic quality, "total quality control" and statistical quality control taught to Japan by W. Edwards Deming turned them from the ruins of World War II into the second largest economy in the world, to the point of such success that Japanese companies were putting American ones to shame by the 70's. They did this by following exactly the model I've described above, decreasing total costs, increasing quality, and improving time to market, all by focusing on their total systems across the organization. The proof is in the success of an entire country's economy, I don't know how it can get much clearer than that.
It is not my fault that everyone in our industry has forgotten these lessons on running companies, but the prevailing methods remain incorrect, as does your original statement.
Earlier quoted context omitted.
That's true. Is there a value to refusing to do work, even though it might not be able to stop something from being built in spite of it? If you know, a priori, refusing to work on something you consider to be unethical will have /zero/ effect on the overall project, but negatively affect you, is it then ethical to do the work, or are you required to fall on your sword?
I'd contend that you don't know that it will have zero effect until you have tried, and how you go about refusing assent is of critical importance. If you simply say "no", your boss will move onto the next person for assent, and you will be left vulnerable. Make it a formal process, it it may well have an effect. 1) Listen to all competing arguments. Make sure you are making a sound decision and your opposition is ge…
We can write software like this. The reason why we don't has been explained many times: It's too expensive, by many orders of magnitude.
If it's financially impossible to make a product that doesn't harm people, you don't say "gosh, sorry, but your safety is incompatible with our fiscal goals." You cancel the fucking product. And beyond that, I don't believe for a moment that--to use the example from the article--it would have increased Grindr's costs by 1000x if they had taken a little while to brainstorm possible negative consequences of an app that…
How many engineers should this company add to act 'ethically'? According to you, they need to cancel the product (I think - only for an ungenerously rigid interpretation of your comment).
Risk is not 'it harms people or not'. Knowing whether your product harms people is more difficult still. And the question of who is responsible, more still. (and that's not just between the supplier and the user - also to other actors. To take the Grindr example, was it an engineering problem, a product design problem, a user education problem or a governance problem? I could argue for all of them. Who should act to mitigate the risk? I don't know, and it's not as clear-cut (in the abstract) as some here are letting on).
We can write software like this. The reason why we don't has been explained many times: It's too expensive, by many orders of magnitude.
If it's financially impossible to make a product that doesn't harm people, you don't say "gosh, sorry, but your safety is incompatible with our fiscal goals." You cancel the fucking product. And beyond that, I don't believe for a moment that--to use the example from the article--it would have increased Grindr's costs by 1000x if they had taken a little while to brainstorm possible negative consequences of an app that…
Escalation steps inevitably involve a 'nuclear option' whose size and expense relegate it to just the most sure and most aggrieved wrong parties. These days that option is a class-action lawsuit. It seems to work best when deterrence is the primary goal, plaintiffs rarely ever get anything of value out of it.
It works as you would expect, but one really wants a pathway to punish negligence without having to get lawyers involved. I believe this is usually done with market solutions to make 'buyer beware' less awful on the ignorant.
But the sheer scale of the stated problem, forcing all web service developers everywhere, to consider possible lines of attack on their software, under pain of loss of livelihood, seems too general and too unenforceable to be useful.
Until it is the CEO..CTO..and vc that is standing under the bridge...nothing will change.
OK, I think we all can agree that programmers almost never begin a project with the intention of causing harm. All the examples cited involved situations where immense pressure applied by higher-up impel a programmer to knuckle-under and agree to such approaches. The solution that is offered seems very specific to now - we take the fall guy who caved in to one sort of incentive and we put an opposite incentive on him…
There's a diffusion of responsibility that makes it hard to point to any given person after the fact. How can we arm people - software engineers, but product designers, project managers, QA engineers, CEOs, and the welders on the assembly line can say, "Hold on, stop, this isn't right!"?
Earlier quoted context omitted.
You've stated that it's not more expensive. But everyone in the industry knows otherwise. Unless you can produce some evidence to the contrary, I'm going to insist it's you who is incorrect here.
That's a highly simplistic and incorrect viewpoint—and the correctness of a model is not dependent on popular vote. When looking at increasing quality (essentially, scope) in isolation of systems and the environment in which that quality is created, then the only way to increase quality is to increase cost. That much is true, but it's also incomplete. When looking at and eventually optimizing the whole system—from th…
I'm betting it's you.
Please do prove us all wrong by starting up a software company that produces bug free software faster and cheaper than every other software company on the planet. There's a big pot of gold at the end of that journey. Go and claim it. If everyone else is as incompetent as you claim, it's going to be a walk in the park.