Elon Musk emails employees about 'extensive and damaging sabotage' by employee • Hacker News Board

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#431

Post by bildung » Tue, Jun 19, 2018, 11:18 AM UTC

toastermoster wrote: post #49

I had heard from a third party vendor on the site that Profibus issues (for communication to drives) was one of the major issues at site. I wonder if that was accurate or still true if so.

Profibus is a generic protocol for sensors and machines talking to each other (analog to JSON for web resources) so that essentially means they had problems with systems integration, e.g. getting machines integrated into the production lines.

Pretty surprising actually, I thought their problems were beyond that hurdle (e.g. the automation already works but has reliability problems).

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#432

Post by wslh » Tue, Jun 19, 2018, 11:19 AM UTC

cottsak wrote: post #84

flyingswift wrote: post #65

I made this same comment on the other discussion. I find it concerning that one person was able to push malicious code to 'production'. To me, this suggests that Tesla, a company building highly sensitive software, does not employ basic branch policies. How is is it that these changes could have made it through a code review process and get deployed? If a company like Microsoft or Google announced that a disgruntled…

This is a very naive comment. There will always be a small handful of engineers that can push the button to move code into PROD or even change code in PROD live. Ideally, with mature controls, the people in this list is short. But to jump to the conclusion that Tesla doesn't use good practises is very short sighted. Who's to say that external parties didn't target this person specifically because of their role/influe…

In addition, it would be easy to inject obfuscated bugs that are extremely difficult to find.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#433

Post by mtgx » Tue, Jun 19, 2018, 11:20 AM UTC

taneq wrote: post #342

flyingswift wrote: post #65

I made this same comment on the other discussion. I find it concerning that one person was able to push malicious code to 'production'. To me, this suggests that Tesla, a company building highly sensitive software, does not employ basic branch policies. How is is it that these changes could have made it through a code review process and get deployed? If a company like Microsoft or Google announced that a disgruntled…

> I find it concerning that one person was able to push malicious code to 'production'. Production line software is almost certainly handled separately from the software that runs their vehicles, and isn't "production" in the usual web sense of being customer facing. This sounds more like someone messed with their factory automation setup.

That may indeed be the case, but I can still see Tesla ship "damaged" cars, that could lead to crashes, because of something like this.

Furthermore, what if Tesla has the same kind of "security" in the software division, too? Can just about anyone in the Tesla team develop and push malicious code to the Autopilot?

I've always thought that Tesla has "better than most" software security for its cars, but I've never thought even that was "good enough". I don't think any carmaker has the kind of extensive security measures put in place that should be required for something that drives you to places on its own at high-speed, can be accessed remotely, and receives OTA updates.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#434

Post by jrs95 » Tue, Jun 19, 2018, 11:22 AM UTC

alxlaz wrote: post #339

cottsak wrote: post #84

Earlier quoted context omitted.

This is a very naive comment. There will always be a small handful of engineers that can push the button to move code into PROD or even change code in PROD live. Ideally, with mature controls, the people in this list is short. But to jump to the conclusion that Tesla doesn't use good practises is very short sighted. Who's to say that external parties didn't target this person specifically because of their role/influe…

Obviously, not all details are available, but the wording in the email suggests that the parent comment is anything but naive: > This included making direct code changes to the Tesla Manufacturing Operating System under false usernames and exporting large amounts of highly sensitive Tesla data to unknown third parties. This sounds like something out of the 1990s, that dark and romantic era of version control when we…

Sounds like he gained the ability to create users, so he probably used that to get around any automated code review process as well.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#435

Post by notacoward » Tue, Jun 19, 2018, 11:23 AM UTC

Ataraxy wrote: post #143

notacoward wrote: post #59

Earlier quoted context omitted.

I was thinking the same thing. Maybe these allegations will turn out to have some merit. On the other hand, Musk has been sounding a bit like Trump lately ... and I don't just say that because I'm not a fan of either. There's a similar "I'm here to blow up the corrupt order and everything bad that happens is because of them conspiring against me" narrative at play. For Trump it's the Deep State. For Musk it's Big Oil…

Just as eery are the 5 comments in a row I just read comparing him to Trump.

Oh, you mean all those responses to one top level comment, half of them quoting and/or disagreeing, vs. the "$10B in shorts" replies to every top-level comment? Totally the same thing. Both sides do it.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#436

Post by finnthehuman » Tue, Jun 19, 2018, 11:27 AM UTC

nordsieck wrote: post #427

finnthehuman wrote: post #391

Earlier quoted context omitted.

Are you trying to us that less stringent controls on manufacturing software is in any way acceptable? Manufacturing process (including mfg software tools) are a huge potential source of product failure. I don't know about automotive QMS specifically, but I work in embedded software for safety critical systems. If an unreleased procedure or unreleased software is used to build the hardware, or unreleased software is r…

>> Production line software is almost certainly handled separately from the software that runs their vehicles, and isn't "production" in the usual web sense of being customer facing. > Are you trying to us that less stringent controls on manufacturing software is in any way acceptable? Yes. It is simply uneconomical to develop all software in a no-bugs-allowed NASA style process. One of the classes of software that c…

Yeah, in that case the controls are moved from the development process to the downstream verification. Sorry if I wasn't clear, I originally had some wording about how known unknowns are handled in my post, but it was getting unnecessarily long and felt like a diversion from the hypothetical I was responding to. I was focused more on documentation are repeatability of process than the quality of the tool themselves.

>> I find it concerning that one person was able to push malicious code to 'production'.

>Production line software is almost certainly handled separately [...] This sounds more like someone messed with their factory automation setup.

For a production line, I would still expect a level of control over exactly what software was running and how it was configured, even if it was SOUP.

edit: SOUP not SOAP. I need to stop posting in the morning.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#437

Post by dpcx » Tue, Jun 19, 2018, 11:30 AM UTC

alxlaz wrote: post #421

alanfranzoni wrote: post #385

Earlier quoted context omitted.

> 1. Why can anyone, regardless of credentials, push mission-critical code without review (or, alternatively, if the changes did go through review, why did the review process not catch multiple malicious changes?) Why do you suppose the unauthorized party was following the company's development practices? Maybe it was from the sysadmin side, somebody who worked on the toolchain used for reviewing and pushing things t…

> So he was able to sidestep the normal review process. He should not have been able to sidestep the normal review process . That's the problem in the first place. Even if you're from the sysadmin side. It should not be possible to do it. You may think that looks exaggerated but I've worked in two places where we implemented such a process, both of them far more boring than Tesla and, I suspect, far less money to bur…

I'm interested to know how you plan to keep someone with `wheel` access from doing anything on a server that they maintain.

No, seriously.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#438

Post by rjdagost » Tue, Jun 19, 2018, 11:33 AM UTC

nordsieck wrote: post #427

finnthehuman wrote: post #391

Earlier quoted context omitted.

Are you trying to us that less stringent controls on manufacturing software is in any way acceptable? Manufacturing process (including mfg software tools) are a huge potential source of product failure. I don't know about automotive QMS specifically, but I work in embedded software for safety critical systems. If an unreleased procedure or unreleased software is used to build the hardware, or unreleased software is r…

>> Production line software is almost certainly handled separately from the software that runs their vehicles, and isn't "production" in the usual web sense of being customer facing. > Are you trying to us that less stringent controls on manufacturing software is in any way acceptable? Yes. It is simply uneconomical to develop all software in a no-bugs-allowed NASA style process. One of the classes of software that c…

No one is saying "no bugs" NASA level code is required. Basic SQA must however be conducted- factory automation software is not something where you can settle for low quality as you seem to imply. Something tells me you've never worked in factory / production environments. There are critical safety requirements that must be checked, especially in environments where machinery is operating in proximity to workers. Not only that, but there are huge financial incentives in place to ensure high quality. A bug in an early step of a manufacturing process can cause massive amounts of rework or even scrap, leading to huge losses.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#439

Post by daemin » Tue, Jun 19, 2018, 11:37 AM UTC

alxlaz wrote: post #423

daemin wrote: post #392

Earlier quoted context omitted.

Some manager asks IT multiple times over the course of a few weeks to create an account for a contractor, then give them permissions to access production type machines. Or a contractor that was fired had their credentials appropriated by this manager, perhaps by that manager removing them from a "delete these accounts" list. Those are a couple of mundane ways of getting a false username to a production machine. This…

> Some manager asks IT multiple times over the course of a few weeks to create an account for a contractor, then give them permissions to access production type machines. And -- keeping in mind that production type machines operate machinery that can kill -- this sounds okay to you? Not to mention this: > Or a contractor that was fired had their credentials appropriated by this manager, perhaps by that manager removi…

Someone has to be responsible for managing people and organising access to the appropriate machines for them to do their job, if it isn't their manager then who is?

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#440

Post by AYBABTME » Tue, Jun 19, 2018, 11:39 AM UTC

flyingswift wrote: post #65

I made this same comment on the other discussion. I find it concerning that one person was able to push malicious code to 'production'. To me, this suggests that Tesla, a company building highly sensitive software, does not employ basic branch policies. How is is it that these changes could have made it through a code review process and get deployed? If a company like Microsoft or Google announced that a disgruntled…

That's all good and nice but in most cases, this is a matter of policy. If someone actively tries to circumvent the policy or the process, odds are that most software shops would fall victim to the same thing. Especially in the case of internal software. Even if you have a system that attempts to enforce the process, odds are that the system isn't without flaw and is not too hard to circumvent.

Most businesses who's primary money maker is not defense or security related... are not running their operation with a focus on defending themselves from themselves.