Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elon Musk emails employees about 'extensive and damaging sabotage' by employee

cnbc.com

Post reply on HN
Print view View original

421–430 of 627 posts

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#421

Post by alxlaz »

alanfranzoni wrote: post #385
alxlaz wrote: post #339

Earlier quoted context omitted.

Obviously, not all details are available, but the wording in the email suggests that the parent comment is anything but naive: > This included making direct code changes to the Tesla Manufacturing Operating System under false usernames and exporting large amounts of highly sensitive Tesla data to unknown third parties. This sounds like something out of the 1990s, that dark and romantic era of version control when we…

> 1. Why can anyone, regardless of credentials, push mission-critical code without review (or, alternatively, if the changes did go through review, why did the review process not catch multiple malicious changes?) Why do you suppose the unauthorized party was following the company's development practices? Maybe it was from the sysadmin side, somebody who worked on the toolchain used for reviewing and pushing things t…

> So he was able to sidestep the normal review process.

He should not have been able to sidestep the normal review process. That's the problem in the first place. Even if you're from the sysadmin side. It should not be possible to do it.

You may think that looks exaggerated but I've worked in two places where we implemented such a process, both of them far more boring than Tesla and, I suspect, far less money to burn on infrastructure.

> This can happen, what is important is that such things are discovered.

No, what is important when working with mission-critical code is that such things are mitigated. Discovering such a problem in production code is already a problem, not a solution.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#422

Post by rootsudo »

flyingswift wrote: post #198
acover wrote: post #96

Earlier quoted context omitted.

I know in Microsoft it would be relatively easy to push destructive code changes. It depends on the product though. I was able to connect to production machines. There was someone running torrents in the past.

From my experience on the O365 team, it would have been difficult to push code without approval because it would have voilated a number of the compliance requirements that area of the business has to meet. But yeah, I suppose there are other areas of the company that might have more relaxed requirements

I disagree on the o365 team, since most of the programmers are WiPro and TaTa and shuffle in and out.

While I was on the Enterprise services team, looking upon IRadaS for something that broke after a push we always debated was it malicious or incompetence.

Then again, what do I know.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#423

Post by alxlaz »

daemin wrote: post #392
alxlaz wrote: post #339

Earlier quoted context omitted.

Obviously, not all details are available, but the wording in the email suggests that the parent comment is anything but naive: > This included making direct code changes to the Tesla Manufacturing Operating System under false usernames and exporting large amounts of highly sensitive Tesla data to unknown third parties. This sounds like something out of the 1990s, that dark and romantic era of version control when we…

Some manager asks IT multiple times over the course of a few weeks to create an account for a contractor, then give them permissions to access production type machines. Or a contractor that was fired had their credentials appropriated by this manager, perhaps by that manager removing them from a "delete these accounts" list. Those are a couple of mundane ways of getting a false username to a production machine. This…

> Some manager asks IT multiple times over the course of a few weeks to create an account for a contractor, then give them permissions to access production type machines.

And -- keeping in mind that production type machines operate machinery that can kill -- this sounds okay to you?

Not to mention this:

> Or a contractor that was fired had their credentials appropriated by this manager, perhaps by that manager removing them from a "delete these accounts" list.

...keeping in mind that production type machines operate machinery that can kill, does it sound OK to you that anyone can get access to an account that they don't own and control it?

This particular case would be enough to have PCI certification come into question (if not for it to be revoked), and that's just about money, not life-and-death stuff.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#424

Post by bencollier49 »

jerry40 wrote: post #334

> As you know, there are a long list of organizations that want Tesla to die. Is it correct to use 'are' here instead of 'is'?

UK vs. US English.

We (English people) say "The Football Team are playing well", for instance.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#425

Post by killaken2000 »

mmjaa wrote: post #260
pwinnski wrote: post #151

Earlier quoted context omitted.

Is the logic here that because some conspiracy theories have turned out to be true, all conspiracy theories are likely true? That seems... unwise.

No, the logic is that just calling something a conspiracy theory doesn't make it any less true. The term "conspiracy theory" has been socially engineered to automatically produce an 'ok, no need to think about it any more then' reaction, which is precisely the opposite of what needs to be done, in many cases, when enough evidence is gathered to the point that such theories are being formulated... especially when it i…

The CIA coined the term conspiracy theory (and the media used it) to discredit people who didn't believe the governments version of the Kennedy Assassination.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#426

Post by gldalmaso »

finnthehuman wrote: post #419
cottsak wrote: post #84

Earlier quoted context omitted.

This is a very naive comment. There will always be a small handful of engineers that can push the button to move code into PROD or even change code in PROD live. Ideally, with mature controls, the people in this list is short. But to jump to the conclusion that Tesla doesn't use good practises is very short sighted. Who's to say that external parties didn't target this person specifically because of their role/influe…

>There will always be a small handful of engineers that can push the button to move code into PROD or even change code in PROD live. What? Why? Nobody on my development team has access to the production code signing keys. And nobody - at all - has the ability to remotely make a production system take an unsigned update.

If a single person holds the keys, that's a different problem altogether. https://en.wikipedia.org/wiki/Bus_factor

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#427

Post by nordsieck »

finnthehuman wrote: post #391
taneq wrote: post #342

Earlier quoted context omitted.

> I find it concerning that one person was able to push malicious code to 'production'. Production line software is almost certainly handled separately from the software that runs their vehicles, and isn't "production" in the usual web sense of being customer facing. This sounds more like someone messed with their factory automation setup.

Are you trying to us that less stringent controls on manufacturing software is in any way acceptable? Manufacturing process (including mfg software tools) are a huge potential source of product failure. I don't know about automotive QMS specifically, but I work in embedded software for safety critical systems. If an unreleased procedure or unreleased software is used to build the hardware, or unreleased software is r…

>> Production line software is almost certainly handled separately from the software that runs their vehicles, and isn't "production" in the usual web sense of being customer facing.

> Are you trying to us that less stringent controls on manufacturing software is in any way acceptable?

Yes.

It is simply uneconomical to develop all software in a no-bugs-allowed NASA style process. One of the classes of software that can easily be placed in a lower scrutiny bucket is industrial software where the the result of the industrial process will be separately checked by a quality control process.

This is in dramatic contrast to safety critical software in cars, for example the battery management software or the fly-by-wire systems. Defects in these systems, in contract to industrial software, are much more difficult to discover by inspection, and failures are much more costly.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#428

Post by finnthehuman »

rndgermandude wrote: post #420
finnthehuman wrote: post #419

Earlier quoted context omitted.

>There will always be a small handful of engineers that can push the button to move code into PROD or even change code in PROD live. What? Why? Nobody on my development team has access to the production code signing keys. And nobody - at all - has the ability to remotely make a production system take an unsigned update.

But somebody has access to the signing keys/signing process. And somebody has access to the production machines. Etc.

Yes, but that changes the scenario from a "small handful of engineers" that can all do it unilaterally, to needing at least one person from N different teams to collaborate.

And in my specific case, the group with the singing keys is also the group paid to tell us "no" whenever a release is blocked by process reasons.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#429

Post by rurban »

amelius wrote: post #388
danpalmer wrote: post #366

Earlier quoted context omitted.

> To me, this suggests that Tesla, a company building highly sensitive software, does not employ basic branch policies. Are they building highly sensitive software? I would suggest that the factory automation systems would be sensitive to downtime more than anything else, and the ability to respond quickly with changes would be very valuable. Merging policies would only slow this down. I would also suggest that there…

> Are they building highly sensitive software? A car is a potential murder weapon. So yes. Especially considering it may be remote-controlled, and deployed by the millions.

This was factory automation SW, not car SW. Robots could kill people also, but afaik those robots are protected from people there, so it's only economical sabotage.

And to those were wondering about robots sabotaging security sensitive car equipment: This should not be possible. In such situations you'll always two independent chains of control, one in HW and one in SW. In the car. HW breaks down eventually and SW is always partially broken.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#430

Post by jacques_chester »

rdl wrote: post #406

Musk has every incentive to play up this incident, as long as it's concluded. A single bad event in the past can now be used to explain any number of problems. I trust him enough to believe there's a kernel of truth here, but the degree to which the sabotage actually affected operations is going to be presented in the greatest possible light.

Codename: Snowball.
Post reply on HN