Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elon Musk emails employees about 'extensive and damaging sabotage' by employee

cnbc.com

Post reply on HN
Print view View original

381–390 of 627 posts

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#381

Post by CaptainZapp »

gnode wrote: post #374
_Codemonkeyism wrote: post #340

Earlier quoted context omitted.

"There will always be a small handful of engineers that can push the button to move code into PROD" I have very different experiences from a SEC regulated company. With SOX there are controls to prevent such a thing to happen. If this is a SOX breakage, Tesla is in deep trouble with the SEC.

Can you explain how Sarbanes-Oxley applies to the situation of an employee sabotaging a production line? Specifically how it would inherently imply wrongdoing on Tesla's part.

In the financial industry part of SOX is segregation of duty.

As a developer I'm not allowed to have write access to any production system, except in an emergency via a break-glass mechanism, which is audited to the hilt and back.

It also means we're not allowed to deploy software to production systems. This has to happen via a specific chain development > regression / user acceptance testing > production.

All those environments need to be physically seperated with very specific access requirements.

The deployment process needed to be signed off by outr auditors.

I can't speak for other banks, but they probably need to implement the same -, or a similar system.

Neither can I speak for SOX requirements regarding software fo car manufacturing.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#382

Post by _Codemonkeyism »

gnode wrote: post #374
_Codemonkeyism wrote: post #340

Earlier quoted context omitted.

"There will always be a small handful of engineers that can push the button to move code into PROD" I have very different experiences from a SEC regulated company. With SOX there are controls to prevent such a thing to happen. If this is a SOX breakage, Tesla is in deep trouble with the SEC.

Can you explain how Sarbanes-Oxley applies to the situation of an employee sabotaging a production line? Specifically how it would inherently imply wrongdoing on Tesla's part.

Part of Sarbanes-Oxley is to make sure IT systems are not manipulated. This includes regulating access to systems and controlling software development. Mainly this means, people who write code can't push code into production systems on their own.

One person writes a requirement, this needs to be OKed by another person, then a third person writes this code and it's pushed to production. Controls are setup - e.g. checking JIRA tickets in git logs - that no code without proper authorization (corrrect JIRA status) is pushed and deployed.

People need to be able to trace every code change to the requirement and the OK.

In the core this only applies to systems that are in one way or the other relevant to financial data (like ordering), but auditors usually want to be better safe than sorry. But Tesla might have a SOX-IT and non-SOX-IT.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#383

Post by _Codemonkeyism »

CaptainZapp wrote: post #381
gnode wrote: post #374

Earlier quoted context omitted.

Can you explain how Sarbanes-Oxley applies to the situation of an employee sabotaging a production line? Specifically how it would inherently imply wrongdoing on Tesla's part.

In the financial industry part of SOX is segregation of duty. As a developer I'm not allowed to have write access to any production system, except in an emergency via a break-glass mechanism, which is audited to the hilt and back. It also means we're not allowed to deploy software to production systems. This has to happen via a specific chain development > regression / user acceptance testing > production. All those…

Not only banks, every company listed in the US. It was the same where I've worked, and it wasn't a bank (Enron was no bank either).

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#384

Post by tomp »

avs733 wrote: post #188

I'm sorry but Musk's email is ridiculous and unprofessional, and the defenses here are extremely concerning. By ArsTechnica's count this is the 5th fire in the plant. 5th. That's insane. Report after report is that things continue to not go well and the email sent out reeks of paranoia and combined with other comments made recently are quite clearly dishonest. They aren't failing at the details of ramping a manufactu…

> He said it was about promotion

The saboteur said it was about promotion. Why would he lie?! /s

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#385

Post by alanfranzoni »

alxlaz wrote: post #339
cottsak wrote: post #84

Earlier quoted context omitted.

This is a very naive comment. There will always be a small handful of engineers that can push the button to move code into PROD or even change code in PROD live. Ideally, with mature controls, the people in this list is short. But to jump to the conclusion that Tesla doesn't use good practises is very short sighted. Who's to say that external parties didn't target this person specifically because of their role/influe…

Obviously, not all details are available, but the wording in the email suggests that the parent comment is anything but naive: > This included making direct code changes to the Tesla Manufacturing Operating System under false usernames and exporting large amounts of highly sensitive Tesla data to unknown third parties. This sounds like something out of the 1990s, that dark and romantic era of version control when we…

> 1. Why can anyone, regardless of credentials, push mission-critical code without review (or, alternatively, if the changes did go through review, why did the review process not catch multiple malicious changes?)

Why do you suppose the unauthorized party was following the company's development practices? Maybe it was from the sysadmin side, somebody who worked on the toolchain used for reviewing and pushing things to production. So he was able to sidestep the normal review process. This can happen, what is important is that such things are discovered.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#386

Post by CaptainZapp »

danpalmer wrote: post #366
flyingswift wrote: post #65

I made this same comment on the other discussion. I find it concerning that one person was able to push malicious code to 'production'. To me, this suggests that Tesla, a company building highly sensitive software, does not employ basic branch policies. How is is it that these changes could have made it through a code review process and get deployed? If a company like Microsoft or Google announced that a disgruntled…

> To me, this suggests that Tesla, a company building highly sensitive software, does not employ basic branch policies. Are they building highly sensitive software? I would suggest that the factory automation systems would be sensitive to downtime more than anything else, and the ability to respond quickly with changes would be very valuable. Merging policies would only slow this down. I would also suggest that there…

> Are they building highly sensitive software?

Yes!

> I would suggest that the factory automation systems would be sensitive to downtime more than anything else

If somebody pulls some shenanigans on the sofware, which is responsible for actually building the cars I'd wager that such software is even more critical than the software in the cars.

Just imagine some twiddling with the break systems being manufactured.

So I don't believe downtime is the major concern. Security of the final product, which is directly derived from such software, is.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#387

Post by alanfranzoni »

CaptainZapp wrote: post #381
gnode wrote: post #374

Earlier quoted context omitted.

Can you explain how Sarbanes-Oxley applies to the situation of an employee sabotaging a production line? Specifically how it would inherently imply wrongdoing on Tesla's part.

In the financial industry part of SOX is segregation of duty. As a developer I'm not allowed to have write access to any production system, except in an emergency via a break-glass mechanism, which is audited to the hilt and back. It also means we're not allowed to deploy software to production systems. This has to happen via a specific chain development > regression / user acceptance testing > production. All those…

> As a developer

What if the employee were a sysadmin-level person that sidestepped the normal process?

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#388

Post by amelius »

danpalmer wrote: post #366
flyingswift wrote: post #65

I made this same comment on the other discussion. I find it concerning that one person was able to push malicious code to 'production'. To me, this suggests that Tesla, a company building highly sensitive software, does not employ basic branch policies. How is is it that these changes could have made it through a code review process and get deployed? If a company like Microsoft or Google announced that a disgruntled…

> To me, this suggests that Tesla, a company building highly sensitive software, does not employ basic branch policies. Are they building highly sensitive software? I would suggest that the factory automation systems would be sensitive to downtime more than anything else, and the ability to respond quickly with changes would be very valuable. Merging policies would only slow this down. I would also suggest that there…

> Are they building highly sensitive software?

A car is a potential murder weapon. So yes. Especially considering it may be remote-controlled, and deployed by the millions.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#389

Post by SmellyGeekBoy »

natch wrote: post #326
twblalock wrote: post #88

Earlier quoted context omitted.

> It's not a bit too far since oil&gas companies and car companies have an established history of doing very evil things. That does not mean they have done every specific evil thing they are accused of, such as sabotaging Tesla's factory.

They haven’t been accused in this case, so let’s not get ahead of ourselves.

Maybe not accused directly but Musk's email seems to heavily imply it.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#390

Post by d0mine »

olivermarks wrote: post #335
FortNightLord wrote: post #262

Earlier quoted context omitted.

Uh it says the guy admitted to it...and gave reasons...a confession is more than enough evidence.

People at Tesla are working incredibly hard, they just had a FTE mass layoff and are relying heavily on 'associates'. This reminds me of the darker days at Zappos with their holocracy business model and all powerful vanity CEO. It's not hard to imagine the disgruntled moving fast and breaking things in frustration

s/holocracy/holAcracy/
Post reply on HN