Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elon Musk emails employees about 'extensive and damaging sabotage' by employee

cnbc.com

Post reply on HN
Print view View original

391–400 of 627 posts

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#391

Post by finnthehuman »

taneq wrote: post #342
flyingswift wrote: post #65

I made this same comment on the other discussion. I find it concerning that one person was able to push malicious code to 'production'. To me, this suggests that Tesla, a company building highly sensitive software, does not employ basic branch policies. How is is it that these changes could have made it through a code review process and get deployed? If a company like Microsoft or Google announced that a disgruntled…

> I find it concerning that one person was able to push malicious code to 'production'. Production line software is almost certainly handled separately from the software that runs their vehicles, and isn't "production" in the usual web sense of being customer facing. This sounds more like someone messed with their factory automation setup.

Are you trying to us that less stringent controls on manufacturing software is in any way acceptable?

Manufacturing process (including mfg software tools) are a huge potential source of product failure. I don't know about automotive QMS specifically, but I work in embedded software for safety critical systems. If an unreleased procedure or unreleased software is used to build the hardware, or unreleased software is run on the hardware, it's not even suitable for QA (let alone going to the field).

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#392

Post by daemin »

alxlaz wrote: post #339
cottsak wrote: post #84

Earlier quoted context omitted.

This is a very naive comment. There will always be a small handful of engineers that can push the button to move code into PROD or even change code in PROD live. Ideally, with mature controls, the people in this list is short. But to jump to the conclusion that Tesla doesn't use good practises is very short sighted. Who's to say that external parties didn't target this person specifically because of their role/influe…

Obviously, not all details are available, but the wording in the email suggests that the parent comment is anything but naive: > This included making direct code changes to the Tesla Manufacturing Operating System under false usernames and exporting large amounts of highly sensitive Tesla data to unknown third parties. This sounds like something out of the 1990s, that dark and romantic era of version control when we…

Some manager asks IT multiple times over the course of a few weeks to create an account for a contractor, then give them permissions to access production type machines.

Or a contractor that was fired had their credentials appropriated by this manager, perhaps by that manager removing them from a "delete these accounts" list.

Those are a couple of mundane ways of getting a false username to a production machine. This is even easier when there is a lot of flux at the company with many people coming and going, a lot of account management happening etc.

It could have been that the accounts were local to specific machines and not managed by the company as a whole.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#393

Post by falsedan »

greglindahl wrote: post #352
masklinn wrote: post #350

Earlier quoted context omitted.

> Even with code review policies, there is still a short list of people who can push to production without going through code review. That's completely unnecessary and should not be the case. If you need something pushed quickly, you can get a colleague with review bit and get them to ack for "urgency" reasons after a quick lookover.

I'm glad to see evidence that security theater fantasies are alive and well!

To be fair, this is more likely to be compliance rather than security.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#394

Post by SuoDuanDao »

whatshisface wrote: post #61

Is it possible to make money by convincing everyone to short your company and then pushing the rug in under them by making earnings projections? Musk's recent announcements have been suspiciously suspicious.

probably... 'short squeezes' do cause price spikes, and if you control the news cycle about a company, I'm sure you could benefit from the extra volatility.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#395

Post by pvaldes »

gnode wrote: post #377
EGreg wrote: post #125

Earlier quoted context omitted.

Why is it naive to expect there not to be a single point of failure like that? Code reviews are a thing.

There may have been. The article speaks of the employee using "false usernames". Code review may stop some foolish person pushing broken code, but it's not going to prevent a determined saboteur who's masquerading as other authorised users.

How the saboteur knew the passwords of all those users before?

If it was a totally new user, shouldn't the system have a list of approved users and a strict protocol to validate a new user?

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#396

Post by falsedan »

alanfranzoni wrote: post #387
CaptainZapp wrote: post #381

Earlier quoted context omitted.

In the financial industry part of SOX is segregation of duty. As a developer I'm not allowed to have write access to any production system, except in an emergency via a break-glass mechanism, which is audited to the hilt and back. It also means we're not allowed to deploy software to production systems. This has to happen via a specific chain development > regression / user acceptance testing > production. All those…

> As a developer What if the employee were a sysadmin-level person that sidestepped the normal process?

The you say in your compliance policy that you run regular audits for these kinds of actions and remediate them on a case-by-case basis.

E.g. run quarterly reports for changes to prod that didn't go through the regular release pipeline and note down which P0 they corresponded to.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#397

Post by sneak »

chillee wrote: post #215
davidgay wrote: post #191

Earlier quoted context omitted.

Then the system that allowed them to do so is badly designed.

At Facebook, you could alter code even after somebody had given the OK for code review. I know some people specifically kept some small commits open after being approved, just so they could quickly make changes without needing approval if they ever needed to.

“Dear respected members of Congress, ...”

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#398

Post by Operyl »

alxlaz wrote: post #339
cottsak wrote: post #84

Earlier quoted context omitted.

This is a very naive comment. There will always be a small handful of engineers that can push the button to move code into PROD or even change code in PROD live. Ideally, with mature controls, the people in this list is short. But to jump to the conclusion that Tesla doesn't use good practises is very short sighted. Who's to say that external parties didn't target this person specifically because of their role/influe…

Obviously, not all details are available, but the wording in the email suggests that the parent comment is anything but naive: > This included making direct code changes to the Tesla Manufacturing Operating System under false usernames and exporting large amounts of highly sensitive Tesla data to unknown third parties. This sounds like something out of the 1990s, that dark and romantic era of version control when we…

It sounds like they might use git and are not commit signing or something of the sorts.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#399

Post by qaq »

alxlaz wrote: post #339
cottsak wrote: post #84

Earlier quoted context omitted.

This is a very naive comment. There will always be a small handful of engineers that can push the button to move code into PROD or even change code in PROD live. Ideally, with mature controls, the people in this list is short. But to jump to the conclusion that Tesla doesn't use good practises is very short sighted. Who's to say that external parties didn't target this person specifically because of their role/influe…

Obviously, not all details are available, but the wording in the email suggests that the parent comment is anything but naive: > This included making direct code changes to the Tesla Manufacturing Operating System under false usernames and exporting large amounts of highly sensitive Tesla data to unknown third parties. This sounds like something out of the 1990s, that dark and romantic era of version control when we…

1. Is possible in pretty much any environment especially one as complicated as manuf. automation.

2. If you already have privileged accounts you can escalate in pretty much any environment. And they obviously did figure it out.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#400

Post by thecatspaw »

JoshCole wrote: post #241
Maybestring wrote: post #156

Earlier quoted context omitted.

The boogyman isn't the sabeteur, it's the mysterious cabal of oil barons and short sellers that put him up to it.

I think the boogeyman is whoever the guy was exporting data too, since they found data exportation features. Because why have data exportation unless you are doing it too somewhere.

I read "exporting data" as "took home a sql dump and now plans to sell email and related data on the darknet". Standard small blackhat stuff, not industrial espionage, but I doubt tesla will publicly say what happened in this case
Post reply on HN