Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elon Musk emails employees about 'extensive and damaging sabotage' by employee

cnbc.com

Post reply on HN
Print view View original

441–450 of 627 posts

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#441

Post by alxlaz »

dpcx wrote: post #437
alxlaz wrote: post #421

Earlier quoted context omitted.

> So he was able to sidestep the normal review process. He should not have been able to sidestep the normal review process . That's the problem in the first place. Even if you're from the sysadmin side. It should not be possible to do it. You may think that looks exaggerated but I've worked in two places where we implemented such a process, both of them far more boring than Tesla and, I suspect, far less money to bur…

I'm interested to know how you plan to keep someone with `wheel` access from doing anything on a server that they maintain. No, seriously.

You don't keep someone with wheel access from doing anything on the server. You:

1. Sign every review 2. Use the review signatures + the manufacturer's keys to sign reproducible builds of the production image (i.e. you cryptographically certify that "this image is authorized, and it includes this list of commits, that has gone through these reviews"). 3. Use a secure boot scheme of your choice to ensure that only signed images can be installed on production servers 4. Keep anyone with 'wheel' access away from the image signing keys, and anyone who can generate images away from 'wheel' access.

This way, you make sure that no one who has 'wheel' access can install a sabotaged image, that any image that can be installed has gone through an auditable trail of reviews, and reduce the attack surface that a malicious developer has control over to stuff that requires root access (which is still a lot of surface, but is harder to sneak past a review).

Root access to production servers does not need to mean that you can install arbitrary code on them, and with the right systems engineering, you can ensure that it does not trivially result in arbitrary code being run on production equipment.

Edit: this is all in the context of "questions that Tesla's answer raises". For all I know, the answer might be that they hired some brilliant genius who figured out how to sneak by whatever secure boot scheme they're using. The point is -- the post that sparked all this is not naive. This is real stuff. Companies that are concerned about it can ensure that unauthorized commits are so difficult to get into production that a disgruntled employee would rather just quit than go through with it.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#442

Post by RVuRnvbM2e »

jerry40 wrote: post #334

> As you know, there are a long list of organizations that want Tesla to die. Is it correct to use 'are' here instead of 'is'?

It depends on which part of the sentence you would like to emphasise. In this case, the _organisations_ want Tesla to die, not the _list_. Hence the plural form.

Not all collective nouns work this way, but in this case either is grammatically correct.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#443

Post by yjftsjthsd-h »

stephengillie wrote: post #415
Fricken wrote: post #231

Earlier quoted context omitted.

There isn't a revolution underway. Everyone switching to an electric personal automoblie won't save us from anything. We'll just be destroying the planet at a marginally slower rate. You don't get to have your cake and eat it too.

> You don't get to have your cake and eat it too. The way to do this is to start with 2 cakes. But where is Earth2?

I seem to recall Musk trying to get to Mars...

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#444

Post by dagw »

dpcx wrote: post #437
alxlaz wrote: post #421

Earlier quoted context omitted.

> So he was able to sidestep the normal review process. He should not have been able to sidestep the normal review process . That's the problem in the first place. Even if you're from the sysadmin side. It should not be possible to do it. You may think that looks exaggerated but I've worked in two places where we implemented such a process, both of them far more boring than Tesla and, I suspect, far less money to bur…

I'm interested to know how you plan to keep someone with `wheel` access from doing anything on a server that they maintain. No, seriously.

Only allow the server to run signed code and make sure that no one with wheel access has access to the signing key.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#445

Post by danpalmer »

amelius wrote: post #388
danpalmer wrote: post #366

Earlier quoted context omitted.

> To me, this suggests that Tesla, a company building highly sensitive software, does not employ basic branch policies. Are they building highly sensitive software? I would suggest that the factory automation systems would be sensitive to downtime more than anything else, and the ability to respond quickly with changes would be very valuable. Merging policies would only slow this down. I would also suggest that there…

> Are they building highly sensitive software? A car is a potential murder weapon. So yes. Especially considering it may be remote-controlled, and deployed by the millions.

I don't think you read my comment. The autonomous driving software is sensitive, much of Tesla's software will not be.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#446

Post by danpalmer »

CaptainZapp wrote: post #386
danpalmer wrote: post #366

Earlier quoted context omitted.

> To me, this suggests that Tesla, a company building highly sensitive software, does not employ basic branch policies. Are they building highly sensitive software? I would suggest that the factory automation systems would be sensitive to downtime more than anything else, and the ability to respond quickly with changes would be very valuable. Merging policies would only slow this down. I would also suggest that there…

> Are they building highly sensitive software? Yes! > I would suggest that the factory automation systems would be sensitive to downtime more than anything else If somebody pulls some shenanigans on the sofware, which is responsible for actually building the cars I'd wager that such software is even more critical than the software in the cars. Just imagine some twiddling with the break systems being manufactured. So…

Sure, changing factory automation in a way that might affect the quality of the finished product would be sensitive. I don't know how much of an opportunity there is to do that – I'd expect it's much more "it works or it doesn't" with factory automation, but I honestly don't know.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#447

Post by skj »

chillee wrote: post #215
davidgay wrote: post #191

Earlier quoted context omitted.

Then the system that allowed them to do so is badly designed.

At Facebook, you could alter code even after somebody had given the OK for code review. I know some people specifically kept some small commits open after being approved, just so they could quickly make changes without needing approval if they ever needed to.

At Google you can mark something as TBR (to be reviewed), as long as you're one of the designated owners for that code.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#448

Post by xnc333 »

SmellyGeekBoy wrote: post #402
Fricken wrote: post #225

Earlier quoted context omitted.

The poison of the internet thrives on 140 characters or less, it facilitates lies wonderfully. The truth needs more words. Twitter is Musk's preferred medium. Trump's too.

It's the Dalai Lama's preferred medium too. I don't really see your point.

With respect to the Dalai Lama Hitchens might see the point.

Fact is that on Twitter there are 90% lying self promoters and 10% somewhat honest self promoters.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#449

Post by givinguflac »

rjdagost wrote: post #438
nordsieck wrote: post #427

Earlier quoted context omitted.

>> Production line software is almost certainly handled separately from the software that runs their vehicles, and isn't "production" in the usual web sense of being customer facing. > Are you trying to us that less stringent controls on manufacturing software is in any way acceptable? Yes. It is simply uneconomical to develop all software in a no-bugs-allowed NASA style process. One of the classes of software that c…

No one is saying "no bugs" NASA level code is required. Basic SQA must however be conducted- factory automation software is not something where you can settle for low quality as you seem to imply. Something tells me you've never worked in factory / production environments. There are critical safety requirements that must be checked, especially in environments where machinery is operating in proximity to workers. Not…

Chill out. Every system that relies on human input can be fooled and broken. How do we know this employee wasn’t the one responsible for loading new code into the machines once validated? Anyone looking to spend years infiltrating a place for a goal will do whatever it takes to circumvent security measures.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#450

Post by erikb »

flyingswift wrote: post #65

I made this same comment on the other discussion. I find it concerning that one person was able to push malicious code to 'production'. To me, this suggests that Tesla, a company building highly sensitive software, does not employ basic branch policies. How is is it that these changes could have made it through a code review process and get deployed? If a company like Microsoft or Google announced that a disgruntled…

You would be surprised how many companies would not catch something like that.
Post reply on HN