Live data from Hacker News

Elon Musk emails employees about 'extensive and damaging sabotage' by employee

cnbc.com

451–460 of 627 posts

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#451

I made this same comment on the other discussion. I find it concerning that one person was able to push malicious code to 'production'. To me, this suggests that Tesla, a company building highly sensitive software, does not employ basic branch policies. How is is it that these changes could have made it through a code review process and get deployed? If a company like Microsoft or Google announced that a disgruntled…

That's all good and nice but in most cases, this is a matter of policy. If someone actively tries to circumvent the policy or the process, odds are that most software shops would fall victim to the same thing. Especially in the case of internal software. Even if you have a system that attempts to enforce the process, odds are that the system isn't without flaw and is not too hard to circumvent. Most businesses who's…

> If someone actively tries to circumvent the policy or the process, odds are that most software shops would fall victim to the same thing.

Too often the focus is entirely on outside attacks, with little consideration given to insider attacks. Previous job was at a cyber security firm. We'd routinely come under attack from criminal and, we believed, occasional nation state attacks as our researchers attributed a few "Axis-of-Evil" nation-state operations.

Then you'd come back from lunch and find the mantrap doors propped open, or someone left a workstation unlocked with root access to something important, or random guests just wandering around. It's a miracle we never were compromised by a disgruntled employee (of which there were many).

New job, new industry, same behavior. Folks leaving "secure" doors open, workstations unlocked with root access, and all that jazz. They're happy to cite a vague SEC regulation that may have applied at their last job, but doesn't apply to ours and even then, it's not like an attacker would ever give a flying fuck about what the SEC thinks.

I'm starting to wonder if this isn't all a comically bad dream.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#452
I can't really put my finger on why, but this looks like a quite stupid move. Usually in such situations this person gets fired and sued and then the top management tries to put silence over that topic. An email to the whole staff is the opposite of that and makes me wonder if Elon tries to cover some own missteps. Any hints in that direction?

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#453
post #376

Earlier quoted context omitted.

The chain we had in ${BIGCORP}: Programmers: read-write to repository Staging team: read-only on repository, read-write to test servers and staging zone Deployment team: read-only on repository and staging zone, read-write to production It wouldn't prevent malicious code going out but at least would require a chain of cooperation between employees, which would be harder to achieve.

So the deployment team could write and push malicious code to production?

They pretty much always can. Sure sure you can imagine some perfect system which would mitigate it but no one - definitely not your bank - is doing that.

It very much sounds like thats the case here - production code was edited, and subsequent auditing has found what should've been deployed and what is deployed differs.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#454

> Looking forward to having a great week with you as we charge up the super exciting ramp to 5000 Model 3 cars per week! This was actually the most surprising part to me -- it appears that they believe that they will actually make the 5K/week goal by the end of this quarter. As to the allegations... the email doesn't say much, but it sounds like a disgruntled employee grabbing data to me, and perhaps modifying some O…

I don't know if you've worked in a large corporation, but during my time in one we would constantly get senior management talking about how we were doing really well and going to do great things and we were making history.... whilst goals and deadlines were missed constantly and everyone knew we had no chance of meeting them. We had a product delayed by 4 years and when it finally shipped the head of software sent out an email congratulating everyone on their hardwork on that final push to get it out before the EOY deadline (the original deadline was EOY 4 years earlier). It's just what senior management does, don't read into it.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#455
post #177

> In 2007, a comprehensive study of markets around the world found that ones where short selling was legal and common were more efficient than ones where it was not. And a 2012 study concluded simply, “Stock prices are more accurate when short sellers are more active.” https://www.newyorker.com/magazine/2015/03/23/in-praise-of-s... Short selling is a _good thing_. Blaming the short sellers is the practices of compani…

I'd argue neither side is "good". It's good that we have both opposing sides. Both can be exploited for bad, but each time it happens there is an opportunity for the other side to gain something. And a smart investor certainly keeps open eyes for both directions.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#456

Earlier quoted context omitted.

> As a developer What if the employee were a sysadmin-level person that sidestepped the normal process?

The you say in your compliance policy that you run regular audits for these kinds of actions and remediate them on a case-by-case basis. E.g. run quarterly reports for changes to prod that didn't go through the regular release pipeline and note down which P0 they corresponded to.

Which would then be roughly why this happened and is coming out now.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#457

I made this same comment on the other discussion. I find it concerning that one person was able to push malicious code to 'production'. To me, this suggests that Tesla, a company building highly sensitive software, does not employ basic branch policies. How is is it that these changes could have made it through a code review process and get deployed? If a company like Microsoft or Google announced that a disgruntled…

That's all good and nice but in most cases, this is a matter of policy. If someone actively tries to circumvent the policy or the process, odds are that most software shops would fall victim to the same thing. Especially in the case of internal software. Even if you have a system that attempts to enforce the process, odds are that the system isn't without flaw and is not too hard to circumvent. Most businesses who's…

Self driving cars are both defense and security related.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#458

Earlier quoted context omitted.

How the saboteur knew the passwords of all those users before? If it was a totally new user, shouldn't the system have a list of approved users and a strict protocol to validate a new user?

I'm baffled at your lack of imagination, but: keyloggers, unlocked terminals, API token sniffing from cookies, reactivated old accounts, changing the reviewing account id in the database, …

"Hey Ed my password expired, can you use yours to get this done just now..."

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#459

Earlier quoted context omitted.

I think you misunderstand. Even with code review policies, there is still a short list of people who can push to production without going through code review. Not from a policy standpoint but from a security and access perspective.

The chain we had in ${BIGCORP}: Programmers: read-write to repository Staging team: read-only on repository, read-write to test servers and staging zone Deployment team: read-only on repository and staging zone, read-write to production It wouldn't prevent malicious code going out but at least would require a chain of cooperation between employees, which would be harder to achieve.

It would not require any chain of cooperation I highly doubt staging team would catch some coefficient used for particular industrial robot being off by 0.1% in some PR.

Re: Elon Musk emails employees about 'extensive and damaging sabotage' by employee

#460
post #360

Earlier quoted context omitted.

Then again, Musk knows that this email will instantly get leaked. This is PR as much as anything.

What if that communication was an attempt at discovering the leak source? Maybe there is an uniquely identifiable token or wording in the original mail(s) (per division, or team). Did any one counted the spaces or looked for invisible Unicode characters?

[deleted]
Post reply on HN