Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A deeper dive into our May 2019 security incident

stackoverflow.blog

Post reply on HN
Print view View original

51–60 of 60 posts

Re: A deeper dive into our May 2019 security incident

#51

Post by YesThatTom2 »

akersten wrote: post #27

Interesting that most of the mitigations are "move resource behind firewall." Kind of an indictment of the whole BeyondCorp idea - unless we really trust our 2FA to never have any access bypass issues like the initial access to the dev environment here. Speaking of that, I didn't see "fix bug allowing unauthenticated access to dev environment" listed as one of the mitigations, but maybe I glossed over it.

You got it backwards. BeyondCorp-like systems would have prevented that.

(FYI: I'm very familiar with BeyondCorp, as I was on an adjacent team when it was invented. Now I am an SRE at Stack Overflow when the incident happened.)

Re: A deeper dive into our May 2019 security incident

#52

Post by TacticalCoder »

pc86 wrote: post #35
TacticalCoder wrote: post #13

> However, there is a route on dev that can show email content to CMs and they use this to obtain the magic link used to reset credentials So many sites do this: allowing major changes to be effective immediately (like resetting credentials/password) by simply opening a "magic link" sent by email. I think that this "immediately" is a major security antipattern. I prefer it when such changes have a "cooldown" period o…

The next time you forget a password and need to reset it, how likely are you to be willing to wait three days?

> The next time you forget a password and need to reset it, how likely are you to be willing to wait three days?

10 minutes like others suggested is way too short that said: this wouldn't catch attacks happening at night.

But to answer your question: it really depends what it is that you are protecting. For most sites I use by very far I don't see how 72 hours without access would be that problematic. Not logged in to StackOverflow for 3 days? Not a problem. Not logged in to HN for 3 days? Not a problem. Not logged in to Twitter for 3 days? I can live with that. Etc.

The question is: how much convenience are you willing to trade for security?

Re: A deeper dive into our May 2019 security incident

#53

Post by TacticalCoder »

Sebb767 wrote: post #22
TacticalCoder wrote: post #13

> However, there is a route on dev that can show email content to CMs and they use this to obtain the magic link used to reset credentials So many sites do this: allowing major changes to be effective immediately (like resetting credentials/password) by simply opening a "magic link" sent by email. I think that this "immediately" is a major security antipattern. I prefer it when such changes have a "cooldown" period o…

While I partially agree, letting an user deny the password change with the old password is pretty horrible in case the password was leaked. And if your dev forgot the password after the holidays, you're looking at 3 additional free days for him. I see your point, but a timeout is a suboptimal solution.

> ... letting an user deny the password change with the old password is pretty horrible in case the password was leaked.

That's a different threat: an attacker knowing your password. Well... How do websites that allow instant credentials resets by email typically deal with a password reset asked by someone who knows the current password? Instant change too ("enter your current password / enter your new password twice"). And the good guy is locked out of his own account. I don't see how it's worse than that.

Re: A deeper dive into our May 2019 security incident

#54

Post by jlericson »

Shog9 wrote: post #16
dsr_ wrote: post #10

"However, there is a route on dev that can show email content to CMs and they use this to obtain the magic link used to reset credentials." Zawinski's Law: "Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can."

Gonna clarify here, because that description is a bit misleading: this wasn't a route that allowed viewing sent emails, it was a route that allowed viewing what would be sent if a password reset was requested. The story behind that route might be interesting... See, originally Stack Overflow didn't have passwords - all logins were done via OpenID, so any credential management you'd need to do was done through your pr…

Yeah. I'm almost certain I was the one who got sick of having no idea what a user would see when they opened their email that I asked for _some way_ of see it. (Otherwise it was this strange dance of "Request a password recovery and tell me what it says.") I don't recall if I ever considered that it might be a _massive security hole_ if anyone got a hold of it. In retrospect . . .

(I overlapped with Shog at Stack Overflow.)

Re: A deeper dive into our May 2019 security incident

#55

Post by saagarjha »

pc86 wrote: post #35
TacticalCoder wrote: post #13

> However, there is a route on dev that can show email content to CMs and they use this to obtain the magic link used to reset credentials So many sites do this: allowing major changes to be effective immediately (like resetting credentials/password) by simply opening a "magic link" sent by email. I think that this "immediately" is a major security antipattern. I prefer it when such changes have a "cooldown" period o…

The next time you forget a password and need to reset it, how likely are you to be willing to wait three days?

Ten days is better than forever, FWIW, which is something that many websites do.

Re: A deeper dive into our May 2019 security incident

#56

Post by kmontrose »

jlericson wrote: post #54
Shog9 wrote: post #16

Earlier quoted context omitted.

Gonna clarify here, because that description is a bit misleading: this wasn't a route that allowed viewing sent emails, it was a route that allowed viewing what would be sent if a password reset was requested. The story behind that route might be interesting... See, originally Stack Overflow didn't have passwords - all logins were done via OpenID, so any credential management you'd need to do was done through your pr…

Yeah. I'm almost certain I was the one who got sick of having no idea what a user would see when they opened their email that I asked for _some way_ of see it. (Otherwise it was this strange dance of "Request a password recovery and tell me what it says.") I don't recall if I ever considered that it might be a _massive security hole_ if anyone got a hold of it. In retrospect . . . (I overlapped with Shog at Stack Ove…

Only became a problem in combination with other missteps (or constraints, like dev needing to be routable).

Which is what makes this kind of stuff so insidious.

Re: A deeper dive into our May 2019 security incident

#57

Post by Hjfrf »

LeifCarrotson wrote: post #41
curiousllama wrote: post #21

Earlier quoted context omitted.

Yea, 10 minutes & a text message would suffice, IMO...

I think the most important part would be to give someone time to vet that it's legitimate. Stack Exchange has on the order of 100 developers, it wouldn't be hard to CC account creation or password reset notices to the manager of a new hire, and in that case, 10 minutes would often be enough to say "Uh, I haven't hired anyone named Curious Llama, who are they and why are they requesting developer access to an obsolete…

That might not have helped in this case, if the email was sent from the same account.

The hacker had access to messages from the sender side without access to the account being reset.

Re: A deeper dive into our May 2019 security incident

#58

Post by glenstein »

lbriner wrote: post #23
lima wrote: post #12

Earlier quoted context omitted.

If that's the case, why would they elevate privileges on the main SO site and draw attention to their successful intrusion?

Because they are working blind. They are trying to find something that they don't necessarily know exists. They also don't know what tripwires exist and after trawling around for so long, they might even have assumed that SE didn't have any monitoring systems.

Of the two responses here, yours strikes me as the more plausible, non-cartoonish one. I think it's good to come to these things with an understanding of how behaviors can be happenstance and come from an attacker negotiating with limited information or their own limited understanding.

Re: A deeper dive into our May 2019 security incident

#59

Post by wolfgang42 »

glenstein wrote: post #58
lbriner wrote: post #23

Earlier quoted context omitted.

Because they are working blind. They are trying to find something that they don't necessarily know exists. They also don't know what tripwires exist and after trawling around for so long, they might even have assumed that SE didn't have any monitoring systems.

Of the two responses here, yours strikes me as the more plausible, non-cartoonish one. I think it's good to come to these things with an understanding of how behaviors can be happenstance and come from an attacker negotiating with limited information or their own limited understanding.

I’m doing my best not to be insulted by your description of my conjecture as “cartoonish.” Maybe one of us has misunderstood the other, but it seems to me that I’ve also proposed a mechanism that involves the attacker having limited understanding.

From the blog post, it sounds like the bit where they were working in the dark was getting things to run on a production database, but the SQL command to give them moderator privileges was prepared ahead of time (which makes sense, they had a local site set up and could prepare and test that part at their leisure). It seems very unlikely to me that they would have spent so much effort on getting that particular SQL to run for no reason, so the moderator privilege in particular probably had some appeal to them from a technical perspective. The main permission I can think of that you get as a mod is the ability to edit posts. It seems unlikely that an attack this sophisticated would be just for vandalism, so I consider the profit motive instead. “For sale: Access to Stack Exchange internal infrastructure. As proof, ask me to edit any post on Stack Overflow.”

What tripped them up was not a technical tripwire, but rather how intimate the SE community is with their moderators. This wouldn’t be obvious from the codebase; only if you’d spent some time on the meta sites would you be aware of the culture surrounding SE mods. On, say, Facebook, getting a global moderator bit isn’t something that a big chunk of the user base would have name recognition for. My stereotype is that someone who’s trying to break in to infrastructure (and not doing responsible disclosure) probably isn’t also volunteering in the review queues, and so wouldn’t realize what a risk this would be.

Of course this is all conjecture and we’ll never know for certain without tracking down the attacker and asking them, but I’d like to think I’ve constructed a reasonable scenario that takes all of the known facts and a probable motive and comes up with a plausible explanation for what was (in hindsight, at least) a fairly significant blunder.

Re: A deeper dive into our May 2019 security incident

#60

Post by eitland »

TrickyRick wrote: post #48
Eduard wrote: post #34

> Fortunately, we have a database containing a log of all traffic to our publicly accessible properties https://stackoverflow.com/legal/privacy-policy GDPR anyone?

> When you visit the Network or use our Apps, Stack Overflow automatically receives and records information from your browser or mobile device, such as your Internet Protocol (IP) address or unique device identifier. Cookies and data about which pages you visit on our Network allow us to operate and optimize the Products and Services we provide to you. This information is stored in secure logs and is collected automa…

Collecting data might well be acceptable with the GDPR.

However what makes it legal isn't if it is written in the TOS or in a cookie banner.

AFAIK what matters is either:

- if you have a specific, valid (according to the GDPR) reason,

- or if you have the users free and informed consent.

... and yes, I think a number of the things I still see on the web is not OK:

- dark pattern where if you click manage settings everything is opted out, but there's a big green "Accept everything" and a small bland "Confirm my choices"? Doesn't fly because the rule that it should be equally easy to opt out.

- Cookie banners with no real opt out? No way.

- Cookie banners where you have to deselect 927 "partners"? Also no way.

The only ones that seems legal are those who either uses a pure minimum of cookies for preserving state and those allow one to opt out directly but inform you that ads might become less relevant.

Post reply on HN