Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A deeper dive into our May 2019 security incident

stackoverflow.blog

Post reply on HN
Print view View original

21–30 of 60 posts

Re: A deeper dive into our May 2019 security incident

#21

Post by curiousllama »

NieDzejkob wrote: post #18
TacticalCoder wrote: post #13

> However, there is a route on dev that can show email content to CMs and they use this to obtain the magic link used to reset credentials So many sites do this: allowing major changes to be effective immediately (like resetting credentials/password) by simply opening a "magic link" sent by email. I think that this "immediately" is a major security antipattern. I prefer it when such changes have a "cooldown" period o…

I don't think it's a good idea to make a password reset take 3 entire days. In this case, I'd say the costs outweigh the benefits.

Yea, 10 minutes & a text message would suffice, IMO...

Re: A deeper dive into our May 2019 security incident

#22

Post by Sebb767 »

TacticalCoder wrote: post #13

> However, there is a route on dev that can show email content to CMs and they use this to obtain the magic link used to reset credentials So many sites do this: allowing major changes to be effective immediately (like resetting credentials/password) by simply opening a "magic link" sent by email. I think that this "immediately" is a major security antipattern. I prefer it when such changes have a "cooldown" period o…

While I partially agree, letting an user deny the password change with the old password is pretty horrible in case the password was leaked. And if your dev forgot the password after the holidays, you're looking at 3 additional free days for him.

I see your point, but a timeout is a suboptimal solution.

Re: A deeper dive into our May 2019 security incident

#23

Post by lbriner »

lima wrote: post #12
ballenf wrote: post #8

Earlier quoted context omitted.

Given the focus on enterprise systems and teams, really looks like it was a Solarwinds type (but lower sophistication) attack where SO wasn't really the target. The targets were users of SO Enterprise or teams products.

If that's the case, why would they elevate privileges on the main SO site and draw attention to their successful intrusion?

Because they are working blind.

They are trying to find something that they don't necessarily know exists. They also don't know what tripwires exist and after trawling around for so long, they might even have assumed that SE didn't have any monitoring systems.

Re: A deeper dive into our May 2019 security incident

#24

Post by weinzierl »

Shog9 wrote: post #16
dsr_ wrote: post #10

"However, there is a route on dev that can show email content to CMs and they use this to obtain the magic link used to reset credentials." Zawinski's Law: "Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can."

Gonna clarify here, because that description is a bit misleading: this wasn't a route that allowed viewing sent emails, it was a route that allowed viewing what would be sent if a password reset was requested. The story behind that route might be interesting... See, originally Stack Overflow didn't have passwords - all logins were done via OpenID, so any credential management you'd need to do was done through your pr…

Excellent writeup and it shows that Stack Overflow's account management came a long way.

Still there is room for improvement. What confused me a lot recently was, that the reset link sent to a certain email is not necessarily for the login associated with that email.

I tried to to login at Stack Overflow after a long time. Entered my current mail and pw. Did not work, clicked pw recovery, received mail, reset pw, got logged in. So far so good.

Logged out, couldn't log in again. After a few password resets I realized that, while the mail was sent to my current address, the reset link actually was for the pw of a login associated with an old email.

At least for me that was not clear from the recovery email. Here is the full text with only email redacted:

> Account Recovery - Stack Overflow

> We received an account recovery request on Stack Overflow for new@example.com.

> If you initiated this request, reset your password here.

> You can use any of the following credentials to log in to your account:

> Email and Password (old@example.com)

> Email and Password (new@example.com)

> Once logged in, you can review existing credentials and add new ones. Simply visit your profile, click on Edit Profile & Settings and My Logins.

To be clear, "reset your password here." is a link and it changes the pw only for old@example.com.

Re: A deeper dive into our May 2019 security incident

#25

Post by wolfgang42 »

lima wrote: post #12
ballenf wrote: post #8

Earlier quoted context omitted.

Given the focus on enterprise systems and teams, really looks like it was a Solarwinds type (but lower sophistication) attack where SO wasn't really the target. The targets were users of SO Enterprise or teams products.

If that's the case, why would they elevate privileges on the main SO site and draw attention to their successful intrusion?

My guess: they were trying to create a public proof that they had access to Stack internals (probably so they could sell it), and weren’t familiar enough with StackExchange administrivia to realize that the users would immediately notice and be suspicious of a new non-elected moderator.

Re: A deeper dive into our May 2019 security incident

#26

Post by Shog9 »

weinzierl wrote: post #24
Shog9 wrote: post #16

Earlier quoted context omitted.

Gonna clarify here, because that description is a bit misleading: this wasn't a route that allowed viewing sent emails, it was a route that allowed viewing what would be sent if a password reset was requested. The story behind that route might be interesting... See, originally Stack Overflow didn't have passwords - all logins were done via OpenID, so any credential management you'd need to do was done through your pr…

Excellent writeup and it shows that Stack Overflow's account management came a long way. Still there is room for improvement. What confused me a lot recently was, that the reset link sent to a certain email is not necessarily for the login associated with that email. I tried to to login at Stack Overflow after a long time. Entered my current mail and pw. Did not work, clicked pw recovery, received mail, reset pw, got…

Yeah, this doesn't surprise me. The login system at SO is unnecessarily complex - by which I mean that the complexity of the interface does not match the complexity of the underlying system, because while both have been redesigned several times they've never been completely redesigned together.

So you end up with weird situations like this, where both the interface and the underlying system support multiple associated email addresses, but not in the same ways or with the same functionality exposed.

It is... A legacy system, with all that that entails.

Re: A deeper dive into our May 2019 security incident

#27

Post by akersten »

Interesting that most of the mitigations are "move resource behind firewall." Kind of an indictment of the whole BeyondCorp idea - unless we really trust our 2FA to never have any access bypass issues like the initial access to the dev environment here. Speaking of that, I didn't see "fix bug allowing unauthenticated access to dev environment" listed as one of the mitigations, but maybe I glossed over it.

Re: A deeper dive into our May 2019 security incident

#28

Post by 120bits »

Thank you SO for being open and listing the best practices. It seems like even few security best practices makes it harder for hackers to get in to your system.

I have database connecting strings and password as ENV variables. But I still don't know what is the best practice. Lets say someone gets access to the server, they can still read the ENV vars, right? It definitely prevents from accidently checking in your code git repo. But still . Does anyone has good recommendation for storing credentials like database passwords in a way secured way.

Re: A deeper dive into our May 2019 security incident

#29

Post by invokestatic »

akersten wrote: post #27

Interesting that most of the mitigations are "move resource behind firewall." Kind of an indictment of the whole BeyondCorp idea - unless we really trust our 2FA to never have any access bypass issues like the initial access to the dev environment here. Speaking of that, I didn't see "fix bug allowing unauthenticated access to dev environment" listed as one of the mitigations, but maybe I glossed over it.

I agree. I’ve employed the BeyondCorp philosophy behind a VPN as an extra measure of security, which is to say that all services are authenticated and encrypted inside the VPN perimeter. As shown in this article, service accounts are a major concern for attacker lateral movement which can’t be effectively protected with just 2FA.

Re: A deeper dive into our May 2019 security incident

#30

Post by cbg0 »

120bits wrote: post #28

Thank you SO for being open and listing the best practices. It seems like even few security best practices makes it harder for hackers to get in to your system. I have database connecting strings and password as ENV variables. But I still don't know what is the best practice. Lets say someone gets access to the server, they can still read the ENV vars, right? It definitely prevents from accidently checking in your co…

I don't think there's a magic way to do this, if your app can connect to the database and someone has access to your app server - they have access to your database as well.
Post reply on HN