Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linode Security Advisory

blog.linode.com

Post reply on HN
Print view View original

61–70 of 119 posts

Re: Linode Security Advisory

#61

Post by jonursenbach »

apandhi wrote: post #29
TheSwordsman wrote: post #23

Hey There, I'm a PagerDuty employee and am the same individual who made this post on the last HN thread: * https://news.ycombinator.com/item?id=10845985 Unfortunately, there are some facts in Linode's post that are not correct. >On July 9 a customer notified us of unauthorized access into their Linode account. The customer learned that an intruder had obtained access to their account after receiving an email notifica…

Linode dropping the ball with PagerDuty is huge. I can't even begin to imagine how much money your customers could lose if PagerDuty infrastructure went down.

Which makes me wonder why they're even with Linode.

Re: Linode Security Advisory

#62

Post by TheSwordsman »

jonursenbach wrote: post #61
apandhi wrote: post #29

Earlier quoted context omitted.

Linode dropping the ball with PagerDuty is huge. I can't even begin to imagine how much money your customers could lose if PagerDuty infrastructure went down.

Which makes me wonder why they're even with Linode.

We aren't. Did a full migration in August, 2015.

Re: Linode Security Advisory

#63

Post by ploxiln »

nsgf wrote: post #31
forgotAgain wrote: post #22

I've been a happy Linode customer for a long, long time. With that I have to say the fact that they released this late on a Friday afternoon leaves a bad taste in my mouth. That in turn leaves me with doubts about the veracity of their statements. I'll probably be looking at alternatives now.

Me too (for about 5 years), but this was the last straw. Seriously. Recently moved to a dedicated kimsufi (worst support but great VFM hardware-wise).

I thought this kimsufi thing sounded interesting so I looked it up, and ... wow the processors in the systems they offer are old. Where do they even get these things? Must be leftover from OVH?

  Xeon E5504 - 45nm process, launched 2009
  Core i5-2300 - 32nm process, launched 2011, discontinued 2012
And the Core i5 does not support ecc memory, of course, so what kind of servers has it been sitting in for 4 years?

Re: Linode Security Advisory

#64

Post by ethomson »

TheSwordsman wrote: post #23

Hey There, I'm a PagerDuty employee and am the same individual who made this post on the last HN thread: * https://news.ycombinator.com/item?id=10845985 Unfortunately, there are some facts in Linode's post that are not correct. >On July 9 a customer notified us of unauthorized access into their Linode account. The customer learned that an intruder had obtained access to their account after receiving an email notifica…

This is very helpful information. Can you say if you've moved to a different provider or if you're now racking your own machines? And if you do have a new provider, can you say who you are and how you evaluated them?

I have been doing some research in my (limited) spare time to try to find a new provider, but I still have not made the switch from Linode.

Re: Linode Security Advisory

#65

Post by joepie91_ »

Sir_Cmpwn wrote: post #12
matt_wulfeck wrote: post #10

> We have been working with federal authorities on these matters and their criminal investigations are ongoing. I cringe when I see companies say this. As if we're supposed to feel like the "hack" was somehow more sophisticated than spearfishing or social engineering because there's feds on the case. There's a hole in your security. Diligently look for that hole. If it's a mistake own up to it fully and apologize. Ma…

Disclaimer: Linode employee Regardless of the severity of the means, the fact is that this sort of attack is entirely illegal and involving law enforcement is a clear requirement.

Point being that your users have no need or desire to know that. It's a cheap point to score for the marketing department, to detract from the issue at hand - how it could have been compromised to begin with.

Re: Linode Security Advisory

#66

Post by eatonphil »

elbee wrote: post #59
msbarnett wrote: post #49

Earlier quoted context omitted.

> Waiting until login until you upgrade to bcrypt is a requirement is compotent password storage It's not even remotely competent. This blog makes it clear they're not even sure how their secret key was stolen. These hashes could be walking out their backdoor as I type this. Keeping vulnerable hashes at rest is insane. It would be far more competent to bcrypt the SHA-2s, so that at least when the hashes wander out th…

That will work as a way to strengthen the hashes (a few other people pointed that out as well). My point was that if you have a system which can go straight from SHA2(password) to bcrypt(password) then the system must be storing the plaintext of the password, which would be very bad.

I don't see anything to indicate they are converting directly from sha2 to bcrypt. When the user logs in next, if it matches the sha2 hashed password, insert the bcrypt hashed password in the bcrypt hash password field and use that from now on.

Re: Linode Security Advisory

#67

Post by alexbecker »

tptacek wrote: post #6

Update: also, read this comment right away. https://news.ycombinator.com/item?id=11136948 I find this update very hard to follow. Can someone tell me if I'm misreading it? I'm going to quote it twice, and then attempt to summarize: After examining the image from our July investigation, we discovered software capable of generating TOTP codes if provided a TOTP key. We found software implementing the decryption method…

As I read it, the 2FA login does some processing of the credentials before submitting it to their authorization server. Someone logging into a machine directly instead of going through their portal would have to do this processing themselves. They found a program designed to do just this on the compromised machine, which a normal user would be very unlikely to have, and thus serves as strong evidence of malicious activity.

Re: Linode Security Advisory

#68

Post by TheSwordsman »

stephendicato wrote: post #48
TheSwordsman wrote: post #23

Hey There, I'm a PagerDuty employee and am the same individual who made this post on the last HN thread: * https://news.ycombinator.com/item?id=10845985 Unfortunately, there are some facts in Linode's post that are not correct. >On July 9 a customer notified us of unauthorized access into their Linode account. The customer learned that an intruder had obtained access to their account after receiving an email notifica…

> Although, Linode's email isn't what notified is, it was our intrusion detection system. Are you able to elaborate on this? I understand you may not want to name specific vendors/products in the name of operational security but it sounds like in this scenario whatever is in place actually did its job.

Absolutely. We have an old blog post that goes over how we did it awhile ago:

* https://www.pagerduty.com/blog/security-monitoring-alerting-...

Most of this is still valid. There may be some differences as we've improved our configuration over time.

We use OSSEC for host-level intrusion detection. This fired off quite a few alerts as the malicious party began to log in as root on the serial console, amongst other things.

We also have supplemented it with other tools, such as an in-house wrapper around nmap, to alert us to hosts that don't match their expected network configuration. So when ports get opened incorrectly, someone is alerted usually within a minute.

Re: Linode Security Advisory

#69

Post by tptacek »

alexbecker wrote: post #67
tptacek wrote: post #6

Update: also, read this comment right away. https://news.ycombinator.com/item?id=11136948 I find this update very hard to follow. Can someone tell me if I'm misreading it? I'm going to quote it twice, and then attempt to summarize: After examining the image from our July investigation, we discovered software capable of generating TOTP codes if provided a TOTP key. We found software implementing the decryption method…

As I read it, the 2FA login does some processing of the credentials before submitting it to their authorization server. Someone logging into a machine directly instead of going through their portal would have to do this processing themselves. They found a program designed to do just this on the compromised machine, which a normal user would be very unlikely to have, and thus serves as strong evidence of malicious act…

Right, there's that, and also (if I'm reading this right) they found key material.

Re: Linode Security Advisory

#70

Post by tptacek »

ethomson wrote: post #64
TheSwordsman wrote: post #23

Hey There, I'm a PagerDuty employee and am the same individual who made this post on the last HN thread: * https://news.ycombinator.com/item?id=10845985 Unfortunately, there are some facts in Linode's post that are not correct. >On July 9 a customer notified us of unauthorized access into their Linode account. The customer learned that an intruder had obtained access to their account after receiving an email notifica…

This is very helpful information. Can you say if you've moved to a different provider or if you're now racking your own machines? And if you do have a new provider, can you say who you are and how you evaluated them? I have been doing some research in my (limited) spare time to try to find a new provider, but I still have not made the switch from Linode.

If you follow the link he posted, you'll see they switched away from Linode almost immediately after the July breach.
Post reply on HN