Security Notification and Linode Manager Password Reset • Hacker News Board

Re: Security Notification and Linode Manager Password Reset

#71

Post by jacquesm » Tue, Jan 05, 2016, 10:31 PM UTC

TheSwordsman wrote: post #42

I'm glad to see that this information has now been publicly disclosed. In July 2015, we suffered a compromise at PagerDuty via the Linode Manager. I hope that we can provide a bit more of an official in-depth post-mortem of our compromise, but I'd be happy to disclose some of the details here. Using the access gained within the Linode Manager, the attacker reset the root password on a few systems, and used Lish to ga…

That's incredibly poor handling by linode, it also quite possibly resulted in many other systems being compromised via the same route or a similar one in the meantime.

Re: Security Notification and Linode Manager Password Reset

#72

Post by AdamGibbins » Tue, Jan 05, 2016, 10:36 PM UTC

madaxe_again wrote: post #18

jpea wrote: post #10

Earlier quoted context omitted.

So, what happened during the AWS outage this past Fall? Or did you restructure to provide failover beyond just moving platforms?

Well, at linode you can't have a structure that is immune to failover, as they have single points of failure within their infrastructure, apart from anything else - all their London kit for instance lives in Telehouse East, in a few adjacent racks. Once we'd done the initial up sticks and move to AWS, our first priority was to use their redundancy and failover to the fullest (six months of sleepless nights due to lin…

I don't think Telehouse East is correct. They're with Telecity, and in their PowerGate facility, i.e. out in Acton: http://www.telecitygroup.com/our-company/news/2010/linode-ex...

Unless they've moved?

Re: Security Notification and Linode Manager Password Reset

#73

Post by mapgrep » Tue, Jan 05, 2016, 10:41 PM UTC

ryanlol wrote: post #26

throway-not-909 wrote: post #24

Earlier quoted context omitted.

A customer warned Linode team about the exposed CF folder. CEO aggressively shrugged it off. "That doesn't matter, it's nothing, that's a non issue." Dev who was a bit of a suck up parroted the same telling support to shut up about it. This was six months before HTP happened.

We were aware of it for probably an year before anyone bothered to spend 10 minutes looking at coldfusion source. That's all the time it took.

V interesting. Do any of the other VPS providers strike you as more secure alternatives?

Re: Security Notification and Linode Manager Password Reset

#74

Post by dfoxinator » Tue, Jan 05, 2016, 10:55 PM UTC

throwaway909009 wrote: post #17

I'm fairly confident that Linode has been compromised since July, if not earlier. PagerDuty moved off of Linode after an incident in July. We've been under strict gag orders from legal about that incident until today when Linode finally announced their compromise. Really, the only way I can see that this attacker could have gotten in the way they did (they logged into our Linode Manager account on the first try using…

I find this very interesting because I think it brings up a specific question. If this is actually from July, and they knew about it, why would Linode choose this specific time to announce the breach publicly? It seems very curious being that they are already getting punished from the DDoS attack and if they waited 6 months they could have waited until the dust settles a little bit.

Unless, of course, the group involved in the breach are also the ones unleashing the DDoS attacks. Which would also make me think there has been communication between the group and Linode, in contrary to what Linode stated.

Re: Security Notification and Linode Manager Password Reset

#75

Post by Thalagyrt » Tue, Jan 05, 2016, 11:02 PM UTC

dfoxinator wrote: post #74

throwaway909009 wrote: post #17

I'm fairly confident that Linode has been compromised since July, if not earlier. PagerDuty moved off of Linode after an incident in July. We've been under strict gag orders from legal about that incident until today when Linode finally announced their compromise. Really, the only way I can see that this attacker could have gotten in the way they did (they logged into our Linode Manager account on the first try using…

I find this very interesting because I think it brings up a specific question. If this is actually from July, and they knew about it, why would Linode choose this specific time to announce the breach publicly? It seems very curious being that they are already getting punished from the DDoS attack and if they waited 6 months they could have waited until the dust settles a little bit. Unless, of course, the group invol…

I think since multiple customers were hit by this and are presumably all putting pressure on them about it, their hand may have been forced.

Re: Security Notification and Linode Manager Password Reset

#76

Post by dfoxinator » Tue, Jan 05, 2016, 11:07 PM UTC

Thalagyrt wrote: post #75

dfoxinator wrote: post #74

Earlier quoted context omitted.

I find this very interesting because I think it brings up a specific question. If this is actually from July, and they knew about it, why would Linode choose this specific time to announce the breach publicly? It seems very curious being that they are already getting punished from the DDoS attack and if they waited 6 months they could have waited until the dust settles a little bit. Unless, of course, the group invol…

I think since multiple customers were hit by this and are presumably all putting pressure on them about it, their hand may have been forced.

Could be, but the timing still seems really weird to me. If they did an investigation like they said they did, I just don't see why you would come out with it right now. Because from Pager Duty post it seems that no one could do anything to make them disclose it sooner.

Re: Security Notification and Linode Manager Password Reset

#77

Post by madaxe_again » Tue, Jan 05, 2016, 11:09 PM UTC

AdamGibbins wrote: post #72

madaxe_again wrote: post #18

Earlier quoted context omitted.

Well, at linode you can't have a structure that is immune to failover, as they have single points of failure within their infrastructure, apart from anything else - all their London kit for instance lives in Telehouse East, in a few adjacent racks. Once we'd done the initial up sticks and move to AWS, our first priority was to use their redundancy and failover to the fullest (six months of sleepless nights due to lin…

I don't think Telehouse East is correct. They're with Telecity, and in their PowerGate facility, i.e. out in Acton: http://www.telecitygroup.com/our-company/news/2010/linode-ex... Unless they've moved?

Ah, that may be it - was a good four years ago, so I guess they were there. Telesomething!

Re: Security Notification and Linode Manager Password Reset

#78

Post by ewindisch » Tue, Jan 05, 2016, 11:10 PM UTC

jacquesm wrote: post #62

ryanlol wrote: post #55

Earlier quoted context omitted.

Difficult question to answer. There was a complete lack of "technical" evidence against me (e.g .bash_history files, wiretaps). The only evidence the prosecution had against me were a list of compromised sites and several coldfusion 0days I had in my possession. They could never prove that I generated the list of compromised sites, but the judges felt that the possession of said list was enough evidence to convict me…

> We would've appealed but there was no point as the sentence was essentially nothing. But you do end up with a record, which is not 'essentially nothing'.

I suppose it depends on what that means for a minor in Finland.

Re: Security Notification and Linode Manager Password Reset

#79

Post by ryanlol » Tue, Jan 05, 2016, 11:13 PM UTC

mapgrep wrote: post #73

ryanlol wrote: post #26

Earlier quoted context omitted.

We were aware of it for probably an year before anyone bothered to spend 10 minutes looking at coldfusion source. That's all the time it took.

V interesting. Do any of the other VPS providers strike you as more secure alternatives?

I'd avoid VPS providers in general, but AWS is on a whole different level than linode. They actually understand what they're doing well enough to do live xen patching etc.

But yeah, people get hacked through their hosts all the time. Best approach is colo with minimum access for the dc staff.

Re: Security Notification and Linode Manager Password Reset

#80

Post by ryanlol » Tue, Jan 05, 2016, 11:17 PM UTC

jacquesm wrote: post #62

ryanlol wrote: post #55

Earlier quoted context omitted.

Difficult question to answer. There was a complete lack of "technical" evidence against me (e.g .bash_history files, wiretaps). The only evidence the prosecution had against me were a list of compromised sites and several coldfusion 0days I had in my possession. They could never prove that I generated the list of compromised sites, but the judges felt that the possession of said list was enough evidence to convict me…

> We would've appealed but there was no point as the sentence was essentially nothing. But you do end up with a record, which is not 'essentially nothing'.

Since I'm not really planning to look for a job, my main worry was potential visa issues. But I haven't had any troubles visiting the few countries I do need a visa for.