Blog is down. Maybe they should host it on WPEngine. (:
This sort of thing almost became background noise when I worked there. I can say that things improved somewhat after the HTP compromise, at least for awhile.
Security Notification and Linode Manager Password Reset
11–20 of 173 posts
Re: Security Notification and Linode Manager Password Reset
#12This is at least the fifth time I can count that linode has been hacked, really? Maybe it's time to ditch that coldfusion stack? Edit: 1.The bitcoin hacks, March 2012 2.HTP hack, April 15, 2013 (CF exploit) 3.Second HTP hack April 16, 2013 (Another CF exploit) 4.MySQL server that allowed anonymous logins (?!?!) January 19, 2014 5.This hack I'm not counting their domain name and various other parts of their infrastruc…
>Maybe it's time to ditch that coldfusion stack? [Linode developer here] Been working on (re)writing things in Python since I started. It takes a while, but I think everyone recognizes that the CF codebase is difficult to maintain. The good news is that significant progress is being made and we're still doing routine audits of the existing stuff.
>we're still doing routine audits of the existing stuff.
This doesn't work, auditing coldfusion code is impossible without auditing the entire platform. The whole platform is so full of bugs and strange behaviour that it's actually impossible to produce secure coldfusion code.
Re: Security Notification and Linode Manager Password Reset
#13This is at least the fifth time I can count that linode has been hacked, really? Maybe it's time to ditch that coldfusion stack? Edit: 1.The bitcoin hacks, March 2012 2.HTP hack, April 15, 2013 (CF exploit) 3.Second HTP hack April 16, 2013 (Another CF exploit) 4.MySQL server that allowed anonymous logins (?!?!) January 19, 2014 5.This hack I'm not counting their domain name and various other parts of their infrastruc…
I was just thinking the same thing. I've been a customer for >10 years but this is getting ridiculous.
First 2013 attack was apparently exacerbated by cleartext password storage for LISH (their management shell) and API tokens https://marco.org/2013/04/16/linode-hacked
The 2012 Bitcoin attack involved a breach of Linode's customer service portal http://arstechnica.com/business/2012/03/bitcoins-worth-22800...
Today's attack is some kind of unspecified or unknown breach involving Linode manager.
I guess the obvious commonality here is that all the attacks target the "soft" Linode layers AROUND managing deploys of Linux and Xen/KVM/UML rather than the "hard" targets of those widely used systems. This also happens to be the layer where Linode should be adding value (as opposed to the cheaper VPS providers out there) and I think it's increasingly troublesome that they continue to have such severe security issues.
Is this company (CEO - Christopher Aker) not investing in security staff, security training, best practices etc, or are they investing tons and just getting breached because they host so many sites? Unclear. But it's easy to imagine it's the former, from the outside, given all these incidents.
Re: Security Notification and Linode Manager Password Reset
#14Earlier quoted context omitted.
>Maybe it's time to ditch that coldfusion stack? [Linode developer here] Been working on (re)writing things in Python since I started. It takes a while, but I think everyone recognizes that the CF codebase is difficult to maintain. The good news is that significant progress is being made and we're still doing routine audits of the existing stuff.
>we're still doing routine audits of the existing stuff. This doesn't work, auditing coldfusion code is impossible without auditing the entire platform. The whole platform is so full of bugs and strange behaviour that it's actually impossible to produce secure coldfusion code.
I can't really deny that CF is that bad, but it'd be irresponsible to just let the codebase rust as we rewrite it - and we are rewriting it.
Re: Security Notification and Linode Manager Password Reset
#15This is at least the fifth time I can count that linode has been hacked, really? Maybe it's time to ditch that coldfusion stack? Edit: 1.The bitcoin hacks, March 2012 2.HTP hack, April 15, 2013 (CF exploit) 3.Second HTP hack April 16, 2013 (Another CF exploit) 4.MySQL server that allowed anonymous logins (?!?!) January 19, 2014 5.This hack I'm not counting their domain name and various other parts of their infrastruc…
I was just thinking the same thing. I've been a customer for >10 years but this is getting ridiculous. First 2013 attack was apparently exacerbated by cleartext password storage for LISH (their management shell) and API tokens https://marco.org/2013/04/16/linode-hacked The 2012 Bitcoin attack involved a breach of Linode's customer service portal http://arstechnica.com/business/2012/03/bitcoins-worth-22800... Today's…
Actually the 2013 hack was caused by linode running blatantly misconfigured CF installations, like doing stuff that the manual has big warnings about.
Re: Security Notification and Linode Manager Password Reset
#16Earlier quoted context omitted.
>we're still doing routine audits of the existing stuff. This doesn't work, auditing coldfusion code is impossible without auditing the entire platform. The whole platform is so full of bugs and strange behaviour that it's actually impossible to produce secure coldfusion code.
I can't really deny that CF is that bad, but it'd be irresponsible to just let the codebase rust as we rewrite it - and we are rewriting it.
When did you start rewriting it? It doesn't take years to replace this stuff.
Re: Security Notification and Linode Manager Password Reset
#17I'm fairly confident that Linode has been compromised since July, if not earlier. PagerDuty moved off of Linode after an incident in July. We've been under strict gag orders from legal about that incident until today when Linode finally announced their compromise. Really, the only way I can see that this attacker could have gotten in the way they did (they logged into our Linode Manager account on the first try using a username that wasn't used anywhere but in Linode Manager, using valid 2fa and valid password) was if they had access to the Linode Manager database. I'm pretty sure that the recent WP Engine compromise was achieved through the same attack vector, too
Re: Security Notification and Linode Manager Password Reset
#18Earlier quoted context omitted.
Sadly I trust them as far as I can throw them, and we moved everything important from them to AWS a few years back. We had left a few static sites there but after the shenanigans over the holidays, we're moving our remaining stuff. Sad. We spent >$10k/month with them for a while, before their shit started falling apart. They didn't appear to care at all when we left, so I suppose they have an awful lot of large custo…
So, what happened during the AWS outage this past Fall? Or did you restructure to provide failover beyond just moving platforms?
Well, at linode you can't have a structure that is immune to failover, as they have single points of failure within their infrastructure, apart from anything else - all their London kit for instance lives in Telehouse East, in a few adjacent racks.
Once we'd done the initial up sticks and move to AWS, our first priority was to use their redundancy and failover to the fullest (six months of sleepless nights due to linode made this rather front and foremost in our minds) - so nothing that's happened at AWS has ever been anything more than an inconvenience - we've managed five nines since the move - before, we managed one.
Re: Security Notification and Linode Manager Password Reset
#19Earlier quoted context omitted.
I can't really deny that CF is that bad, but it'd be irresponsible to just let the codebase rust as we rewrite it - and we are rewriting it.
When did you start rewriting it? It doesn't take years to replace this stuff.
I was hired in July and have been driving most of this effort and we're shipping rewritten versions of some parts of our infra soon™
Re: Security Notification and Linode Manager Password Reset
#20Earlier quoted context omitted.
Sadly I trust them as far as I can throw them, and we moved everything important from them to AWS a few years back. We had left a few static sites there but after the shenanigans over the holidays, we're moving our remaining stuff. Sad. We spent >$10k/month with them for a while, before their shit started falling apart. They didn't appear to care at all when we left, so I suppose they have an awful lot of large custo…
So, what happened during the AWS outage this past Fall? Or did you restructure to provide failover beyond just moving platforms?
Strange how this works. I moved everything to cloud providers because of availability issues. Now a few years later, I'm moving everything back to dedicated hardware in several different datacenters because of availability issues. Thanks, Docker!! <3