Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Security Notification and Linode Manager Password Reset

blog.linode.com

Post reply on HN
Print view View original

21–30 of 173 posts

Re: Security Notification and Linode Manager Password Reset

#21

Post by sarciszewski »

ryanlol wrote: post #15
mapgrep wrote: post #13

Earlier quoted context omitted.

I was just thinking the same thing. I've been a customer for >10 years but this is getting ridiculous. First 2013 attack was apparently exacerbated by cleartext password storage for LISH (their management shell) and API tokens https://marco.org/2013/04/16/linode-hacked The 2012 Bitcoin attack involved a breach of Linode's customer service portal http://arstechnica.com/business/2012/03/bitcoins-worth-22800... Today's…

Actually the 2013 hack was caused by linode running blatantly misconfigured CF installations, like doing stuff that the manual has big warnings about.

For those not in the know, ryanlol was one of the people on the team involved in the 2013 hacks.

Re: Security Notification and Linode Manager Password Reset

#22

Post by artursapek »

madaxe_again wrote: post #18
jpea wrote: post #10

Earlier quoted context omitted.

So, what happened during the AWS outage this past Fall? Or did you restructure to provide failover beyond just moving platforms?

Well, at linode you can't have a structure that is immune to failover, as they have single points of failure within their infrastructure, apart from anything else - all their London kit for instance lives in Telehouse East, in a few adjacent racks. Once we'd done the initial up sticks and move to AWS, our first priority was to use their redundancy and failover to the fullest (six months of sleepless nights due to lin…

Only one 9? Even through this crap during the holidays I've managed 3 9's on my service hosted on several servers in Linode Dallas (the most hard-hit region in this DDoS attack). I would have moved to AWS by now if Linode didn't have such cheaper bandwidth.

Re: Security Notification and Linode Manager Password Reset

#24

Post by throway-not-909 »

ryanlol wrote: post #15
mapgrep wrote: post #13

Earlier quoted context omitted.

I was just thinking the same thing. I've been a customer for >10 years but this is getting ridiculous. First 2013 attack was apparently exacerbated by cleartext password storage for LISH (their management shell) and API tokens https://marco.org/2013/04/16/linode-hacked The 2012 Bitcoin attack involved a breach of Linode's customer service portal http://arstechnica.com/business/2012/03/bitcoins-worth-22800... Today's…

Actually the 2013 hack was caused by linode running blatantly misconfigured CF installations, like doing stuff that the manual has big warnings about.

A customer warned Linode team about the exposed CF folder. CEO aggressively shrugged it off. "That doesn't matter, it's nothing, that's a non issue." Dev who was a bit of a suck up parroted the same telling support to shut up about it. This was six months before HTP happened.

Re: Security Notification and Linode Manager Password Reset

#25

Post by throway-not-909 »

sarciszewski wrote: post #21
ryanlol wrote: post #15

Earlier quoted context omitted.

Actually the 2013 hack was caused by linode running blatantly misconfigured CF installations, like doing stuff that the manual has big warnings about.

For those not in the know, ryanlol was one of the people on the team involved in the 2013 hacks.

Really? Thomas Asaro told us they were all in jail.

Re: Security Notification and Linode Manager Password Reset

#26

Post by ryanlol »

throway-not-909 wrote: post #24
ryanlol wrote: post #15

Earlier quoted context omitted.

Actually the 2013 hack was caused by linode running blatantly misconfigured CF installations, like doing stuff that the manual has big warnings about.

A customer warned Linode team about the exposed CF folder. CEO aggressively shrugged it off. "That doesn't matter, it's nothing, that's a non issue." Dev who was a bit of a suck up parroted the same telling support to shut up about it. This was six months before HTP happened.

We were aware of it for probably an year before anyone bothered to spend 10 minutes looking at coldfusion source. That's all the time it took.

Re: Security Notification and Linode Manager Password Reset

#27

Post by ryanlol »

throway-not-909 wrote: post #25
sarciszewski wrote: post #21

Earlier quoted context omitted.

For those not in the know, ryanlol was one of the people on the team involved in the 2013 hacks.

Really? Thomas Asaro told us they were all in jail.

Nobody went to jail, I'm the only person being prosecuted. I won't go to jail.

Re: Security Notification and Linode Manager Password Reset

#28

Post by madaxe_again »

artursapek wrote: post #22
madaxe_again wrote: post #18

Earlier quoted context omitted.

Well, at linode you can't have a structure that is immune to failover, as they have single points of failure within their infrastructure, apart from anything else - all their London kit for instance lives in Telehouse East, in a few adjacent racks. Once we'd done the initial up sticks and move to AWS, our first priority was to use their redundancy and failover to the fullest (six months of sleepless nights due to lin…

Only one 9? Even through this crap during the holidays I've managed 3 9's on my service hosted on several servers in Linode Dallas (the most hard-hit region in this DDoS attack). I would have moved to AWS by now if Linode didn't have such cheaper bandwidth.

Yeah... our issues mostly arose from the fact that at the time, they were advertising 1Gbps node interconnect - which actually turned out to be 1Gbps HOST interconnect, with the nodes actually throttled down to 50Mbps. We use memcached extensively, and this was absolutely crippling for performance. It didn't help that they furiously denied that they were throttling until we demonstrated it beyond all doubt.

They did obligingly increase these caps when we begged them to do so, but at that point the writing was on the wall, and we kept on bumping into other weird and wonderful limitations and issues, such as the fact that someone running an intense job on the same host could bring our VPS's to an absolute crawl.

It really is a shame, as we desperately wanted to make it work, as we liked Chris's hands on approach (very much like ours), but ultimately our confidence in them was so eroded by the point that things started going genuinely wrong on their end that we had no choice but to leave.

As I said, we kept random small single-server stuff (wordpress sites mostly) there, as if you're not dealing with their networking, performance is generally OK - but the network limitations were an absolute clincher for us, and it was at one point literally every day that we'd find that one of their switches had broken, or we couldn't ARP IP's for no apparent reason, etc. etc.

Re: Security Notification and Linode Manager Password Reset

#29

Post by Mahn »

dmuth wrote: post #23

Looks like the reason their blog is down is because... it's now being targeted by a DoS: http://status.linode.com/incidents/kldhjpjnfnkj Attacking a blog talking about the hack? It sure seems that someone has a grudge against Linode. :-/

At this point I'm starting to wonder whether this isn't a competitor putting their investors money to work. It's otherwise utterly bizzare that someone would be so obsessive in damaging Linode. I really hope they make the details of the investigation public...

Re: Security Notification and Linode Manager Password Reset

#30

Post by threeseed »

ryanlol wrote: post #15
mapgrep wrote: post #13

Earlier quoted context omitted.

I was just thinking the same thing. I've been a customer for >10 years but this is getting ridiculous. First 2013 attack was apparently exacerbated by cleartext password storage for LISH (their management shell) and API tokens https://marco.org/2013/04/16/linode-hacked The 2012 Bitcoin attack involved a breach of Linode's customer service portal http://arstechnica.com/business/2012/03/bitcoins-worth-22800... Today's…

Actually the 2013 hack was caused by linode running blatantly misconfigured CF installations, like doing stuff that the manual has big warnings about.

The best thing about the 2013 hack was that news of it was on Slashdot days before it was mentioned to customers.

I've been consistently saying this for years. Linode is a joke and you would be crazy to use them for anything other than toy/non-critical use cases.

Post reply on HN