Live data from Hacker News

Security Notification and Linode Manager Password Reset

blog.linode.com

161–170 of 173 posts

Re: Security Notification and Linode Manager Password Reset

#161

Earlier quoted context omitted.

Could be just a coincidence if you keep SSH open on standard port. SSH bots don't sleep.

The host was only accessible via private key or Linode's LISH shell. That's what seems most suspicious. There is some minor evidence remaining in the .bash_history that is curious.

I'd be interested in a write-up about anything you find in that .bash_history or logs, would you consider writing one?

Re: Security Notification and Linode Manager Password Reset

#162

Not the first time they poorly handled a security issue: https://blog.linode.com/2014/01/19/an-old-system-and-a-swat-...

There are also a number of reviews on Glassdoor where supposed Linode employees say they were asked by management to lie to customers about security breaches.

We were.

Re: Security Notification and Linode Manager Password Reset

#163
post #91

This is Yet Another Reminder to use unique, unguessable, unmemorable passwords for all online services. It's a question of when, not if, any particular password database will be compromised. While 'password1' and 'this is my long unguessable password' and 'm4r1g0ld' and 'false lemur capacitor paperclip' will all eventually be guessed, 'sF0PSQMwK85fe9xanqJRm9nty9cJGHJsVmti' will never, ever be.

Why was this downvoted? It's true. The example password is 36 characters; a mixed-case-and-numbers alphabet has just over 6 bit per character; the example password is thus 217 bits. It's not going to be guessed.

Re: Security Notification and Linode Manager Password Reset

#164

Earlier quoted context omitted.

The host was only accessible via private key or Linode's LISH shell. That's what seems most suspicious. There is some minor evidence remaining in the .bash_history that is curious.

I'd be interested in a write-up about anything you find in that .bash_history or logs, would you consider writing one?

Here's the main bulk of the obviously bad things:

    3  ls
    4  ls -al
    5  chown syslog auth.log
    6  ls -al
    7  chown syslog kern.log
    8  ls -al
    9  chown syslog syslog
   10  ls -al
   11  echo -n '' > /media/xvda/root/.bash_history
   12  echo -n '' > /root/.bash_history
   13  echo -n '' > /root/.viminfo
   14  L=$(find /var/log -type f); for F in $L; do echo -n '' > $F; done
   15  rm -rf /etc/ssh/*_key* #remove host keys
   16  rm -rf /var/lib/dhcp/* # dhcp leases
   17  echo "echo 'options rotate' >> /etc/resolv.conf" > /etc/dhcp/dhclient-exit-hooks.d/rotate
   18  ls
   19  ls -al /var/log
   20  ls
   21  ls -al /var/log
   22  exit
   23  ls
   24  ls -al /var/log
   27  adduser in
   28  su - in
   29  vi /etc/sudoers
   30  vi /etc/gro
   31  vi /etc/group
   32  groupadd --help
   33  groups
   34  groupmod
   35  groupadd --help
   36  vi /etc/group
   37  su - in

Re: Security Notification and Linode Manager Password Reset

#165
post #91

This is Yet Another Reminder to use unique, unguessable, unmemorable passwords for all online services. It's a question of when, not if, any particular password database will be compromised. While 'password1' and 'this is my long unguessable password' and 'm4r1g0ld' and 'false lemur capacitor paperclip' will all eventually be guessed, 'sF0PSQMwK85fe9xanqJRm9nty9cJGHJsVmti' will never, ever be.

Except passwords aren't even a relevant part of this breach. They're worth absolutely nothing compared to the ridiculous amounts of customer data linode controls.

Re: Security Notification and Linode Manager Password Reset

#166

Earlier quoted context omitted.

I can't speak for the other folks that were compromised this way, but we decided to just cut our losses and move on at PagerDuty and spent the 30 days after the compromise migrating everything that was running there over to Azure. No point in putting pressure on a company that stonewalls you.

That's a good point. Not worth your time for a company like Linode that doesn't really care about its customers. I think people mistake the quick support responses to basic questions as them caring, but when it really comes down to the important things like security and communication during a crisis, it's clear that there is a huge lapse from the leadership level down. Someone in this post wrote about how they stoppe…

When I worked there support was under intense pressure to respond to tickets quickly.

Not nearly as much pressure to respond correctly.

To clarify and reiterate--employees who responded to tickets quickly were praised, even though there response contained half truths or outright falsehoods. If someone took 15-60 minutes ( or more ) to deep dive into an issue for a real fix for a customer, they were shamed and got a talking to.

Re: Security Notification and Linode Manager Password Reset

#167

I'm glad to see that this information has now been publicly disclosed. In July 2015, we suffered a compromise at PagerDuty via the Linode Manager. I hope that we can provide a bit more of an official in-depth post-mortem of our compromise, but I'd be happy to disclose some of the details here. Using the access gained within the Linode Manager, the attacker reset the root password on a few systems, and used Lish to ga…

I'm curious what kind of losses you've seen since your own data breach. I'm sorry but you have far too many plausible ulterior motives for taking the position that you are taking.

1) As someone else pointed out, you're an ex-employee of Linode. You went out of your way to hide this fact. I'll refrain from listing all of the very obvious reasons why your word on Linode should be taken with a grain of salt at the very least.

2) Being able to blame Linode for your own data breach is a fantastically easy (although lazy) way to pacify customers about the fact that their personal data was just pilfered by someone on your watch.

All that being said, what have you presented that can be proven? All that can be proven is that you're an ex Linode employee. Everything else is hot air that we're all meant to take your word. Tons of appeal to authority in your explanations. You keep invoking some mysterious third party "expert" security group who conveniently agrees with everything your own company "discovered." If you were actually confident in your own abilities and that of your team members, there wouldn't be an immediate appeal following every attempted assertion.

Plus, even if you really did hire someone, what company isn't going to just say "yes" and agree to whatever PR campaign their customer is saying while dumping wheelbarrows of cash into their pockets? Frankly I don't believe you, and I find your consistent drumming against Linode to be highly suspicious in the wake of these attacks. You're not involved, are you?

The Linode post you're referring to is just saying that they expired everyone's passwords. That's not admitting anything, especially not admitting anything about a separate incident from a year ago. What lawyer would ever take this and say "okay, you can legally publicly blame linode now?" No lawyer worth his salt. In other words, you're full of shit. Your story if full of holes, tells, and I think you should stop posting so much garbage before you're on the receiving end of a lawsuit or are considered a suspect.

Re: Security Notification and Linode Manager Password Reset

#168

Earlier quoted context omitted.

Well, at linode you can't have a structure that is immune to failover, as they have single points of failure within their infrastructure, apart from anything else - all their London kit for instance lives in Telehouse East, in a few adjacent racks. Once we'd done the initial up sticks and move to AWS, our first priority was to use their redundancy and failover to the fullest (six months of sleepless nights due to lin…

Only one 9? Even through this crap during the holidays I've managed 3 9's on my service hosted on several servers in Linode Dallas (the most hard-hit region in this DDoS attack). I would have moved to AWS by now if Linode didn't have such cheaper bandwidth.

ATL was much worse than Dallas in this go around. You did not get 3 9s in the last month if your service was located solely in ATL. They were completely null routed for the majority of 2 days.

Re: Security Notification and Linode Manager Password Reset

#169
post #80

Earlier quoted context omitted.

Since I'm not really planning to look for a job, my main worry was potential visa issues. But I haven't had any troubles visiting the few countries I do need a visa for.

How are you in a position where looking for a job in the future is not really necessary?

Talented hackers are not exactly unemployable.

Re: Security Notification and Linode Manager Password Reset

#170

I'm glad to see that this information has now been publicly disclosed. In July 2015, we suffered a compromise at PagerDuty via the Linode Manager. I hope that we can provide a bit more of an official in-depth post-mortem of our compromise, but I'd be happy to disclose some of the details here. Using the access gained within the Linode Manager, the attacker reset the root password on a few systems, and used Lish to ga…

I'm curious what kind of losses you've seen since your own data breach. I'm sorry but you have far too many plausible ulterior motives for taking the position that you are taking. 1) As someone else pointed out, you're an ex-employee of Linode. You went out of your way to hide this fact. I'll refrain from listing all of the very obvious reasons why your word on Linode should be taken with a grain of salt at the very…

Are you Jesse Nicholson? Is Linode paying you to write this? This reads like a Linode executive wrote it. That's the kind of thing you should disclose. Your only two HN comments are regarding Linode; pretty suspicious. You also commented on their blog post:

TechnikEmpire January 6th, 2016 at 10:28 pm. It's hilarious watching all of these armchair experts criticize Linode for the actions of another.

PagerDuty and WP Engine were both compromised 'inexplicably' during the same timeframe at the same hosting provider. Seems pretty self explanatory. Linode didn't disclose their "security firm" so why should PagerDuty? Linode couldn't explain how accounts were accessed and it isn't the first time! Linode is hacked once a year; it's a feature. They need to get their shit together and stop pretending security is a game.

Post reply on HN