Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linode Security Advisory

blog.linode.com

Post reply on HN
Print view View original

71–80 of 119 posts

Re: Linode Security Advisory

#71

Post by snuxoll »

ploxiln wrote: post #63
nsgf wrote: post #31

Earlier quoted context omitted.

Me too (for about 5 years), but this was the last straw. Seriously. Recently moved to a dedicated kimsufi (worst support but great VFM hardware-wise).

I thought this kimsufi thing sounded interesting so I looked it up, and ... wow the processors in the systems they offer are old. Where do they even get these things? Must be leftover from OVH? Xeon E5504 - 45nm process, launched 2009 Core i5-2300 - 32nm process, launched 2011, discontinued 2012 And the Core i5 does not support ecc memory, of course, so what kind of servers has it been sitting in for 4 years?

kimsufi === OVH.

Re: Linode Security Advisory

#72

Post by ryanlol »

dang wrote: post #44
ryanlol wrote: post #8

I'm just going to leave this glassdoor review here: https://i.imgur.com/sJd56AT.png

Please don't post off-topic replies to the top comment. Or to any comment, really, but with the top comment people sometimes do so to get their own post closer to the top of the page, which is not legit. We detached this subthread from https://news.ycombinator.com/item?id=11136707 and marked it off-topic.

I'm not sure if you actually read the top comment, because my response was very much relevant.

tptacek asked about inconsistencies in the Linode advisory, I posted a screenshot of an apparent Linode employee claiming that they lie to their customers regarding security issues. (Which is a claim I am willing to personally back up)

Re: Linode Security Advisory

#73

Post by stephendicato »

TheSwordsman wrote: post #68
stephendicato wrote: post #48

Earlier quoted context omitted.

> Although, Linode's email isn't what notified is, it was our intrusion detection system. Are you able to elaborate on this? I understand you may not want to name specific vendors/products in the name of operational security but it sounds like in this scenario whatever is in place actually did its job.

Absolutely. We have an old blog post that goes over how we did it awhile ago: * https://www.pagerduty.com/blog/security-monitoring-alerting-... Most of this is still valid. There may be some differences as we've improved our configuration over time. We use OSSEC for host-level intrusion detection. This fired off quite a few alerts as the malicious party began to log in as root on the serial console, amongst other thi…

Thank you very much!

Re: Linode Security Advisory

#74

Post by ljoshua »

moondev wrote: post #58
anaphor wrote: post #47

I've been using Linode for years and I haven't had too many problems with the service itself. That being said, this makes me think twice about staying with them. If I wanted to switch, is the only real competitor Digital Ocean or are there any other good choices?

Why not AWS? They have nano instance types for as little as $5 monthly and infinity more flexibility.

For me, because once you get beyond the $5 nano/free tier, things get expensive really quick.

For example, I run about 10 different sites off one Linode, but only one of them gets any substantial traffic. Still, in order to run that site, which works just fine on a $20/mo 2GB/2core unit on Linode, I'll push out about 150GB in outbound bandwidth a month, and require around 15-20GB in storage. Pricing that out on AWS, ignoring the free tier, I'm looking at at least $40/mo, so double the price.

These are hobby sites or side projects, so they are not necessarily mission critical, but it would be sad if they went down.

There's a serious benefit to getting a VPS when you're on the low end like that, where power and bandwidth per dollar are greatly maximized.

Re: Linode Security Advisory

#75

Post by Veratyr »

anaphor wrote: post #47

I've been using Linode for years and I haven't had too many problems with the service itself. That being said, this makes me think twice about staying with them. If I wanted to switch, is the only real competitor Digital Ocean or are there any other good choices?

Vultr is the best I've found. The UI and website isn't as pretty as DO or AWS but:

- It's consistently faster on most metrics than other providers I've tested.

- It's full KVM and allows you to run any OS you like. As long as it's KVM virtualisable and you can install it from an ISO it's an option.

- It's present in a ton of DCs.

Re: Linode Security Advisory

#76

Post by TheSwordsman »

drewcrawford wrote: post #60
tptacek wrote: post #34

Earlier quoted context omitted.

I'm not just trying to be argumentative. Here, let me quote more specifically: no evidence of abuse or misuse of Linode’s infrastructure that would have resulted in the disclosure of customer credentials. I feel like I must be misreading something. Didn't they say earlier that they found secrets from their account credentials database on a customer instance that was used to attack (apparently) PagerDuty? That's not "…

you forgot "that would have resulted in the disclosure of customer credentials". I'm speculating, but having been a fly on the wall at this kind of meeting before, here's my theory of how this went down: Manager: "So somebody got the key to generate one-time password tokens for PagerDuty. How did that happen?" Engineer: "I have no idea." Manager: "What about that Lish vulnerability? Could it have been that?" Engineer…

In our case someone logged in to the Linode Manager first. Whomever logged in was confident they knew the password of the user and the MFA token.

That would indicate that Linode's password hashes may have been cracked offline.

Re: Linode Security Advisory

#77

Post by anaphor »

jsmthrowaway wrote: post #38
ryanlol wrote: post #37

Earlier quoted context omitted.

Pretending to do that is ok? I can't believe you're still employed there. Also, I upvoted the comment because I think it should stay visible. I hope other users do that too instead of just downvoting.

He's not. And I can describe his penis, so. There was also stabbing employees (to the point of requiring an ambulance) while fooling around with a knife, setting the building on fire more than once, and tormenting other employees who he didn't like. All of that was tolerated and dismissed by management, specifically Chris Aker and Tom Asaro, which should tell you what you need to know about ever working there. One of…

Not surprised they have such a toxic/brogrammer work environment. Some of the people on their IRC channel seemed really rude when I used to go on there (including the person who did that). Also I used to know a woman from one of the FOSS communities I'm involved with who worked there a few years ago and it definitely seemed like they treated her in a very tokenized way.

Re: Linode Security Advisory

#78

Post by msbarnett »

elbee wrote: post #59
msbarnett wrote: post #49

Earlier quoted context omitted.

> Waiting until login until you upgrade to bcrypt is a requirement is compotent password storage It's not even remotely competent. This blog makes it clear they're not even sure how their secret key was stolen. These hashes could be walking out their backdoor as I type this. Keeping vulnerable hashes at rest is insane. It would be far more competent to bcrypt the SHA-2s, so that at least when the hashes wander out th…

That will work as a way to strengthen the hashes (a few other people pointed that out as well). My point was that if you have a system which can go straight from SHA2(password) to bcrypt(password) then the system must be storing the plaintext of the password, which would be very bad.

> My point was that if you have a system which can go straight from SHA2(password) to bcrypt(password) then the system must be storing the plaintext of the password, which would be very bad.

Yes, I understand that. It's just completely irrelevant to the question of whether or not it's competent practice to store vulnerable hashes indefinitely, awaiting customer log in.

Again, it is not a competent practice. Wrap vulnerable hashes in strong ones immediately; they're a huge liability to leave sitting in your storage even when you don't have evidence that there's a backdoor in your systems that you cannot seem to find.

Re: Linode Security Advisory

#79

Post by Thalagyrt »

ethomson wrote: post #64
TheSwordsman wrote: post #23

Hey There, I'm a PagerDuty employee and am the same individual who made this post on the last HN thread: * https://news.ycombinator.com/item?id=10845985 Unfortunately, there are some facts in Linode's post that are not correct. >On July 9 a customer notified us of unauthorized access into their Linode account. The customer learned that an intruder had obtained access to their account after receiving an email notifica…

This is very helpful information. Can you say if you've moved to a different provider or if you're now racking your own machines? And if you do have a new provider, can you say who you are and how you evaluated them? I have been doing some research in my (limited) spare time to try to find a new provider, but I still have not made the switch from Linode.

We use AWS us-west-1, us-west-2 and Azure Fresno. Azure replaced Linode very quickly after the July incident.

Re: Linode Security Advisory

#80

Post by anaphor »

Veratyr wrote: post #75
anaphor wrote: post #47

I've been using Linode for years and I haven't had too many problems with the service itself. That being said, this makes me think twice about staying with them. If I wanted to switch, is the only real competitor Digital Ocean or are there any other good choices?

Vultr is the best I've found. The UI and website isn't as pretty as DO or AWS but: - It's consistently faster on most metrics than other providers I've tested. - It's full KVM and allows you to run any OS you like. As long as it's KVM virtualisable and you can install it from an ISO it's an option. - It's present in a ton of DCs.

Awesome, I'll look into it. I definitely like being able to create my own custom ISO and install it. Thanks!
Post reply on HN