Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Namecheap live chat social engineering leads to loss of 2 VPS

postphp.com

Post reply on HN
Print view View original

181–190 of 426 posts

Re: Namecheap live chat social engineering leads to loss of 2 VPS

#181

Post by thomnottom »

tyingq wrote: post #25

Not trying to be snarky, but the biggest lesson here seems to be "don't operate without off-host backups". Cheap VPS providers don't typically offer that sort of thing as a standard feature. Even when they do, the backups would be on the same infrastructure, and easily wiped from the same (compromised) console. You could have just as easily lost all the data in an accidental way, with no malice or 3rd party involved.…

I don't see how that can be the biggest lesson. Someone at the service provider bypassed their own protocols in order to hand control of the system over to an unauthorized user. Even with local backups he would have needed to restore the servers because Namecheap royally screwed up.

Yes, you should always have more backups than you need. But wouldn't you be moving to a different provider after something like this anyway?

Re: Namecheap live chat social engineering leads to loss of 2 VPS

#182

Post by endemic »

TheSockStealer wrote: post #126
matthewdrussell wrote: post #102

Earlier quoted context omitted.

Everyone should practice a good backup routine and take responsibility for backups.

Agreed. However, is this messaged anywhere in your documentation or setup instructions? Do you provide instructions how how to set this up with a 3rd party or list of 3rd parties? Although backups are #1 item on any list of best practices, making an easy, and tested, implementation method would be a good practice on your part.

When the product is an unmanaged VPS, I think certain assumptions can (hopefully) be made about the capabilities of the customer.

Re: Namecheap live chat social engineering leads to loss of 2 VPS

#183

Post by gherkin0 »

tamar wrote: post #149
pfarnsworth wrote: post #130

Social engineering in tech has been around since before Kevin Mitnick publicized it and went to jail (unjustly). Why do we keep making the same mistakes over and over again as an industry? We NEED UNIFORM security standards with ALL trusted companies with customer support, where we have tiers of support, and 1st tier doesn't have any access that could compromised security. Similar to ISO standards. This means there c…

I hate to break it to you, but "uniform" security standards that are out there in the open would be like a whole can of worms. That is like showing someone "here's a lock and what's inside of it." In time, someone will pick that lock. Uniformity is what you don't need, nor would you want to know the nuances of how security and privacy are handled at a company so that you know exactly what holes need to be exposed. Yo…

You're basically advocating for security through obscurity.

A standardized process design could be carefully examined and improved to plug the holes, so all you're left with are implementation bugs. You'll never get there with a thousand disparate processes: they'll have design and implementation bugs, as well situations where system compromises data used to secure another.

Plus, standardized processes would allow implementation of more expensive processes. For instance, you could have a higher-grade fallback "prove who you are" process that involves going to some designated office in-person with all the right documents. Not even Google would pay to setup such offices in every city, but if Google, Amazon, Online Banks, etc. all would use it, it might be possible.

Re: Namecheap live chat social engineering leads to loss of 2 VPS

#184

Post by tamar »

beeboop wrote: post #145
mindcrime wrote: post #137

Earlier quoted context omitted.

It's not an argument you're going to win. Unless you sign up for a managed service that claims to include backups or whatever, you are responsible for your own backups. What's controversial about that?

The issue is that Namecheap was the one that fucked up here, and now is not the time to emphasize "you should really be prepared for us fucking up in this manner". It's victim blaming. It looks shitty. The argument I refer to isn't "you should have offsite backups". The argument is that Namecheap is implicitly victim blaming, and they're not going to convince many people that they aren't.

It's not victim blaming. It's simply a reiteration that it helps to have this in place if you are specifically opting to rent/lease a server that does not offer it.

Also, it's stated in the knowledgebase that it is advisable to set up server backups of your own if you do not have a managed server: https://www.namecheap.com/support/knowledgebase/article.aspx...

Re: Namecheap live chat social engineering leads to loss of 2 VPS

#185

Post by HemanHeartYou »

kelukelugames wrote: post #148
tamar wrote: post #117

Earlier quoted context omitted.

I don't think it was personal, simply a reminder that it always helps to have good backup procedures in place. Even my managed services have offsite backups. Better be safe than sorry, I always say.

"Better be safe than sorry" - namecheap for when you lose your stuff on their services. I don't think the best way to respond to a public vent is "Here's what you should have done instead". Responses might be technically correct but they lack empathy for the customer.

"Hard drives never fail" - kelukelugames

Re: Namecheap live chat social engineering leads to loss of 2 VPS

#186

Post by brianwawok »

pilom wrote: post #147
brianwawok wrote: post #77

Earlier quoted context omitted.

It's tricky because a lot of customers really DO lock themselves out of a service, and forget their password reset code. Fun story time. I use to play MTGO, the online Magic the Gathering game. Played it from beta for a few years say 2002-2004. Wanted to check it out in 2014 to see how it changed. Failed password reset online, had to call in to support. The support guy was like chortle what was your security passcode…

The answer to "I don't know my password and I don't know my security question/answer" is, "Sorry, for security reasons we can't help you access this account, you'll need to create a new account." This isn't a problem for banks, why is it a problem for tech companies?

With MTGO above I had maybe $500 in virtual stuff on my account. As the gatekeeper I'm not sure that would go over well.

With servers a similar thing. Say my only copy of a database is on my VPS. May have a business value of $50k. Can't really just say no unconditionally. Need some process to unlock..

Re: Namecheap live chat social engineering leads to loss of 2 VPS

#187

Post by swanson »

matthewdrussell wrote: post #21

Disclaimer: I'm CIO @ Namecheap 1. The credentials were resent to an already compromised email account 2. This is an isolated case 3. Established procedure was not followed 4. With thissaid, we've used this as a learning example and additional training has been provided to the individual involved 5. Anyone with any self-managed server with ANY provider should always keep their own multiple backups

My hobby: role-playing how I would respond as the CEO if my company was getting skewered on HN. Here is my version!

---

Disclaimer: I'm [not] CIO @ Namecheap

We messed up, big time. While we handle 1000s of live chat sessions everyday without issue, I realize that even one breakdown in security protocol can cause huge problems and a loss of trust for our customers.

In response to this isolated case (in which our established procedure was not followed), we will be creating additional training material for all our live support staff. Additionally, we will be exploring technical solutions to try to make this kind of breakdown much harder. Mistakes happen, but if we can prevent them, it is worth doing.

We also would like to take this opportunity to remind folks that any self-managed server (regardless of provider) should always be backed up in multiple places. For information on how to do this with Namecheap, we've published a guide here:

I've reached out to author of the post already by email and we are working to help them resolve any outstanding issues.

Re: Namecheap live chat social engineering leads to loss of 2 VPS

#188

Post by opendomain »

matthewdrussell wrote: post #21

Disclaimer: I'm CIO @ Namecheap 1. The credentials were resent to an already compromised email account 2. This is an isolated case 3. Established procedure was not followed 4. With thissaid, we've used this as a learning example and additional training has been provided to the individual involved 5. Anyone with any self-managed server with ANY provider should always keep their own multiple backups

Matthew,

It sounds like you are confirming that this incident did happen and it was your fault for not following your procedures.

I am not a lawyer, but since there was signification loss, it would probably be in your best interest to offer better reparations.

OpenDomain has several domains that are on NameCheap - I will transfer them immediately since it appears you do not care about customers.

Re: Namecheap live chat social engineering leads to loss of 2 VPS

#189

Post by tamar »

kelukelugames wrote: post #65
matthewdrussell wrote: post #21

Disclaimer: I'm CIO @ Namecheap 1. The credentials were resent to an already compromised email account 2. This is an isolated case 3. Established procedure was not followed 4. With thissaid, we've used this as a learning example and additional training has been provided to the individual involved 5. Anyone with any self-managed server with ANY provider should always keep their own multiple backups

I love namecheap but 5 sounds like victim blaming. Come on. EDIT: My use of the term is a bit strong. I feel frustrated that company execs cannot explicitly admit a mistake or apologize. I should have worded it differently. EDIT2: just for Tamar. By explicit I mean literally using the words "sorry", "apologize", or "mistake". What we have is the standard corporate nonapology. EDIT3: congrats to Tamar for being promot…

Re: Your edit - just a note, it is very clear here that in points 1-4, Namecheap has acknowledged a mistake. That's exactly why there was a lot of training (and retraining) internally to ensure this mistake does not recur. But we do acknowledge it is an isolated incident. That doesn't mean it's not less important - we're fully aware of what happened here and it will not recur.

Re: Namecheap live chat social engineering leads to loss of 2 VPS

#190

Post by peterwwillis »

matthewdrussell wrote: post #128
kelukelugames wrote: post #114

Earlier quoted context omitted.

This is not good damage control/PR. You are letting ego get in the way.

I respectfully disagree. I'm here, along with Tamar, reviewing and considering each point posted. There's some good suggestions and we're listening. The opposite of what I'm suggesting is that people - individuals/companies - do not look after their own backups. That's a dangerous precedent.

Imagine you just lost two servers you can't replace, or you're a potential customer reading this thread, and are afraid of the same.

This is what they read as the company's response to this loss:

"Anyone with any self-managed server with ANY provider should always keep their own multiple backups. Dumbass."

Note the change I made at the end to reflect how some people [who are empathizing with someone who was attacked and lost their property] will interpret that statement. Did any of that statement help the situation at all? Did it help customers feel better? Or did it have the opposite effect? Would this be considered a good way to engender goodwill for your brand?

Now consider this reinterpretation of the statement:

"With self-managed servers, it is good best practice to keep multiple backups for yourself, no matter who your service provider is."

Post reply on HN