Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Security Hole in Sendgrid

chunkhost.com

Post reply on HN
Print view View original

51–60 of 97 posts

Re: Security Hole in Sendgrid

#51

Post by danielweber »

mantrax3 wrote: post #48
jtchang wrote: post #26

Social engineering will almost always work. I don't really fault Sendgrid for this (though I could see this not working as well if you were using Amazon SES...no support to even talk to!). It sucks that they got caught with their pants down but I bet a good social engineering attempt on ChunkHost might have yielded similar results. The lesson here is to have multiple defenses. 2 factor auth is a great start and it wo…

In a nutshell, social engineering can be countered by proper software & processes engineering. So if social engineering is possible, blame the software architects. Maybe in extreme cases changing the email of an account might be needed, but there's no excuse a first level rep was able to do it. Least thing, he/she should've been forced by the system to escalate to someone above her, who has a much lesser chance to sc…

They didn't say it was first-level rep. Maybe the rep passed it up the chain.

This was a pretty weak request, though. This one should have been pretty easy to at least make some attempt to verify.

But as a human being I can verify how hard it is to not help the person crying on the other end of the phone because they need help RIGHT NOW. This didn't get to that level, but if you are designing a recovery process, you really need to think about how you handle that situation, and make sure the people making the call have the guts to say "I'm sorry but we need to do this one by the book."

Re: Security Hole in Sendgrid

#52

Post by danielweber »

ef47d35620c1 wrote: post #50

Email accounts are the weak link for many things... DNS registrations, web hosting, etc. Get the email account and you have it all.

I'm coming around to thinking you should get a secret email address for your sensitive accounts. Treat the string of that email address as almost as important as the string of your password. The problem is that you do have to tell it to people who probably don't share that philosophy to the same extreme.

Re: Security Hole in Sendgrid

#53

Post by gtaylor »

voltagex_ wrote: post #34
gtaylor wrote: post #28

Earlier quoted context omitted.

We just switched off of SES because of the lack of bounce/rejection diagnostics, and their internal blacklisting policies are really aggressive. If someone's email server goes down for a few hours, they're blacklisted for quite some time, even after it comes back up. After using SES for close to two years, I'd suggest looking elsewhere if you really care about deliverability or stats/metrics. We switched to Mandrill…

Does open/click reporting still work for GMail with the changes to the way they load images?

Check out the bottom of this post for where they stand on that: http://blog.mandrill.com/rejection-imports-subaccount-enhanc...

They get accurate data for clicks, but geolocation/browser stats for opens aren't 100%. They still track the rates, but it looks like they're coming from one of Google's proxies, so this is the best they can do for now.

Re: Security Hole in Sendgrid

#54

Post by alanning »

np422 wrote: post #35
staunch wrote: post #5

Another title for this submission could have been: "Massive Security Hole in ChunkHost. Non-2FA accounts can be owned." Because it turns out anyone with a Sendgrid Support account also effectively had potential access to any account at ChunkHost not using two-factor authentication. Which is also true of thousands of other companies that are relaying their password reset emails through third party SMTP services. SendG…

That would be because sendgrid is lame. I've recently tried to use their service to send emails, club member newsletters. I've never thought much about bulk e-mails before and I thought it was a "solved problem" by now. Sendgrid are well known so they were my first choice. During initial testing I found both bugs in the API and missing functionality. I've worked over 20 years with IT and yet sendgrid support is easil…

We looked into sendgrid and also went with mailgun. Been with them for over a year now. Originally started with their mailing list API but had enough service delay problems that we were considering switching entirely. Don't remember exactly but may have had some outages as well. Their support suggested we change to using their batch sending API instead and I'm glad we stuck with them because it's been solid for over 5 months now. (Well they did just have two issues recently, ssl cert and duplicated email sending but it didn't affect us too much.)

Overall they've been a good service provider and their tech support has always been helpful and pleasant to deal with. (I know! Crazy, right?).

Re: Security Hole in Sendgrid

#55

Post by wojcikstefan »

It's an important lesson for all of us. I've seen a lot of privacy ignorance when it comes to support (e.g. folks handing over sensitive data after an anonymous request on Olark). We should all go an extra mile and verify the identity of the requestor.

1) If your support chat doesn't enforce authorization, always ask the requestor to send you an email. 2) Make sure the domain is correct (that's where Sendgrid screwed up). 3) Never agree on replying to a different email address than the one of the sender.

Re: Security Hole in Sendgrid

#56

Post by cheeseprocedure »

gtaylor wrote: post #28
toomuchtodo wrote: post #22

Earlier quoted context omitted.

If you invoke sending email through Amazon SES, it's free for the first 2K per day, due EC2 blocks being on spam lists. http://aws.amazon.com/ses/pricing/ "You can send 2,000 messages for free each day when you call Amazon SES from an Amazon EC2 instance directly or through AWS Elastic Beanstalk."

We just switched off of SES because of the lack of bounce/rejection diagnostics, and their internal blacklisting policies are really aggressive. If someone's email server goes down for a few hours, they're blacklisted for quite some time, even after it comes back up. After using SES for close to two years, I'd suggest looking elsewhere if you really care about deliverability or stats/metrics. We switched to Mandrill…

> [...] the lack of bounce/rejection diagnostics [...]

My employer has been making more and more use of SES, and we've just started looking at automated processing of feedback notifications [1]. Did you find them lacking?

[1] http://docs.aws.amazon.com/ses/latest/DeveloperGuide/notific...

Re: Security Hole in Sendgrid

#57

Post by balls187 »

toomuchtodo wrote: post #3

Use Amazon SES to generate your outbound emails; ensure proper IAM policies, and that you're using 2 factor auth to login to your AWS account.

Is AWS any less susceptible to Social Engineering attack of this type?

Specifically--can AWS support staff grant access to AWS accounts, and if so:

what are their criteria for doing so, and what are the policies in place to ensure those criteria are met, and how are those policies audited?

As a TechStars alum, my company was granted $50k in AWS credits, which were tied to my AWS account[1]. When I left the Company, the CEO was able to get the credits moved to a different AWS account that was company owned, without my intervention at all, even though I was the only account owner.

The fact that he could have credits moved out of the account without any kind of verification from me[2], should be cause for concern.

[1] I should have created a new Amazon account for a group email [2] Obviously the credits belong to the company; they weren't mine to use, so I would have authorized the migration.

Re: Security Hole in Sendgrid

#58

Post by arjie »

cjbprime wrote: post #4

Looks like a deeply unsatisfactory response from SendGrid. They don't even know for sure ("it appears .. pretty much confirms") that their own support staff changed the email address?

This shows why so-called PR-speak is necessary. Email like this should not be in a conversational style because it can easily be misinterpreted.

Re: Security Hole in Sendgrid

#59

Post by steven2012 »

mantrax3 wrote: post #43
steven2012 wrote: post #38

We use Sendgrid and have hundreds of thousands of customers that might be phished by this social engineering trick. It's absolutely unacceptable that such a crucial piece of infrastructure is vulnerable to such a simple trick. I'm going to bring this up with our team and see if there's another vendor that can more reliably protect our customers.

Your logic is a bit weird. Sendgrid just experienced this major embarrassment and are currently re-training their staff to avoid it again at all costs. And you're going to move away from them now ?

I'll talk it over with my group. Retraining doesn't guarantee anything. It could be that this problem is endemic to the company itself.

For example, why should 1st level support have the ability to make major changes like this? It sounds like only 2nd level support, a smaller group of more highly trained support staff, should have the ability to do this. Does SendGrid have enough money/resources to split their team into 1st and 2nd level support? Would a larger company have those type of resources that would better protect my customers?

These are questions I will talk over with my team.

Re: Security Hole in Sendgrid

#60

Post by vertex-four »

hadoukenio wrote: post #37
vertex-four wrote: post #33

Earlier quoted context omitted.

The solution is to do what everyone who actually needs authentication from a company does; require a posted signed letter from a director, possibly along with an outbound (from SendGrid to the director) phone call to confirm. There's plenty of low-tech ways to confirm that a company really wants to do something.

Please, no. Consider a determined attacker. A posted signed letter has zero cost and is easily forged and a phone call is free via Skype. There's plenty of low-tech ways to circumvent security.

How exactly does Skype let me take over a business's phone number? I am saying that SendGrid should call the company to verify, not the other way round.
Post reply on HN