IAM is hard – Thoughts on $80M fine from the Capital One Breach • Hacker News Board

Re: IAM is hard – Thoughts on $80M fine from the Capital One Breach

#111

Post by partyboat1586 » Sun, Aug 09, 2020, 8:14 AM UTC

thayne wrote: post #109

gray_-_wolf wrote: post #78

Earlier quoted context omitted.

In our company we went the other road. We have the developers write the policies (since I mean, they know what their app needs) and test it in dev environments. After that, ops guys step in during code review to check for too broad allow in the policies. So far it seems to work in acceptable manner.

In my company, developers write policies, but they have to be approved by someone with ops expertise (usually me). And the policies are often either too broad, or not sufficiently tested, and missing permissions for things it needed. Sometimes the same policy has both problems. I don't blame the developers. You can hardly expect every developer to become an expert on AWS's IAM system. Especially given how inconsisten…

>or not sufficiently tested

How do you test a policy?

Re: IAM is hard – Thoughts on $80M fine from the Capital One Breach

#112

Post by partyboat1586 » Sun, Aug 09, 2020, 8:21 AM UTC

sushshshsh wrote: post #92

brendoelfrendo wrote: post #88

Earlier quoted context omitted.

This sounds like a good process, but it depends a lot on the relationship between dev and ops... I've seen too many dev shops push against changes requested by ops or security because their main pressure is to ship features fast. And then it turns into a management fight and whoever has the more influential management gets the final say while the other side is forced to grumble.

Currently experiencing this at my current employer. I have a suspicion that slowing the dev and deployment process is in everyone's perceived best interest, until of course the day where we are out competed.

And rightly or wrongly you will be outcompeted by places with more risk appetite, until they have a security breach before they are big enough to swallow the cost.

Re: IAM is hard – Thoughts on $80M fine from the Capital One Breach

#113

Post by partyboat1586 » Sun, Aug 09, 2020, 8:23 AM UTC

scarface74 wrote: post #97

brendoelfrendo wrote: post #88

Earlier quoted context omitted.

This sounds like a good process, but it depends a lot on the relationship between dev and ops... I've seen too many dev shops push against changes requested by ops or security because their main pressure is to ship features fast. And then it turns into a management fight and whoever has the more influential management gets the final say while the other side is forced to grumble.

Well, developers can fight all they want to. I was the dev lead at a company that had to be HIPAA compliant. My neck was on the line if we were found to be out of compliance along with the security director and the operations people. Now as a consultant working with many large customers, “shipping fast” is nowhere near as important as security.

I'm in the same position and we haven't built our infrastructure yet.

Did you use a managed service?

Or did you build it slowly and carefully on AWS?

I'm currently stuck between the two options. Managed service seems the easiest way to be HIPAA compliant but I'd rather we managed our own infrastructure on AWS since it gives us more flexibility for stuff like blue green deploys and it would be cheaper.

Re: IAM is hard – Thoughts on $80M fine from the Capital One Breach

#114

Post by partyboat1586 » Sun, Aug 09, 2020, 8:41 AM UTC

tgsovlerkhgsel wrote: post #91

freedomben wrote: post #12

Earlier quoted context omitted.

I've seen this a ton. I have been an on/off security professional so academically I am committed to the principle of least privilege, but holy hell it can be painful or impossible in real life. Where possible I've started adopting the "run it and see" or audit2allow approach (there are awesome tools that can do this for AWS IAM perms too), but then before applying the policy, somebody needs to put a quick line beside…

> Not perfect but sometimes perfect is the enemy of good. This. Add enough hurdles, and people will a) spend all the energy they're willing to spend on your process, hate it, and as a result never do anything except what you force them to, and in particular, never voluntarily reduce permissions unless forced (because they might need them later and then it'll be a pain). They'll also see you as an enemy, not a partner…

Thank you for acknowledging this. I worked at a place like that and it put me off security to the point where I didn't want to ever deny any person or app or service permission to anything and was just happy with securing the edge nodes of our network. I got over it and now I'm in a position where my neck is on the line if there is a security breach I feel differently but I can still relate to myself back then and understand the need to be pragmatic and massage people into following the process rather than forcing them.

Re: IAM is hard – Thoughts on $80M fine from the Capital One Breach

#115

Post by scarface74 » Sun, Aug 09, 2020, 9:09 AM UTC

partyboat1586 wrote: post #113

scarface74 wrote: post #97

Earlier quoted context omitted.

Well, developers can fight all they want to. I was the dev lead at a company that had to be HIPAA compliant. My neck was on the line if we were found to be out of compliance along with the security director and the operations people. Now as a consultant working with many large customers, “shipping fast” is nowhere near as important as security.

I'm in the same position and we haven't built our infrastructure yet. Did you use a managed service? Or did you build it slowly and carefully on AWS? I'm currently stuck between the two options. Managed service seems the easiest way to be HIPAA compliant but I'd rather we managed our own infrastructure on AWS since it gives us more flexibility for stuff like blue green deploys and it would be cheaper.

Back then, I was building a green field project on prem. It was more about limiting access and auditing. In the middle of the implementation a mandate came from on high to “move to the cloud”.

I didn’t know anything about AWS back then, they hired an MSP who was just a bunch of old school netops people who knew how to click around on the console and gave us a bunch of VMs.

Long story short, I studied for one AWS certification so I could talk the talk. I learned both all of the things that I could have taken advantage of and saw how much they were making and that changed my whole m.o. and decided to get some experience with AWS and go into consulting.

Next company I went to, the founders outsourced everything technically to an outsourcing company - software and infrastructure and they treated AWS as an overpriced colo. Everything was in one account and everyone had access to it. At first, they were just aggregating publicly available information about doctors for hospitals so it wasn’t a big deal.

They brought a new CTO in and started bringing development in house. I led the charge to first separate out the environment to different accounts, establish a sane CI/CD process and then lock down who had access to prod.

Of course they had secret access keys in config files everywhere. We had to audit the code to make sure that no code was using keys. Locally, every SDK can automatically retrieve the keys from your global config file (that’s nowhere near your git repo) and on AWS it gets permissions based on the attached roles.

Then of course we had to lock down roles. But we couldn’t have the granular permissions we needed because even though they had lots of microservices (we sold access to our APIs to businesses). They were all on two “pet” EC2 instances.

Next step was to move the .Net Core APIs to Docker/Fargate and further restrict the attached roles to those.

Finally, we had to audit all of our AWS dependencies and add encryption where necessary and then sign a BAA with AWS and bring in auditors.

By the time I left a month ago, we could pass the needed certifications and expand our offerings.

It took a lot of upskilling, hiring an internal ops person (I’m a developer who knows AWS) instead of depending heavily on the MSP.

I left for greener pastures - I’m a consultant with AWS.

Re: IAM is hard – Thoughts on $80M fine from the Capital One Breach

#116

Post by ozim » Sun, Aug 09, 2020, 11:31 AM UTC

Aeolun wrote: post #105

brendoelfrendo wrote: post #88

Earlier quoted context omitted.

This sounds like a good process, but it depends a lot on the relationship between dev and ops... I've seen too many dev shops push against changes requested by ops or security because their main pressure is to ship features fast. And then it turns into a management fight and whoever has the more influential management gets the final say while the other side is forced to grumble.

We have machines to tell people their policy (in dev) is insecure and will be deleted in 30 minutes. Not really easy to argue with the machine.

Nice that just like having bots in discord. When administrators hide behind bots with decisions so users cannot retaliate. It also is nice combined with shadow bans, user thinks his posts are going through but no one replies.

Unfortunately for development it is better to get tight integration of dev and ops so you could solve it by discussion and cooperation. Not sure if you can build such teams that often but that would be great.

Re: IAM is hard – Thoughts on $80M fine from the Capital One Breach

#117

Post by mlthoughts2018 » Sun, Aug 09, 2020, 11:42 AM UTC

Spivak wrote: post #104

mlthoughts2018 wrote: post #65

Earlier quoted context omitted.

I always find the lack of communication goes the other direction. Security and compliance teams just enforce an IAM policy without talking to application or product teams. It gets rolled out and lots of things break, even things that have legitimate needs to work the way they do and have considered security best practices heavily already, and then security and compliance just throws their hands up and says too bad, r…

I have never seen a compliance team make the choice to just break production unless they (and by proxy you) are in hot water with auditors.

That’s weird, because I see it constantly, even for minor systems where the relationship to a compliance requirement is minor / optional. I’ve actually never seen it happen when there is a real security or auditor issue at stake - I’ve only (repeatedly) seen compliance & security teams demand enforcement of a policy that breaks production in circumstances where the whole thing could have been easily prevented if they had gone to product teams and had a conversation first, but they didn’t.

The most recent one I lived through a few months ago was when compliance just all of a sudden decided to wholsesale enforce a bunch or org-wide settings changes to every GitHub repo in the company, and it caused several outages and a huge amount of unplanned triage work as the settings were very sensitive for a bunch of continuous integration systems and jobs.

This was at a Fortune 500 company with a big, well-staffed compliance team. They had to roll back their changes and delay the new settings by several months because only through breaking production did they realize their proposes settings workflow was not feasible given in-house system requirements.

And of course, no apologies at all.

This is pretty run of the mill. I’ve seen the same thing from compliance and security teams in a few other large, “household name” tech companies, and also in a few mid-range startups.

Compliance teams number one MO is to blame product teams for not partnering with them, but it’s the compliance teams who refuse to do the partnering.

Re: IAM is hard – Thoughts on $80M fine from the Capital One Breach

#118

Post by cactus2093 » Sun, Aug 09, 2020, 2:55 PM UTC

StreamBright wrote: post #57

cactus2093 wrote: post #49

One thing I find interesting is that AWS has added some safeguards to the console to protect against exactly this, since it's presumably a very common issue. As of the last couple years when you make any S3 bucket open to the world you see a big warning about it. However if you're following the "industry best practices" and using something like Terraform to manage all your resources including IAM policies, you won't…

>> we've decided that having infra teams manage hundreds or thousands of lines of not-very-human-readable JSON across all the IAM resources they manage is the proper way to do things. This is exactly why with few of my friends started to work on a tool that uses a typed language to express IaC. We can leverage and or relations for AWS objects. One quick example. S3 resource is PublicWebsite or ForwardOnly or PrivateB…

Sounds really interesting, is there a public repo up yet to take a look at?

Re: IAM is hard – Thoughts on $80M fine from the Capital One Breach

#119

Post by digianarchist » Sun, Aug 09, 2020, 9:21 PM UTC

Ozzie_osman wrote: post #27

Apart from IAM complexity, I found it interesting that the author puts the cost of this breach at 80M. Yes, that's the cost of the fine, but the true cost including reputation, etc are definitely way higher than that. 80M to a company like Cap One with almost 30B in revenue is an easy write off. The other costs can be really hard to recover from.

You’re missing development cost to remediate all of the security issues revealed via auditing.

Re: IAM is hard – Thoughts on $80M fine from the Capital One Breach

#120

Post by tatersolid » Sun, Aug 09, 2020, 10:38 PM UTC

mlthoughts2018 wrote: post #117

Spivak wrote: post #104

Earlier quoted context omitted.

I have never seen a compliance team make the choice to just break production unless they (and by proxy you) are in hot water with auditors.

That’s weird, because I see it constantly, even for minor systems where the relationship to a compliance requirement is minor / optional. I’ve actually never seen it happen when there is a real security or auditor issue at stake - I’ve only (repeatedly) seen compliance & security teams demand enforcement of a policy that breaks production in circumstances where the whole thing could have been easily prevented if they…

> I’ve only (repeatedly) seen compliance & security teams demand enforcement of a policy that breaks production in circumstances where the whole thing could have been easily prevented if they had gone to product teams and had a conversation first, but they didn’t.

This is exactly backwards. Product devs need to reach out to security early in the design phase. There’s no way for a separate security org to understand the app or use case after the fact.

If you want to do $newthing your product dev management needs to involve security, finance, compliance, legal, etc. That’s their job. Developers don’t get to ignore all the normal business constraints the real world offers.

Building within constraints is what engineering is all about.