Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We indexed the Delve audit leak: 533 reports, 455 companies, 99.8% identical

trustcompliance.xyz

Post reply on HN
Print view View original

31–40 of 83 posts

Re: We indexed the Delve audit leak: 533 reports, 455 companies, 99.8% identical

#31

Post by staticassertion »

tptacek wrote: post #25
staticassertion wrote: post #18

Earlier quoted context omitted.

It's complicated. In theory, SOC2 forces you to do some important stuff, like define your threat model and say "I can mitigate against the threats and prove that my mitigations are in place". The problem is always that the companies that care don't need it but are burdened with it while the companies that don't care will just checkbox their way through it. It sort of enforces a very baseline security posture, in theo…

You can get a long, long way without SOC2; virtually every prospective customer you run into that asks for a SOC2 will have an alternate on-ramp for vendors without it, and the ones that don't will sign a contingent PO on your Type I, which (again) you are guaranteed to get. The idea that SOC2 forces you to do important stuff gets it backwards; SOC2 documents your existing practice, and demands only extremely high-le…

> You can get a long, long way without SOC2;

Yes, that's true. I edited my post to be a bit clearer about this. When you need a SOC2 is going to depend a lot on your business. Lots of companies can make exceptions very easily. Type 1 is easy, I would highly recommend starting there pretty much no matter what since it'll be good practice before your SOC2.

> The idea that SOC2 forces you to do important stuff gets it backwards;

It's the goal behind SOC2. You're assuming a company has a security practice that informs the SOC2 but I think the idea is that companies have no security practice and the SOC2 is what forces them to sit down and build one. What you're describing is more like what happens when a company that actually cares about security goes through SOC2 - you take what you have, put it into a NIST format, and map minimal controls from your practices to the CCs. Most companies have nothing to start with.

Re: We indexed the Delve audit leak: 533 reports, 455 companies, 99.8% identical

#32

Post by bearjaws »

SOC2 has been in trouble for a while now. Completely gamified. I was managing an acquisition of a healthtech company and asked if they did an internal risk assessment as part of their audit. Nope.

SOC2 certified, has never actually put to paper "here's what we know we're doing wrong, here is how we plan to remediate it."

Re: We indexed the Delve audit leak: 533 reports, 455 companies, 99.8% identical

#34

Post by mikeocool »

fadijob wrote: post #2

We analyzed the leaked Delve audit reports and found some wild patterns: - The same auditor license number (PAC-FIRM-LIC-47383) appears in 487 out of 494 reports - Every Type II report has identical page numbers: Section 4 at page 30, tests at page 59, Section 5 at page 82 - 220+ "No exceptions noted" per report, across every single client - The system descriptions were copy-pasted from each company's marketing websi…

The no exceptions noted piece is kinda funny. Most SOC2 auditors at least put in the minimal effort of finding one person who didn’t do their cybersecurity training, so the report’s not total boiler plate.

Re: We indexed the Delve audit leak: 533 reports, 455 companies, 99.8% identical

#36

Post by jiveturkey »

ecshafer wrote: post #30
preinheimer wrote: post #23

Looking at our SOC 2 report (we don't use Delve, our auditor isn't on their list) I don't think this is quite the smoking gun it might look like if you're not reading SOC 2 reports for a living. There's a fair amount of boiler plate language in these reports, and a bunch of re-stating the SOC 2 controls. I'd expect two reports (same auditors, same platforms) to be nearly identical. If they're both using AWS, Github,…

This mirrors my thoughts. A page of boiler play text with some check boxes, with some checked vs unchecked is going to be 99.8% similar between companies as well. A lot of audits are very much forms with boiler plate and fill in the blank. There is no point rewriting everything from scratch.

boilerplate is one word. sorry for the nit, feel free to backpfeifengesicht

Re: We indexed the Delve audit leak: 533 reports, 455 companies, 99.8% identical

#37

Post by ecshafer »

jiveturkey wrote: post #36
ecshafer wrote: post #30

Earlier quoted context omitted.

This mirrors my thoughts. A page of boiler play text with some check boxes, with some checked vs unchecked is going to be 99.8% similar between companies as well. A lot of audits are very much forms with boiler plate and fill in the blank. There is no point rewriting everything from scratch.

boilerplate is one word. sorry for the nit, feel free to backpfeifengesicht

I don't think that is an important point.

Re: We indexed the Delve audit leak: 533 reports, 455 companies, 99.8% identical

#38

Post by viccis »

ppqqrr wrote: post #9

what do you expect? if you’re “automating” an audit, it already means you don’t care. the LLM is there to blur the calculus of responsibility, take the blame if someone cares enough to look. happy customers, until someone “delves” a little too deep (like you did) and ruins the slumber party.

This was my general reaction when it was determined that the "log review" portion of the PCI checklist (can't remember what level) could be satisfied by computer "review", and that newer PCI versions were moving towards preferring automated "review"

Re: We indexed the Delve audit leak: 533 reports, 455 companies, 99.8% identical

#39

Post by preinheimer »

adriand wrote: post #10

Is SOC 2 legit? I have this on my roadmap but now I’m wondering if it’s just security theatre?

We did SOC 2 a few years ago, I'm glad we did it.

In my mind getting a clean report required three kinds of work:

1. Work that actively improved our security posture. 2. Work that didn't change much, but made our security posture easier to understand. 3. Busy work.

I think for most companies all three kinds of work will be required, but you can also make decisions that will push the percentages around. SOC 2 required us to start doing an annual security table top exercise. You could sit down, run a scenario, run it as fast as you can, and come up with a few pre-determined "improvements" that would help if you actually had that problem in the future. Or you could sit down and really put work into it, and see what works well and what doesn't.

As an example in our last tabletop I "exfiltrated" some data from one of our servers, and challenged the team to figure out what I'd done. The easy way out would have been for someone to say "We'll look at the logs and figure it out", but instead I asked them to actually try and find it. We discovered that the sheer volume of logs for that system made them hard to work with. So we made some changes to make them easier to work with and repeated the exercise later.

It could have been busy work, but instead we got real value from it.

Re: We indexed the Delve audit leak: 533 reports, 455 companies, 99.8% identical

#40

Post by truetraveller »

tptacek wrote: post #15
adriand wrote: post #10

Is SOC 2 legit? I have this on my roadmap but now I’m wondering if it’s just security theatre?

That's a difficult question to answer. It shouldn't be, but it is. The reality is, SOC2 is a sales-enablement tool. You should: * Run a SOC2/compliance program that is entirely disjoint from your security practice. * Defer SOC2 until the work required to sell into customers demanding it (phone calls, questionnaires) exceeds the cost of obtaining SOC2. * Prepare for SOC2 by making simple best-practices engineering dec…

Since you know a lot about SOC: is SOC2 Type I (point in time) enough to close enterprise sales? Is it worth getting for a new startup (seems super simple)?
Post reply on HN