First American Financial Corp. Leaked Hundreds of Millions of Insurance Records • Hacker News Board

Re: First American Financial Corp. Leaked Hundreds of Millions of Insurance Records

#51

Post by tptacek » Sat, May 25, 2019, 12:02 AM UTC

eganist wrote: post #50

tptacek wrote: post #47

Earlier quoted context omitted.

There are no effective static source code security analyzers. Static analyzers aren't a bad thing to add to a CI pipeline, because why not, but anyone depending on static analysis is playing to lose. This is absolutely not the kind of vulnerability that pentests tend to miss; rather, they're the first thing pentesters check for. You can miss bugs like this when they're in obscure backend features and your client or t…

> This is absolutely not the kind of vulnerability that pentests tend to miss You're right; they don't. Which is why I called out automated dynamic analysis. I.e. the web scanning tools which you subsequently mentioned: > Web scanning tools will miss findings like this. --- > As for code review: a competently constructed application shouldn't be relying on developers to catch every possible instance where numeric ids…

I'll amend my previous comment to say that I only dispute much of the analysis, not "the whole" analysis.

A sibling comment makes the obvious point that no pre-auth endpoint should be touching this kind of data to begin with, which is another layer of "stuff you can just regex for".

Re: First American Financial Corp. Leaked Hundreds of Millions of Insurance Records

#52

Post by mattmanser » Sat, May 25, 2019, 12:04 AM UTC

dvlsg wrote: post #48

mattmanser wrote: post #44

Earlier quoted context omitted.

I once made an app not using sequential integers as object ids, as you suggest. It was an absolute nightmare. Maintenance was a nightmare, you're constantly having to generate or replicate these things that add an extra layer of complexity to everything, and almost always unnecessarily. It's also extremely bad for db performance, causes massive page fragmentation, indexes become useless almost straight after rebuildi…

You can generate uuids that play nicer with database storage / indexing. NEWSEQUENTIALID() in MSSQL, for example. The keys will be easier to guess again, but if all you have to do is guess a primary key to get access to the underlying data, something else isn't right anyways.

I know, but as they're easier to guess, what's the point? Might as well just go back to ints.

Re: First American Financial Corp. Leaked Hundreds of Millions of Insurance Records

#53

Post by eganist » Sat, May 25, 2019, 12:04 AM UTC

tptacek wrote: post #51

eganist wrote: post #50

Earlier quoted context omitted.

> This is absolutely not the kind of vulnerability that pentests tend to miss You're right; they don't. Which is why I called out automated dynamic analysis. I.e. the web scanning tools which you subsequently mentioned: > Web scanning tools will miss findings like this. --- > As for code review: a competently constructed application shouldn't be relying on developers to catch every possible instance where numeric ids…

I'll amend my previous comment to say that I only dispute much of the analysis, not "the whole" analysis. A sibling comment makes the obvious point that no pre-auth endpoint should be touching this kind of data to begin with, which is another layer of "stuff you can just regex for".

> I'll amend my previous comment to say that I only dispute much of the analysis, not "the whole" analysis.

That's fine, but I'd appreciate it if you just read the entire analysis next time. It shows that you respect the time people invest into constructing and presenting guidance, even if you don't necessarily respect the guidance itself.

---

Editing mine to match your edit... as if to make my point about reading the analysis in its entirety:

> A sibling comment makes the obvious point that no pre-auth endpoint should be touching this kind of data to begin with, which is another layer of "stuff you can just regex for".

Correct, something which I'd also stated:

> Checks including linting for specific privacy defects (direct object referencing using sensitive data or iterative identifiers as opposed to hashes/guids/etc) can help with catching them during development, and as you might've guessed, such checks tend to be custom for a given environment rather than out of the box.

Re: First American Financial Corp. Leaked Hundreds of Millions of Insurance Records

#54

Post by birken » Sat, May 25, 2019, 12:05 AM UTC

scarface74 wrote: post #49

throwawaymath wrote: post #22

Yet another security vulnerability caused by: 1. Using sequentially incremented integer sequences as object IDs, and 2. Failing to protect sensitive data using some kind of authentication and authorization check. This is becoming a trend with data breaches. Several of Krebs' other reports on behalf of security researchers were originally identified by (trivially) walking across object IDs on public URLs. My cynical t…

There is nothing wrong with using sequential ids in and of themselves. The typical web app has the concept of a validated user session per request. How hard is it really to Select ... From Documents where documentid = ? and userid = ? So even if the user does a GET /Document/{id+1} No documents would be returned. Every web framework that I am aware of let’s you add one piece of middleware that validates a user sessio…

Maybe not "wrong", but there are some very obvious downsides to exposing sequential IDs vs a randomized token:

- It exposes the count you have of a particular item

- It exposes your growth rate of those items

- If a developer accidentally breaks your authentication (or somebody hacks it), it becomes trivially easy to download all your items very quickly

And it isn't like using a randomized token is hard. In the most common implementation, it is just one additional column that gets filled with a random string and an index on the column.

Re: First American Financial Corp. Leaked Hundreds of Millions of Insurance Records

#55

Post by tptacek » Sat, May 25, 2019, 12:07 AM UTC

eganist wrote: post #53

tptacek wrote: post #51

Earlier quoted context omitted.

I'll amend my previous comment to say that I only dispute much of the analysis, not "the whole" analysis. A sibling comment makes the obvious point that no pre-auth endpoint should be touching this kind of data to begin with, which is another layer of "stuff you can just regex for".

> I'll amend my previous comment to say that I only dispute much of the analysis, not "the whole" analysis. That's fine, but I'd appreciate it if you just read the entire analysis next time. It shows that you respect the time people invest into constructing and presenting guidance, even if you don't necessarily respect the guidance itself. --- Editing mine to match your edit... as if to make my point about reading th…

Yeah, no, I think you got this wrong, but more than that I was motivated to comment by the implication you made that these were "easy to miss" vulnerabilities because bullshit security tools that don't work miss them. I don't so much care whether you're right or wrong, but I do want to take every opportunity I can get to disabuse people about the effectiveness of scanners.

Re: First American Financial Corp. Leaked Hundreds of Millions of Insurance Records

#56

Post by eganist » Sat, May 25, 2019, 12:07 AM UTC

tptacek wrote: post #55

eganist wrote: post #53

Earlier quoted context omitted.

> I'll amend my previous comment to say that I only dispute much of the analysis, not "the whole" analysis. That's fine, but I'd appreciate it if you just read the entire analysis next time. It shows that you respect the time people invest into constructing and presenting guidance, even if you don't necessarily respect the guidance itself. --- Editing mine to match your edit... as if to make my point about reading th…

Yeah, no, I think you got this wrong, but more than that I was motivated to comment by the implication you made that these were "easy to miss" vulnerabilities because bullshit security tools that don't work miss them. I don't so much care whether you're right or wrong, but I do want to take every opportunity I can get to disabuse people about the effectiveness of scanners.

> "easy to miss" vulnerabilities because bullshit security tools that don't work miss them

> I do want to take every opportunity I can get to disabuse people about the effectiveness of scanners.

This entire exchange is frustrating because it's exactly what I said in my root comment:

> these are also some of the easiest vulnerabilities to miss even with out-of-the-box static analysis (code scanning and data analysis), automated dynamic analysis (pentests [edit to clarify for tptacek: automated pentests]), and a basic code review process.

[...]

> Checks including linting for specific privacy defects (direct object referencing using sensitive data or iterative identifiers as opposed to hashes/guids/etc) can help with catching them during development, and as you might've guessed, such checks tend to be custom for a given environment rather than out of the box.

---

I'm going to step away from my keyboard a bit; please forgive me.

Re: First American Financial Corp. Leaked Hundreds of Millions of Insurance Records

#57

Post by prophesi » Sat, May 25, 2019, 12:08 AM UTC

dvlsg wrote: post #48

mattmanser wrote: post #44

Earlier quoted context omitted.

I once made an app not using sequential integers as object ids, as you suggest. It was an absolute nightmare. Maintenance was a nightmare, you're constantly having to generate or replicate these things that add an extra layer of complexity to everything, and almost always unnecessarily. It's also extremely bad for db performance, causes massive page fragmentation, indexes become useless almost straight after rebuildi…

You can generate uuids that play nicer with database storage / indexing. NEWSEQUENTIALID() in MSSQL, for example. The keys will be easier to guess again, but if all you have to do is guess a primary key to get access to the underlying data, something else isn't right anyways.

I think this gets to the crux of the issue.

It's not about using hard-to-guess UUIDs[0], but restricting access to the underlying data[1].

[0] https://en.m.wikipedia.org/wiki/Security_through_obscurity

[1] https://en.m.wikipedia.org/wiki/Access_control

Re: First American Financial Corp. Leaked Hundreds of Millions of Insurance Records

#58

Post by sverhagen » Sat, May 25, 2019, 12:14 AM UTC

mattmanser wrote: post #44

throwawaymath wrote: post #22

Yet another security vulnerability caused by: 1. Using sequentially incremented integer sequences as object IDs, and 2. Failing to protect sensitive data using some kind of authentication and authorization check. This is becoming a trend with data breaches. Several of Krebs' other reports on behalf of security researchers were originally identified by (trivially) walking across object IDs on public URLs. My cynical t…

I once made an app not using sequential integers as object ids, as you suggest. It was an absolute nightmare. Maintenance was a nightmare, you're constantly having to generate or replicate these things that add an extra layer of complexity to everything, and almost always unnecessarily. It's also extremely bad for db performance, causes massive page fragmentation, indexes become useless almost straight after rebuildi…

I also don't think using UUIDs as a security (by obscurity) strategy is valid. But there are other reasons someone may choose to use UUIDs. For instance, it's convenient to generate identifiers in a decentralized manner. I want to counter your one bad experience with my (equally anecdotal) many-multiple good experiences. Databases do just fine with UUIDs. Though we may be working on different kinds of systems, and optimizing for different things. I don't frown upon using integers (well, longs) for identifiers, but I personally prefer UUIDs.

Re: First American Financial Corp. Leaked Hundreds of Millions of Insurance Records

#59

Post by scarface74 » Sat, May 25, 2019, 12:14 AM UTC

birken wrote: post #54

scarface74 wrote: post #49

Earlier quoted context omitted.

There is nothing wrong with using sequential ids in and of themselves. The typical web app has the concept of a validated user session per request. How hard is it really to Select ... From Documents where documentid = ? and userid = ? So even if the user does a GET /Document/{id+1} No documents would be returned. Every web framework that I am aware of let’s you add one piece of middleware that validates a user sessio…

Maybe not "wrong", but there are some very obvious downsides to exposing sequential IDs vs a randomized token: - It exposes the count you have of a particular item - It exposes your growth rate of those items - If a developer accidentally breaks your authentication (or somebody hacks it), it becomes trivially easy to download all your items very quickly And it isn't like using a randomized token is hard. In the most…

In that simple scenario. What are some ways that a hacker could break your front end API to allow it to serve requests for multiple users without having access to multiple account logins? I understand that they could possibly get access to your database but that’s a different threat.

If they could somehow change your code, all hope is already lost.

But I do agree with it does allow someone to determine rate of growth which would be valuable more from a business intelligence side than a privacy violation.

The larger issue is that a developer forgets to add the “and userid = ?”

I guess the work around for that is to have a database that ties user authentication to records in the table/object store directly like DynamoDB or S3.

Re: First American Financial Corp. Leaked Hundreds of Millions of Insurance Records

#60

Post by throwawaymath » Sat, May 25, 2019, 12:15 AM UTC

scarface74 wrote: post #49

throwawaymath wrote: post #22

Yet another security vulnerability caused by: 1. Using sequentially incremented integer sequences as object IDs, and 2. Failing to protect sensitive data using some kind of authentication and authorization check. This is becoming a trend with data breaches. Several of Krebs' other reports on behalf of security researchers were originally identified by (trivially) walking across object IDs on public URLs. My cynical t…

There is nothing wrong with using sequential ids in and of themselves. The typical web app has the concept of a validated user session per request. How hard is it really to Select ... From Documents where documentid = ? and userid = ? So even if the user does a GET /Document/{id+1} No documents would be returned. Every web framework that I am aware of let’s you add one piece of middleware that validates a user sessio…

No, nothing wrong with it intrinsically. But if UUIDs were used instead, the lack of authentication or authorization checks wouldn't be as catastrophic. That would be somewhat comparable to having a reset password token which doesn't expire. Still bad, but not as bad.

The other commenter's point about leaking information is also correct. In the finance industry one of the basic tricks to obtaining alternative data is to scrape it from private APIs which expose sequential IDs corresponding to a source of revenue. For example, a publicly traded car company might have its revenue extrapolated from an open API which sequentially increments an ID every time a vehicle is sold. Research groups will reverse engineer mobile apps from companies with only one or two dimensions of revenue, find the private API endpoints (reversing request signing as needed), and then look for object IDs which can be thrown into a timeseries on a quarterly basis.

Generally speaking the risk and compliance department of a hedge fund disallows this kind of data if it's gathered from an actual security vulnerability (e.g. leaks PII). It needs to be "only" a neutral information side channel without sensitive data, so that doesn't really apply in this specific scenario. But it does apply for people considering using integer IDs for user-facing APIs.