Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Copilot broke audit logs, but Microsoft won't tell customers

pistachioapp.com

Post reply on HN
Print view View original

191–200 of 324 posts

Re: Copilot broke audit logs, but Microsoft won't tell customers

#191

Post by jeltz »

lupusreal wrote: post #166
carschno wrote: post #161

Earlier quoted context omitted.

> I am having a really hard time communicating this problem to executives When you hit such a wall, you might not be failing to communicate, nor them failing to understand. In reality, said executives have probably chosen to ignore the issue, but also don't want to take accountability for the eventual leaks. So "not understanding" is the easiest way to blame the engineers later.

It doesn't even need to be blaming the engineers in this case, they can blame "the AI" and most people will accept that and let whatever incident happened slide. If somebody questions the wisdom of putting AI in such a position, they can be dismissed as not appreciating new technology (even though their concern is valid.)

Yeah, it is usually not about blaming the engineers in my experience. It is about so they can make a descion they want to make without having to think too hard or take any accountability. If nobody knew at the time it was bad everyone can just act surprised and call it an accident and just go on with their lives making similar uninformed descisions.

In their dream world the engineers would not know about it either.

Edit: Maybe we should call this style vibe management. :D

Re: Copilot broke audit logs, but Microsoft won't tell customers

#192

Post by falcor84 »

Drakim wrote: post #184
eschneider wrote: post #172

Earlier quoted context omitted.

Oh, so there's a complete copy (or something that can be reassembled into a copy) completely OUTSIDE of audit controls. That's so much worse. :0

It's roughly the same problem as letting a search engine build indexes (with previews!) of sites without authentication. It's kinda crazy that things were allowed to go this far with such a fundamental flaw.

I'm unclear on what the "flaw" is - isn't this precisely the "feature" that search engines provide to both sides and that site owners put a ton of SEO effort into optimizing?

Re: Copilot broke audit logs, but Microsoft won't tell customers

#193

Post by ezconnect »

NameForComment wrote: post #159
degamad wrote: post #67

One thing that's not clear in the write-up here: *which* audit log is he talking about? Sharepoint file accesses? Copilot actions? Purview? Something else?

In one of the footnotes: "The audit log will not show that the user accessed the file as a normal SharePointFileOperation FileAccessed event, but rather as a CopilotInteraction record. That’s intended, and in my opinion correct. It would be weird to make it as if the user directly accessed the file when they only did so via Copilot."

So it is not logged as Copilot and the user using Copilot? I guess the chat is logged so it's another way to trace accountability.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#194

Post by Drakim »

falcor84 wrote: post #192
Drakim wrote: post #184

Earlier quoted context omitted.

It's roughly the same problem as letting a search engine build indexes (with previews!) of sites without authentication. It's kinda crazy that things were allowed to go this far with such a fundamental flaw.

I'm unclear on what the "flaw" is - isn't this precisely the "feature" that search engines provide to both sides and that site owners put a ton of SEO effort into optimizing?

If you have public documents, you can obviously let a public search engine index them and show previews. All is good.

If you have private documents, you can't let a public search engine index and show previews of those private documents. Even if you add an authentication wall for normal users if they try to open the document directly. They could still see part of the document in google's preview.

My explanation sounds silly because surely nobody is that dumb, but this is exactly what they have done. They gave access to ALL documents, both public and private, to an AI, and then got surprised when the AI leaked some private document details. They thought they were safe because users would be faced with an authentication wall if they tried to open the document directly. But that doesn't help if copilot simply tells you all the secret in it's own words.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#195

Post by p3rls »

carschno wrote: post #161
planb wrote: post #158

I am assigned to develop a company internal chatbot that accesses confidential documents and I am having a really hard time communicating this problem to executives: As long as not ALL the data the agent hat access too is checked against the rights of the current user placing the request, there WILL be ways to leak data. This means Vector databases, Search Indexes or fancy "AI Search Databases" would be required on a…

> I am having a really hard time communicating this problem to executives When you hit such a wall, you might not be failing to communicate, nor them failing to understand. In reality, said executives have probably chosen to ignore the issue, but also don't want to take accountability for the eventual leaks. So "not understanding" is the easiest way to blame the engineers later.

In this case it looks like the executives should fire the OP and hire the 2nd poster who came up with a solution. C'mon lazy executives.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#196

Post by tatersolid »

gpm wrote: post #4
nzeid wrote: post #3

Hard to count the number of things that can go wrong by relying directly on an LLM to manage audit/activity/etc. logs. What was their bug fix? Shadow prompts?

I'd hope that if a tool the LLM uses reveals any part of the file to the LLM it counts as a read by every user who sees any part of the output that occurred after that revelation was added to the context.

How would you handle “company name” or other common phrases in search? Log 1M documents every time that phrase appeared in a copilot response for any user?

Re: Copilot broke audit logs, but Microsoft won't tell customers

#197

Post by mpeg »

adtac wrote: post #179
whizzter wrote: post #150

Earlier quoted context omitted.

Even better, _ALL USEFUL_ AI retrival systems are insecure by design, because all those RAG vectors that sells vector-databases? That's basically your documents lossily encoded.

>That's basically your documents lossily encoded. Vector embeddings are lossy encodings of documents roughly in the same way a SHA256 hash is a lossy encoding. It's virtually impossible to reverse the embedding vector to recover the original document. Note: when vectors are combined with other components for search and retrieval, it's trivial to end up with a horribly insecure system, but just vector embeddings are u…

I wouldn't say those two are equivalent. A cryptographic hash requires the exact full document to be available to "recover it" from the hash. With a vector embedding you can extract information related to the document from the embedding alone as long as you know (or can guess) what embedding model was used. You won't be able to reconstruct the document but you will be able to infer some meaning from the vector alone

Re: Copilot broke audit logs, but Microsoft won't tell customers

#198

Post by doomslice »

krisoft wrote: post #165
planb wrote: post #158

I am assigned to develop a company internal chatbot that accesses confidential documents and I am having a really hard time communicating this problem to executives: As long as not ALL the data the agent hat access too is checked against the rights of the current user placing the request, there WILL be ways to leak data. This means Vector databases, Search Indexes or fancy "AI Search Databases" would be required on a…

> This means Vector databases, Search Indexes or fancy "AI Search Databases" would be required on a per user basis or track the access rights along with the content, which is infeasible and does not scale. I don't understand why you think tracking user access rights would be infeasible and would not scale. There is a query. You search for matching documents in your vector database / index. Once you have found the pot…

Let's say you have 100000 documents in your index that match your query but only 10 of them the user has access to:

A basic implementation will return the top, let's say 1000, documents and then do the more expensive access check on each of them. Most of the time, you've now eliminated all of your search results.

Your search must be access aware to do a reasonable job of pre-filtering the content to documents the user has access to, at which point you then can apply post-filtering with the "100% sure" access check.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#199

Post by Scubabear68 »

Drakim wrote: post #184
eschneider wrote: post #172

Earlier quoted context omitted.

Oh, so there's a complete copy (or something that can be reassembled into a copy) completely OUTSIDE of audit controls. That's so much worse. :0

It's roughly the same problem as letting a search engine build indexes (with previews!) of sites without authentication. It's kinda crazy that things were allowed to go this far with such a fundamental flaw.

Yep. Many years ago I worked at one of the top brokerage houses in the United States, they had a phenomenal Google search engine in house that made it really easy to navigate the whole company and find information.

Then someone discovered production passwords on a site that was supposed to be secured but wasn’t.

Found such things in several places.

The solution was to make searching work only if you opted-in your website.

After that internal search was effectively broken and useless.

All because a few actors did not think about or care about proper authentication and authorization controls.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#200

Post by smarx007 »

planb wrote: post #158

I am assigned to develop a company internal chatbot that accesses confidential documents and I am having a really hard time communicating this problem to executives: As long as not ALL the data the agent hat access too is checked against the rights of the current user placing the request, there WILL be ways to leak data. This means Vector databases, Search Indexes or fancy "AI Search Databases" would be required on a…

Two points/questions:

1. Why is tracking access rights "on a per user basis or [...] along with the content" is not feasible? A few mentions: Google Zanzibar (+Ory Keto as OSS impl) - makes authz for content othoronal to apps (i.e. possible to have it in one place, s.t. both Jira and a Jira MCP server can use the same API to check authz - possible to have a 100% faithful authz logic in the MCP server), Eclipse Biscuit (as far as I understand, this is a Dassault's attempt to make JWTs on steroids by adding Datalog and attenuation to the tokens, going in the Zanzibar direction but not requiring a network call for every single check), Apache Accumulo (DBMS with a cell-level security) and others. The way I see it, the tech is there but so far, not enough attention has been put on the problem of a high-fidelity authz throughout the enterprise on a granular level.

2. What is the scale needed? Enterprises with more than 10000 employees are quite rare, many individual internal IT systems even in large companies have less than 100 regular users. At these levels of scale, a lot more approaches are feasible that would not be considered possible at Google scale (i.e. more expensive algorithms w.r.t. big-O are viable).

Post reply on HN