Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Copilot broke audit logs, but Microsoft won't tell customers

pistachioapp.com

Post reply on HN
Print view View original

241–250 of 324 posts

Re: Copilot broke audit logs, but Microsoft won't tell customers

#241

Post by keyringlight »

ocdtrekkie wrote: post #221
whizzter wrote: post #147

Earlier quoted context omitted.

Just because malware authors have used winget doesn't mean package managers are virus-infested by default since it's used to deliver plenty of MS's own tools, you just need to be restrictive (or do you remove apt-get from Debian decendent distros also?). 100% agreed on the Edge-front page showing up on server machines being nasty though, server deployments should always have an empty page as the default for browsers…

I really need to emphasize winget is way, way different than a Linux software repository. Debian's repository is carefully maintained and packages have to reach a level of notability for inclusion. Even the Microsoft Store uses overseas reviewers paid by Microsoft to review if store apps meet their guidelines. winget has none of that. winget is run by one Microsoft dude who when pressed about reviewing submissions ga…

What stands out to me is that winget has the appearance and is often perceived as a package manager, yet it's more of a CLI front end to an index, and that index seems to either point to the windows store or a URL to download a regular setup file which it'll run silently (adobe acrobat is the example that springs to mind).

Re: Copilot broke audit logs, but Microsoft won't tell customers

#242

Post by chrisweekly »

Faark wrote: post #135
catmanjan wrote: post #36

Earlier quoted context omitted.

As someone else mentioned the file isnt actually accessed by copilot, rather copilot is reading the pre-indexed contents of the file in a search engine... Really Microsoft should be auditing the search that copilot executes, its actually a bit misleading to be auditing the file as accessed when copilot has only read the indexed content of the file, I don't say I've visited a website when I've found a result of it in…

What's the solution then? Chain 2 AIs, the first one is fine tuned on / has RAG access to your content telling a second that actually produces content what files are relevant (and logged)? Or just a system prompt "log where all the info comes from"...

Someone please confirm my idea (or remedy my ignorance) about this rule of thumb:

Don't train a model on sensitive info, if there will ever be a need for authZ more granular than implied by access to that model. IOW, given a user's ability to interact w/ a model, assume that everything it was trained on is visible to that user.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#243

Post by thunky »

scott_w wrote: post #224
thunky wrote: post #222

Earlier quoted context omitted.

You only have to update the metadata, not do a full reindex.

You'd have to reindex the metadata (roles access), which may be substantial if you have a complex enough schema with enough users/roles.

> You'd have to reindex the metadata (roles access), which may be substantial if you have a complex enough schema with enough users/roles.

Right, but this compare this to the original proposal:

> A basic implementation will return the top, let's say 1000, documents and then do the more expensive access check on each of them

Using an index is much better than that.

And it should be possible to update the index without a substantial cost, since most of the 100000 documents likely aren't changing their role access very often. You only have to reindex a document's metadata when that changes.

This is also far less costly than updating the actual content index (the vector embeddings) when the document content changes, which you have to do regardless of your permissions model.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#244

Post by bongodongobob »

albert_e wrote: post #58
bongodongobob wrote: post #17

Earlier quoted context omitted.

This is it. M365 uses RAG on your enterprise data that you allow it to access. It's not actually accessing the files directly in the cases he provided. It's working as intended.

If this is indeed how copilot is archtected, then it needs clear documentation -- that it is a non-audited data store. But how then did MS "fix" this bug? Did they stop pre-ingesting, indexing, and caching the content? I doubt that. Pushing (defaulting) organizations to feed all their data to Copilot and then not providing an audit trail of data access on that replica data store -- feels like a fundamental gap that s…

How would you audit that?

Re: Copilot broke audit logs, but Microsoft won't tell customers

#245

Post by bongodongobob »

sailfast wrote: post #52
crooked-v wrote: post #32

Earlier quoted context omitted.

If that's the case, then as noted in the article, the 'as intended' is probably violating liability requirements around various things.

Correct. It is precisely that a user can ask about someone’s medical history (or whatever else) and not be reported that would be in violation of any heavily audited system. LLM Summaries break the compliance.

You allow what it can and can't see. If you include PII and medical records, that's your fault, not MS's.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#246

Post by thayne »

andrewaylett wrote: post #237
Drakim wrote: post #194

Earlier quoted context omitted.

If you have public documents, you can obviously let a public search engine index them and show previews. All is good. If you have private documents, you can't let a public search engine index and show previews of those private documents. Even if you add an authentication wall for normal users if they try to open the document directly. They could still see part of the document in google's preview. My explanation sound…

You say that, but it happens — "Experts Exchange", for example, certainly used to try to hide the answers from users who hadn't paid while encouraging search engines to index them.

That's not quite the same. Experts Exchange wanted the content publicly searchable, and explicitly allowed search engines to index it. In this case, many customers probably aren't aware that there is a separate search index that contains much of the data in their private documents that may be searchable and accessible by entities that otherwise shouldn't have access.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#247

Post by ocdtrekkie »

keyringlight wrote: post #241
ocdtrekkie wrote: post #221

Earlier quoted context omitted.

I really need to emphasize winget is way, way different than a Linux software repository. Debian's repository is carefully maintained and packages have to reach a level of notability for inclusion. Even the Microsoft Store uses overseas reviewers paid by Microsoft to review if store apps meet their guidelines. winget has none of that. winget is run by one Microsoft dude who when pressed about reviewing submissions ga…

What stands out to me is that winget has the appearance and is often perceived as a package manager, yet it's more of a CLI front end to an index, and that index seems to either point to the windows store or a URL to download a regular setup file which it'll run silently (adobe acrobat is the example that springs to mind).

Exactly! It's like curl|bash for Windows but where you don't even see the URL.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#248

Post by shawnz »

lixtra wrote: post #183
redwood wrote: post #175

Earlier quoted context omitted.

But you process that list in a trusted audited app tier not in the client environment

A naive approach could still leak information through side channels. E.g. if you search regularly for foobar, the answer might suddenly get slower if foobar appears more in the document base. Depending on the context it could be relevant.

That's nothing specific to LLM-enhanced search features though, right? Any search feature will have that side channel risk

Re: Copilot broke audit logs, but Microsoft won't tell customers

#249

Post by perlgeek »

doomslice wrote: post #198
krisoft wrote: post #165

Earlier quoted context omitted.

> This means Vector databases, Search Indexes or fancy "AI Search Databases" would be required on a per user basis or track the access rights along with the content, which is infeasible and does not scale. I don't understand why you think tracking user access rights would be infeasible and would not scale. There is a query. You search for matching documents in your vector database / index. Once you have found the pot…

Let's say you have 100000 documents in your index that match your query but only 10 of them the user has access to: A basic implementation will return the top, let's say 1000, documents and then do the more expensive access check on each of them. Most of the time, you've now eliminated all of your search results. Your search must be access aware to do a reasonable job of pre-filtering the content to documents the use…

"Fun" Fact: ServiceNow simply passes this problem on to its users.

I've seen a list of what was supposed to be 20 items of something, it only showed 2, plus a comment "18 results were omitted to insufficient permissions".

(Servicenow has at least three different ways to do permissions, I don't know if this applies to all of them).

Re: Copilot broke audit logs, but Microsoft won't tell customers

#250

Post by admissionsguy »

lupusreal wrote: post #166
carschno wrote: post #161

Earlier quoted context omitted.

> I am having a really hard time communicating this problem to executives When you hit such a wall, you might not be failing to communicate, nor them failing to understand. In reality, said executives have probably chosen to ignore the issue, but also don't want to take accountability for the eventual leaks. So "not understanding" is the easiest way to blame the engineers later.

It doesn't even need to be blaming the engineers in this case, they can blame "the AI" and most people will accept that and let whatever incident happened slide. If somebody questions the wisdom of putting AI in such a position, they can be dismissed as not appreciating new technology (even though their concern is valid.)

worse, they can be dismissed as an abstract ”ai is dangerous” and used to justify funnelling money to the various ai safety charlatans
Post reply on HN