Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Copilot broke audit logs, but Microsoft won't tell customers

pistachioapp.com

Post reply on HN
Print view View original

221–230 of 324 posts

Re: Copilot broke audit logs, but Microsoft won't tell customers

#221

Post by ocdtrekkie »

whizzter wrote: post #147
ocdtrekkie wrote: post #43

Earlier quoted context omitted.

I would say "insecure by default". I talked to some Microsoft folks around the Windows Server 2025 launch, where they claimed they would be breaking more compatibility in the name of their Secure Future Initiative. But Server 2025 will load malicious ads on the Edge start screen[1] if you need to access a web interface of an internal thing from your domain controller, and they gleefully announced including winget, a…

Just because malware authors have used winget doesn't mean package managers are virus-infested by default since it's used to deliver plenty of MS's own tools, you just need to be restrictive (or do you remove apt-get from Debian decendent distros also?). 100% agreed on the Edge-front page showing up on server machines being nasty though, server deployments should always have an empty page as the default for browsers…

I really need to emphasize winget is way, way different than a Linux software repository. Debian's repository is carefully maintained and packages have to reach a level of notability for inclusion. Even the Microsoft Store uses overseas reviewers paid by Microsoft to review if store apps meet their guidelines.

winget has none of that. winget is run by one Microsoft dude who when pressed about reviewing submissions gave some random GitHub users who have not been vetted moderator powers. There is no criteria for inclusion, if you can pack it and get it by the automated scanner, it ships. And anyone can submit changes to any winget package: They built a feature to let a developer restrict a package be only updated by a trusted user but never implemented it. (Doing so requires a "business process" but being a one-man sideshow that winget is, setting that up is beyond Microsoft's ability.)

winget is a complete joke that no professional could stand for if they understand how amateur hour it is, and the fact it is now baked into every Windows install is absolutely embarrassing. But I bet shipping it got that Microsoft engineer a promotion!

Re: Copilot broke audit logs, but Microsoft won't tell customers

#222

Post by thunky »

beaviskhan wrote: post #217
thunky wrote: post #205

Earlier quoted context omitted.

> Let's say you have 100000 documents in your index that match your query If the docs were indexed by groups/roles and you had some form of RBAC then this wouldn't happen.

If you take this approach, you have to reindex when groups/roles changes - not always a feasible choice

You only have to update the metadata, not do a full reindex.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#223

Post by scott_w »

krisoft wrote: post #165
planb wrote: post #158

I am assigned to develop a company internal chatbot that accesses confidential documents and I am having a really hard time communicating this problem to executives: As long as not ALL the data the agent hat access too is checked against the rights of the current user placing the request, there WILL be ways to leak data. This means Vector databases, Search Indexes or fancy "AI Search Databases" would be required on a…

> This means Vector databases, Search Indexes or fancy "AI Search Databases" would be required on a per user basis or track the access rights along with the content, which is infeasible and does not scale. I don't understand why you think tracking user access rights would be infeasible and would not scale. There is a query. You search for matching documents in your vector database / index. Once you have found the pot…

> I don't understand why you think tracking user access rights would be infeasible and would not scale.

Allow me to try to inject my understanding of how these agents work vs regular applications.

A regular SaaS will have an API endpoint that has permissions attached. Before the endpoint processes anything, the user making the request has their permissions checked against the endpoint itself. Once this request succeeds, anything that endpoint collects is considered "ok" ship to the user.

AI Agents, instead, directly access the database, completely bypassing this layer. That means you need to embed the access permissions into the individual rows, rather than at the URL/API layer. It's much more complex as a result.

For your bank analogy: they actually work in a similar way to how I described above. A temporary access is granted to the resources but, once it's granted, any data included in those screens is assumed to be ok. They won't see something like a blank box somewhere because there's info they're not supposed to see.

DISCLAIMER: I'm making an assumption on how these AI Agents work, I could be wrong.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#224

Post by scott_w »

thunky wrote: post #222
beaviskhan wrote: post #217

Earlier quoted context omitted.

If you take this approach, you have to reindex when groups/roles changes - not always a feasible choice

You only have to update the metadata, not do a full reindex.

You'd have to reindex the metadata (roles access), which may be substantial if you have a complex enough schema with enough users/roles.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#225

Post by jsnell »

Isn0gud wrote: post #220
DannyBee wrote: post #207

Earlier quoted context omitted.

Yes. But this is still an incredibly well known and solved problem. As an example - google's internal structured search engines did this decades ago at scale.

Which solutions are you referring to? With access that is highly diverse and changing, this is still an unsolved problem to my knowledge.

Probably Google Zanzibar (and the various non-Google systems that were created as a result of the paper describing Zanzibar).

Re: Copilot broke audit logs, but Microsoft won't tell customers

#226

Post by giamma »

planb wrote: post #158

I am assigned to develop a company internal chatbot that accesses confidential documents and I am having a really hard time communicating this problem to executives: As long as not ALL the data the agent hat access too is checked against the rights of the current user placing the request, there WILL be ways to leak data. This means Vector databases, Search Indexes or fancy "AI Search Databases" would be required on a…

I believe most vector databases allow you to annotate vectors with additional metadata. Why not simply add as metadata the list of principals (roles/groups) who have access to the information (e.g. HR, executives) ? Then when a user makes a request to the chatbot, you expand the user identity to his/her principals (e.g. HR) and use those as implicit filtering criteria for finding the closest vectors in the database.

In this way you exclude up-front the documents that the current user cannot see.

Of course, this requires you to update the vector metadata any time the permissions change at the document level (e.g. a given document originally visible only to HR is now also visibile to executives -> you need to add the principal executives to the metadata of the vector resulting from the document in your vector database)

Re: Copilot broke audit logs, but Microsoft won't tell customers

#227

Post by GloriousMEEPT »

scott_w wrote: post #223
krisoft wrote: post #165

Earlier quoted context omitted.

> This means Vector databases, Search Indexes or fancy "AI Search Databases" would be required on a per user basis or track the access rights along with the content, which is infeasible and does not scale. I don't understand why you think tracking user access rights would be infeasible and would not scale. There is a query. You search for matching documents in your vector database / index. Once you have found the pot…

> I don't understand why you think tracking user access rights would be infeasible and would not scale. Allow me to try to inject my understanding of how these agents work vs regular applications. A regular SaaS will have an API endpoint that has permissions attached. Before the endpoint processes anything, the user making the request has their permissions checked against the endpoint itself. Once this request succee…

The solution to this problem is to develop your agents to use delegation and exchange tokens for access to other services using an on-behalf-of flow. Agents are never operating under their own identity, but as the user.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#228

Post by pc86 »

scott_w wrote: post #223
krisoft wrote: post #165

Earlier quoted context omitted.

> This means Vector databases, Search Indexes or fancy "AI Search Databases" would be required on a per user basis or track the access rights along with the content, which is infeasible and does not scale. I don't understand why you think tracking user access rights would be infeasible and would not scale. There is a query. You search for matching documents in your vector database / index. Once you have found the pot…

> I don't understand why you think tracking user access rights would be infeasible and would not scale. Allow me to try to inject my understanding of how these agents work vs regular applications. A regular SaaS will have an API endpoint that has permissions attached. Before the endpoint processes anything, the user making the request has their permissions checked against the endpoint itself. Once this request succee…

> I'm making an assumption on how these AI Agents work, I could be wrong.

I don't understand the desire - borderline need - of folks on HN to just make stuff up. That is likely why you're being downvoted. I know we all love to do stuff "frOM fIRsT PRiNcIPlEs" around here but "let me just imagine how I think AI agents work then pass that off as truth" is taking it a bit far IMO.

This is the human equivalent of an AI hallucination. You are just making stuff up, passing it off as truth ("injecting your understanding"), then adding a one-line throwaway "this might be completely wrong lol" at the end.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#229

Post by awirth »

planb wrote: post #158

I am assigned to develop a company internal chatbot that accesses confidential documents and I am having a really hard time communicating this problem to executives: As long as not ALL the data the agent hat access too is checked against the rights of the current user placing the request, there WILL be ways to leak data. This means Vector databases, Search Indexes or fancy "AI Search Databases" would be required on a…

What you're describing is a specific case of a confused deputy problem: https://en.wikipedia.org/wiki/Confused_deputy_problem

This is captured in the OWASP LLM Top 10 "LLM02:2025 Sensitive Information Disclosure" risk: https://genai.owasp.org/llmrisk/llm022025-sensitive-informat... although in some cases the "LLM06:2025 Excessive Agency" risk is also applicable.

I believe that some enterprise RAG solutions create a per user index to solve this problem when there are lots of complex ACLs involved. How vendors manage this problem is an important question to ask when analyzing RAG solutions.

At my current company at least we call this "権限混同" in Japanese - Literally "authorization confusion" which I think is a more fun name

Re: Copilot broke audit logs, but Microsoft won't tell customers

#230

Post by scott_w »

pc86 wrote: post #228
scott_w wrote: post #223

Earlier quoted context omitted.

> I don't understand why you think tracking user access rights would be infeasible and would not scale. Allow me to try to inject my understanding of how these agents work vs regular applications. A regular SaaS will have an API endpoint that has permissions attached. Before the endpoint processes anything, the user making the request has their permissions checked against the endpoint itself. Once this request succee…

> I'm making an assumption on how these AI Agents work, I could be wrong. I don't understand the desire - borderline need - of folks on HN to just make stuff up. That is likely why you're being downvoted. I know we all love to do stuff "frOM fIRsT PRiNcIPlEs" around here but "let me just imagine how I think AI agents work then pass that off as truth" is taking it a bit far IMO. This is the human equivalent of an AI h…

And yet if we didn't do this, HN would be almost completely silent because 99% of commenters have a clue what they're talking about most of the time and nobody would ever have a chance to learn.
Post reply on HN