Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Copilot broke audit logs, but Microsoft won't tell customers

pistachioapp.com

Post reply on HN
Print view View original

321–324 of 324 posts

Re: Copilot broke audit logs, but Microsoft won't tell customers

#321

Post by justincormack »

planb wrote: post #158

I am assigned to develop a company internal chatbot that accesses confidential documents and I am having a really hard time communicating this problem to executives: As long as not ALL the data the agent hat access too is checked against the rights of the current user placing the request, there WILL be ways to leak data. This means Vector databases, Search Indexes or fancy "AI Search Databases" would be required on a…

I listened to a podcast from someone at Glean about this, they do that, but she pointed out it is not enough as permissions are wrong often, and a good AI search can find you a bunch of documents with salary information that you have permission to see bit should not.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#322

Post by jsshapiro »

planb wrote: post #158

I am assigned to develop a company internal chatbot that accesses confidential documents and I am having a really hard time communicating this problem to executives: As long as not ALL the data the agent hat access too is checked against the rights of the current user placing the request, there WILL be ways to leak data. This means Vector databases, Search Indexes or fancy "AI Search Databases" would be required on a…

Depending on how you construct this, it may be a lot harder than you are saying.

If your approach is to build a chatbot that scans the documents, you can enforce access on a per-session basis by limiting the documents available to the application.

But if the approach is to train a neural net on this body of sensitive documents then you have a bigger problem. Actually two. The first is that the access control requirements have to be accounted for in the scoring function, which amounts to building a different engine for each user context. Though I suppose you could think of it as a composed neural net whose first net maps the input onto the [0,1] range on a per-user basis using the access control rules, and whose second net takes those results and runs them through additional layers.

The second is that the trailing neural net won't converge the same way for different inputs, and (so far as I'm aware) there isn't any theory for how to propagate access restrictions across a neural net.

Before inventing an old wheel, does anybody know of work on this in the research literature?

Re: Copilot broke audit logs, but Microsoft won't tell customers

#323

Post by kephasp »

planb wrote: post #158

I am assigned to develop a company internal chatbot that accesses confidential documents and I am having a really hard time communicating this problem to executives: As long as not ALL the data the agent hat access too is checked against the rights of the current user placing the request, there WILL be ways to leak data. This means Vector databases, Search Indexes or fancy "AI Search Databases" would be required on a…

One typical way to resolve this is to use Voluntary Oblivious Compliance (VOC). In this capability-based pattern, every storage service could provide an opaque handler to a user that represents their authorization and it can be used to restrict on which documents indexing is done.

http://wiki.erights.org/wiki/Walnut/Secure_Distributed_Compu...

http://www.skyhunter.com/marcs/ewalnut.html#proofOfPurchase

If the opaque handle is part of the Membrane pattern, you can even avoid most race conditions, because even during the indexing, the capabilities can be used to access documents and that removes the possibility of a TOCTOU race.

http://wiki.erights.org/wiki/Walnut/Secure_Distributed_Compu...

Re: Copilot broke audit logs, but Microsoft won't tell customers

#324

Post by kephasp »

DannyBee wrote: post #211
planb wrote: post #158

I am assigned to develop a company internal chatbot that accesses confidential documents and I am having a really hard time communicating this problem to executives: As long as not ALL the data the agent hat access too is checked against the rights of the current user placing the request, there WILL be ways to leak data. This means Vector databases, Search Indexes or fancy "AI Search Databases" would be required on a…

"would be required on a per user basis or track the access rights along with the content, which is infeasible and does not scale" Citation needed. Most enterprise (homegrown or not) search engine products have to do this, and have been able to do it effectively at scale, for decades at this point. This is a very well known and well-solved problem, and the solutions are very directly applicable to the products you lis…

Do you know papers or technical reports that demonstrate the scalability of authorization-preserving search indexes?

I don't doubt they exist but what we hear about are the opposite cases, where this was obviously not implemented and sensitive data was leaked.

Post reply on HN