Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Copilot broke audit logs, but Microsoft won't tell customers

pistachioapp.com

Post reply on HN
Print view View original

151–160 of 324 posts

Re: Copilot broke audit logs, but Microsoft won't tell customers

#151

Post by bw86 »

internetter wrote: post #86
catmanjan wrote: post #36

Earlier quoted context omitted.

As someone else mentioned the file isnt actually accessed by copilot, rather copilot is reading the pre-indexed contents of the file in a search engine... Really Microsoft should be auditing the search that copilot executes, its actually a bit misleading to be auditing the file as accessed when copilot has only read the indexed content of the file, I don't say I've visited a website when I've found a result of it in…

> I don't say I've visited a website when I've found a result of it in Google I mean, it depends on how large the index window is, because if google returned the entire webpage content without leaving (amp moment), you did visit the website. fine line.

The challenge then is to differentiate between "I wanted to access the secret website/document" and "Google/Copilot gave me the secret website/document, but it was not my intention to access that".

Re: Copilot broke audit logs, but Microsoft won't tell customers

#152

Post by aetherspawn »

Trying to get off Microsoft right now for LOB apps … the incompetence (multiple hacks over the last few months, SSO zero day, and now learn Copilot ignores permissions when searching because the indexer runs as global admin) is getting just plain scary.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#153

Post by mrweasel »

Foobar8568 wrote: post #105

We have cases were purview were missing logs. Fun stuff when we tried to figure out a postmortem at my work. Microsoft tools can't be trust anymore, something really broke since COVID...

When Nadella became chairman of the board at Microsoft? That happened in 2021. It's entirely possible that with a new chairman came a new business strategy.

Satya Nadella is a cloud guy and a lot of the complaints people have of the changes in Microsoft products is that they are increasingly reliant on cloud infrastructure.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#155

Post by oeitho »

catmanjan wrote: post #36
lokar wrote: post #8

Wait, copilot operates as some privileged user (that can bypass audit?), not as you (or better, you with some restrictions) That can’t be right, can it?

As someone else mentioned the file isnt actually accessed by copilot, rather copilot is reading the pre-indexed contents of the file in a search engine... Really Microsoft should be auditing the search that copilot executes, its actually a bit misleading to be auditing the file as accessed when copilot has only read the indexed content of the file, I don't say I've visited a website when I've found a result of it in…

> Really Microsoft should be auditing the search that copilot executes, its actually a bit misleading to be auditing the file as accessed when copilot has only read the indexed content of the file, I don't say I've visited a website when I've found a result of it in Google

Not my domain of expertise, but couldn't you at some point argue that the indexed content itself is an auditable file?

It's not literally a file necessarily, but if they contain enough information that they can be considered sensitive, then where is the significant difference?

Re: Copilot broke audit logs, but Microsoft won't tell customers

#156

Post by pnt12 »

RachelF wrote: post #84
degamad wrote: post #67

One thing that's not clear in the write-up here: *which* audit log is he talking about? Sharepoint file accesses? Copilot actions? Purview? Something else?

Lots of things aren't clear. Copilot is accessing the indexed contents of the file, not the file itself, when you tell it not to access the file. The blog writer/marketer needs to look at the index access logs.

But those are technicalities.

I imagine the intended feature is learning about who read some information, and who modified it.

The implementation varies, but on a CRUD app it seems easy: an authenticated GET or PUT request against a file path - easy audit log.

If you are copying information to another place, and make it accessible there in a lossy way that is hard to audit... you broke your auditing system.

Maybe it's useful, maybe it's a trade-off, but is something that should be disclosed.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#157

Post by throw-qqqqq »

eleveriven wrote: post #141
Foobar8568 wrote: post #105

We have cases were purview were missing logs. Fun stuff when we tried to figure out a postmortem at my work. Microsoft tools can't be trust anymore, something really broke since COVID...

Between product sprawl, rushed AI integrations, and weird transparency decisions, it feels like reliability and accountability took a backseat

Are reliability and accountability core values at MSFT?

I don’t personally see that company as reliable or trustworthy at all.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#158

Post by planb »

I am assigned to develop a company internal chatbot that accesses confidential documents and I am having a really hard time communicating this problem to executives:

As long as not ALL the data the agent hat access too is checked against the rights of the current user placing the request, there WILL be ways to leak data. This means Vector databases, Search Indexes or fancy "AI Search Databases" would be required on a per user basis or track the access rights along with the content, which is infeasible and does not scale.

And as access rights are complex and can change at any given moment, that would still be prone to race conditions.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#159

Post by NameForComment »

degamad wrote: post #67

One thing that's not clear in the write-up here: *which* audit log is he talking about? Sharepoint file accesses? Copilot actions? Purview? Something else?

In one of the footnotes: "The audit log will not show that the user accessed the file as a normal SharePointFileOperation FileAccessed event, but rather as a CopilotInteraction record. That’s intended, and in my opinion correct. It would be weird to make it as if the user directly accessed the file when they only did so via Copilot."

Re: Copilot broke audit logs, but Microsoft won't tell customers

#160

Post by cryptonym »

planb wrote: post #158

I am assigned to develop a company internal chatbot that accesses confidential documents and I am having a really hard time communicating this problem to executives: As long as not ALL the data the agent hat access too is checked against the rights of the current user placing the request, there WILL be ways to leak data. This means Vector databases, Search Indexes or fancy "AI Search Databases" would be required on a…

True, per user doesn't scale.

Knowledge should be properly grouped and have rights on database, documents, and chatbot managed by groups. For instance specific user can use the Engineering chatbot but not the Finance one. If you fail to define these groups, feels like you don't have a solid strategy. In the end, if that's what they want, let them experience open knowledge.

Post reply on HN