Wait, copilot operates as some privileged user (that can bypass audit?), not as you (or better, you with some restrictions) That can’t be right, can it?
> That can’t be right, can it?
Copilot broke audit logs, but Microsoft won't tell customers
11–20 of 324 posts
Re: Copilot broke audit logs, but Microsoft won't tell customers
#12Wait, copilot operates as some privileged user (that can bypass audit?), not as you (or better, you with some restrictions) That can’t be right, can it?
> That can’t be right, can it? https://knowyourmeme.com/memes/james-franco-first-time
lol. I’ve avoided MS my entire (30+ year) career. Every now and then I’m reminded I made the right choice.
Re: Copilot broke audit logs, but Microsoft won't tell customers
#13are they even sure that the AI even accessed the content that second time? LLMs are really good and making up shit. I have tested this by asking various LLMs to scrape data from my websites while watching access logs. Many times, they don't and just rely on some sort of existing data or spout a bunch of BS. Gemini is especially bad like this.
I have not used copilot myself, but my experience with other AI makes me curious about this.
Re: Copilot broke audit logs, but Microsoft won't tell customers
#14Hard to count the number of things that can go wrong by relying directly on an LLM to manage audit/activity/etc. logs. What was their bug fix? Shadow prompts?
> Hard to count the number of things that can go wrong by relying directly on an LLM to manage audit/activity/etc. logs.
Nothing in this post suggests that they're relying on the LLM itself to append to the audit logs. That would be a preposterous design. It seems far more likely the audit logs are being written by the scaffolding, not by the LLM, but they instrumented the wrong places. (I.e. emitting on a link or maybe a link preview being output, rather than e.g. on the document being fed to the LLM as a result of RAG or a tool call.)
(Writing the audit logs in the scaffolding is probably also the wrong design, but at least it's just a bad design rather than a totally absurd one.)
Re: Copilot broke audit logs, but Microsoft won't tell customers
#15[flagged]
Re: Copilot broke audit logs, but Microsoft won't tell customers
#16Generally speaking, anyone can file a CVE. Go file one yourself and force their response. This blogpost puts forth reasonably compelling evidence.
It’s true. The form is right here. When they support PGP, I suspect they know what they’re doing and why, and have probably been continuously doing so for longer than I have been alive. Just look at their sponsors and partners.
Please only use this for legitimate submissions.
Re: Copilot broke audit logs, but Microsoft won't tell customers
#17are they even sure that the AI even accessed the content that second time? LLMs are really good and making up shit. I have tested this by asking various LLMs to scrape data from my websites while watching access logs. Many times, they don't and just rely on some sort of existing data or spout a bunch of BS. Gemini is especially bad like this. I have not used copilot myself, but my experience with other AI makes me cu…
This is it. M365 uses RAG on your enterprise data that you allow it to access. It's not actually accessing the files directly in the cases he provided. It's working as intended.
Re: Copilot broke audit logs, but Microsoft won't tell customers
#18This seems like a five alarm fire for HIPPA, is there something I’m missing?
Re: Copilot broke audit logs, but Microsoft won't tell customers
#19This seems like a five alarm fire for HIPPA, is there something I’m missing?
It's HIPAA.
Re: Copilot broke audit logs, but Microsoft won't tell customers
#20I am so tired of Microsoft cramming Copilot into everything. Search at $dayjob is completely borked right now. It shows a page of results, but the immediately pops up some warning dialog you cannot dismiss that Copilot can’t access some file “” or something. Every VSCode update I feel like I have to turn off Copilot in some new way. And now apparently it’ll be added to Excel as well. Thankfully I don’t have to use an…
Everything except the best thing they could have brought back: Clippy! </3