Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Copilot broke audit logs, but Microsoft won't tell customers

pistachioapp.com

Post reply on HN
Print view View original

71–80 of 324 posts

Re: Copilot broke audit logs, but Microsoft won't tell customers

#71

Post by mlyle »

AdieuToLogic wrote: post #65
TheRoque wrote: post #49

In my opinion, using AI tools for programming at the moment, unless in a sandboxed environment and on a toy project, is just ludicrous. The amount of shady things going on in this domain (AI trained on stolen content, no proper attribution, not proper way to audit what's going out to third party servers etc.) should be a huge red flag for any professional developer.

> In my opinion, using AI tools for programming at the moment, unless in a sandboxed environment and on a toy project, is just ludicrous. Well put. The fundamental flaw is in trying to employ nondeterministic content generation based on statistical relevance defined by an unknown training data set, which is what commercial LLM offerings are, in an effort to repeatably produce content satisfying a strict mathematical…

Nearly as bad: trying to use systems made out of meat, evolved from a unrelated background and trained on an undocumented and chaotic corpus of data, to try and produce content satisfying a strict mathematical model.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#72

Post by lokar »

p_ing wrote: post #63
gpm wrote: post #60

Earlier quoted context omitted.

I think you just defined away the entire category of vulnerability known as "privilege escalation".

This isn’t an example of escalation. Copilot is using the user’s token similar to any other OAuth app that needs to act on behalf of the user.

If that is true, then how did it not get logged? The audit should not be under the control of the program making the access.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#73

Post by cut3 »

TheRoque wrote: post #49

In my opinion, using AI tools for programming at the moment, unless in a sandboxed environment and on a toy project, is just ludicrous. The amount of shady things going on in this domain (AI trained on stolen content, no proper attribution, not proper way to audit what's going out to third party servers etc.) should be a huge red flag for any professional developer.

[flagged]

Re: Copilot broke audit logs, but Microsoft won't tell customers

#74

Post by HelloImSteven »

db48x wrote: post #5
jayofdoom wrote: post #2

Generally speaking, anyone can file a CVE. Go file one yourself and force their response. This blogpost puts forth reasonably compelling evidence.

Fun, but it doesn’t deserve a CVE. CVEs are for vulnerabilities that are common across multiple products from multiple sources. Think of a vulnerability in a shared library that is used in most Linux distributions, or is statically linked into multiple programs. Copilot doesn’t meet that criteria. Honestly, the worst thing about this story is that apparently the Copilot LLM is given the instructions to create audit l…

CVEs aren’t just for common dependencies. The “Common” part of the name is about having standardized reporting that over time helps reveal common issues occurring across multiple CVEs. Individually they’re just a way to catalog known vulnerabilities and indicate their severity to anyone impacted, whether that’s a hundred people or billions. There are high severity CVEs for individual niche IoT thermostats and light strips with obscure weaknesses.

Technically, CVEs are meant to only affect one codebase, so a vulnerability in a shared library often means a separate CVE for each affected product. It’s only when there’s no way to use the library without being vulnerable that they’d generally make just one CVE covering all affected products. [1]

Even ignoring all that, people are incorporating Copilot into their development process, which makes it a common dependency.

[1]: https://www.redhat.com/en/topics/security/what-is-cve

Re: Copilot broke audit logs, but Microsoft won't tell customers

#75

Post by zavec »

Just to make sure I'm understanding footnote one correctly: it shows up (sometimes before and hopefully every time now) as a copilot event in the log, and there's no corresponding sharepoint event?

From a brief glance at the O365 docs it seems like the 'AISystemPluginData` field indicates that the event in the screenshot showing the missing access is a copilot event (or maybe they all get collapsed into one event, I'm not super familiar with O365 audit logs), and I'm inferring from the footnote that there's not another sharepoint event somewhere in either the old or new version. But if there is one that could at least be a mitigation if you needed to do such a search on the activity before the fix.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#76

Post by jdiff »

mlyle wrote: post #71
AdieuToLogic wrote: post #65

Earlier quoted context omitted.

> In my opinion, using AI tools for programming at the moment, unless in a sandboxed environment and on a toy project, is just ludicrous. Well put. The fundamental flaw is in trying to employ nondeterministic content generation based on statistical relevance defined by an unknown training data set, which is what commercial LLM offerings are, in an effort to repeatably produce content satisfying a strict mathematical…

Nearly as bad: trying to use systems made out of meat, evolved from a unrelated background and trained on an undocumented and chaotic corpus of data, to try and produce content satisfying a strict mathematical model.

Meaty feet can be held to a fire. To quote IBM, "A computer can never be held accountable."

Re: Copilot broke audit logs, but Microsoft won't tell customers

#77

Post by Sparkyte »

TheRoque wrote: post #49

In my opinion, using AI tools for programming at the moment, unless in a sandboxed environment and on a toy project, is just ludicrous. The amount of shady things going on in this domain (AI trained on stolen content, no proper attribution, not proper way to audit what's going out to third party servers etc.) should be a huge red flag for any professional developer.

[flagged]

Re: Copilot broke audit logs, but Microsoft won't tell customers

#78

Post by AdieuToLogic »

mlyle wrote: post #71
AdieuToLogic wrote: post #65

Earlier quoted context omitted.

> In my opinion, using AI tools for programming at the moment, unless in a sandboxed environment and on a toy project, is just ludicrous. Well put. The fundamental flaw is in trying to employ nondeterministic content generation based on statistical relevance defined by an unknown training data set, which is what commercial LLM offerings are, in an effort to repeatably produce content satisfying a strict mathematical…

Nearly as bad: trying to use systems made out of meat, evolved from a unrelated background and trained on an undocumented and chaotic corpus of data, to try and produce content satisfying a strict mathematical model.

Except that the "systems made out of meat" are the entities which both define the problem needing to be solved and are the sole determiners if said problem has been solved.

Of note too is that the same "systems made out of meat" have been producing content satisfying the strict mathematical model for decades and continue to do so beyond the capabilities of the aforementioned algorithms.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#79

Post by fulafel »

> CVEs are given to fixes deployed in security releases when customers need to take action to stay protected. In this case, the mitigation will be automatically pushed to Copilot, where users do not need to manually update the product and a CVE will not be assigned.

Is this a feature of CVE or of Microsoft's way of using CVE? It would seem this vulnerability would still benefit from having a common ID to be refrenced in various contexts (eg vulnerability research). Maybe there needs to be another numbering system that will enumerate these kinds of cases and doesn't depend on the vendor.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#80

Post by MobiusHorizons »

jdiff wrote: post #76
mlyle wrote: post #71

Earlier quoted context omitted.

Nearly as bad: trying to use systems made out of meat, evolved from a unrelated background and trained on an undocumented and chaotic corpus of data, to try and produce content satisfying a strict mathematical model.

Meaty feet can be held to a fire. To quote IBM, "A computer can never be held accountable."

This is the question I keep asking leaders (I literally asked a VP this question once in an all hands). How do we approach the risk associated mistakes made by AI?(process, legal, security, insurance etc) We have process and legal agreements in place to deal with humans that work for a business making mistakes. We need analogs for AI if we want to use it in similar ways.
Post reply on HN