Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Copilot broke audit logs, but Microsoft won't tell customers

pistachioapp.com

Post reply on HN
Print view View original

261–270 of 324 posts

Re: Copilot broke audit logs, but Microsoft won't tell customers

#261

Post by ceridwyn »

swiftcoder wrote: post #187
bw86 wrote: post #151

Earlier quoted context omitted.

The challenge then is to differentiate between "I wanted to access the secret website/document" and "Google/Copilot gave me the secret website/document, but it was not my intention to access that".

Access is access. Regardless of whether you intended to view the document, you are now aware of its content in either case, and an audit entry must be logged.

Strongly agree. Consider the case of a healthcare application where, during the course of business, staff may perform searches for patients by name. When "Ada Lovelace" appears even briefly in the search results of a "search-as-you-type" for some "Adam _lastname", has their privacy has been compromised? I think so, and the audit log should reflect that.

I'm a fan of FHIR (a healthcare api standard, but far from widely adopted), and they have a secondary set of definitions for Audit log patterns (BALP) that recommends this kind of behaviour. https://profiles.ihe.net/ITI/BALP/StructureDefinition-IHE.Ba...

"[Given a query for patients,] When multiple patient results are returned, one AuditEvent is created for every Patient identified in the resulting search set. Note this is true when the search set bundle includes any number of resources that collectively reference multiple Patients."

Re: Copilot broke audit logs, but Microsoft won't tell customers

#262

Post by seanw265 »

doomslice wrote: post #198
krisoft wrote: post #165

Earlier quoted context omitted.

> This means Vector databases, Search Indexes or fancy "AI Search Databases" would be required on a per user basis or track the access rights along with the content, which is infeasible and does not scale. I don't understand why you think tracking user access rights would be infeasible and would not scale. There is a query. You search for matching documents in your vector database / index. Once you have found the pot…

Let's say you have 100000 documents in your index that match your query but only 10 of them the user has access to: A basic implementation will return the top, let's say 1000, documents and then do the more expensive access check on each of them. Most of the time, you've now eliminated all of your search results. Your search must be access aware to do a reasonable job of pre-filtering the content to documents the use…

If you can move your access check to the DB layer, you skip a lot of this trouble.

Index your ACLs, index your users, index your docs. Your database can handle it.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#263

Post by krisoft »

scott_w wrote: post #223
krisoft wrote: post #165

Earlier quoted context omitted.

> This means Vector databases, Search Indexes or fancy "AI Search Databases" would be required on a per user basis or track the access rights along with the content, which is infeasible and does not scale. I don't understand why you think tracking user access rights would be infeasible and would not scale. There is a query. You search for matching documents in your vector database / index. Once you have found the pot…

> I don't understand why you think tracking user access rights would be infeasible and would not scale. Allow me to try to inject my understanding of how these agents work vs regular applications. A regular SaaS will have an API endpoint that has permissions attached. Before the endpoint processes anything, the user making the request has their permissions checked against the endpoint itself. Once this request succee…

> AI Agents, instead, directly access the database, completely bypassing this layer.

If so, then as the wise man says: "well, there‘s your problem!"

I don't doubt there are implementations like that out there, but we should not judge the potential of a technology by the mistakes of the most boneheaded implementation.

Doing the same in the bank analogy would be like giving root SQL access to the phone operators and then asking them pretty please to be careful with it.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#264

Post by frakt0x90 »

mpeg wrote: post #197
adtac wrote: post #179

Earlier quoted context omitted.

>That's basically your documents lossily encoded. Vector embeddings are lossy encodings of documents roughly in the same way a SHA256 hash is a lossy encoding. It's virtually impossible to reverse the embedding vector to recover the original document. Note: when vectors are combined with other components for search and retrieval, it's trivial to end up with a horribly insecure system, but just vector embeddings are u…

I wouldn't say those two are equivalent. A cryptographic hash requires the exact full document to be available to "recover it" from the hash. With a vector embedding you can extract information related to the document from the embedding alone as long as you know (or can guess) what embedding model was used. You won't be able to reconstruct the document but you will be able to infer some meaning from the vector alone

Yes there have been multiple papers showing information extraction from embedding vectors if you know the model used. SHA by design maps similar strings pseud-randomly. Embeddings by design map similar strings similarly.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#265

Post by tough »

krapp wrote: post #233
pc86 wrote: post #228

Earlier quoted context omitted.

> I'm making an assumption on how these AI Agents work, I could be wrong. I don't understand the desire - borderline need - of folks on HN to just make stuff up. That is likely why you're being downvoted. I know we all love to do stuff "frOM fIRsT PRiNcIPlEs" around here but "let me just imagine how I think AI agents work then pass that off as truth" is taking it a bit far IMO. This is the human equivalent of an AI h…

>I don't understand the desire - borderline need - of folks on HN to just make stuff up. Hacker News is addictive. This forum is designed to reward engagement with imaginary internet points and that operant conditioning works just as well here as everywhere else.

karma please go up

Re: Copilot broke audit logs, but Microsoft won't tell customers

#266

Post by jdiff »

galaxyLogic wrote: post #116
MobiusHorizons wrote: post #80

Earlier quoted context omitted.

This is the question I keep asking leaders (I literally asked a VP this question once in an all hands). How do we approach the risk associated mistakes made by AI?(process, legal, security, insurance etc) We have process and legal agreements in place to deal with humans that work for a business making mistakes. We need analogs for AI if we want to use it in similar ways.

My question is if I get some code from AI, save it to a file, then modify it or add some functions to it, can I still claim the copyright for it at the top of the file? Do I need to give the AI any credit? I'm asking because I read somewhere that "AI produced output cannot be copyrighted". But what if I modify that output myself? I am then a co-creator, right, and I think I should have a right to some copyright prote…

First, a few disclaimers, I am not a lawyer and this is an actively evolving area.

The answer that most aligns with current precedent to my knowledge is that the parts you modify are protected by your copyright, but the rest remains uncopyrightable. With the exception of any chunks generated that align with someone's existing copyrighted code, as long as those chunks are substantial and unique enough.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#267

Post by tough »

flir wrote: post #238
lupusreal wrote: post #166

Earlier quoted context omitted.

It doesn't even need to be blaming the engineers in this case, they can blame "the AI" and most people will accept that and let whatever incident happened slide. If somebody questions the wisdom of putting AI in such a position, they can be dismissed as not appreciating new technology (even though their concern is valid.)

"the AI did it" is going to be the new "somebody hacked my facebook account" I wish I had a way of ensuring culpability remains with the human who published the text, regardless of who/what authored it.

if you're in a regulated field like law or medicine and you fuck up signing some AI slop with your name, you should loose your license at the very least

tools are fine to use, personal responsability is still required. Companies already fuck up with this too much

Re: Copilot broke audit logs, but Microsoft won't tell customers

#268

Post by michael1999 »

lokar wrote: post #8

Wait, copilot operates as some privileged user (that can bypass audit?), not as you (or better, you with some restrictions) That can’t be right, can it?

Have you met Microsoft?

This is the organization that pushed code-signing as their security posture for a decade.

Re: Copilot broke audit logs, but Microsoft won't tell customers

#269

Post by plaguuuuuu »

perlgeek wrote: post #249
doomslice wrote: post #198

Earlier quoted context omitted.

Let's say you have 100000 documents in your index that match your query but only 10 of them the user has access to: A basic implementation will return the top, let's say 1000, documents and then do the more expensive access check on each of them. Most of the time, you've now eliminated all of your search results. Your search must be access aware to do a reasonable job of pre-filtering the content to documents the use…

"Fun" Fact: ServiceNow simply passes this problem on to its users. I've seen a list of what was supposed to be 20 items of something, it only showed 2, plus a comment "18 results were omitted to insufficient permissions". (Servicenow has at least three different ways to do permissions, I don't know if this applies to all of them).

I'm not sure if enumerating the hidden results are a great idea :0

Re: Copilot broke audit logs, but Microsoft won't tell customers

#270

Post by MobiusHorizons »

jpcosta wrote: post #83
MobiusHorizons wrote: post #80

Earlier quoted context omitted.

This is the question I keep asking leaders (I literally asked a VP this question once in an all hands). How do we approach the risk associated mistakes made by AI?(process, legal, security, insurance etc) We have process and legal agreements in place to deal with humans that work for a business making mistakes. We need analogs for AI if we want to use it in similar ways.

What was the answer? Asking for a vp friend

In the all hands I got an answer about techniques that would be used to reduce the likelihood of mistakes. Ie not an answer.
Post reply on HN