Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we rooted Copilot

research.eye.security

Post reply on HN
Print view View original

101–110 of 147 posts

Re: How we rooted Copilot

#101

Post by wizzwizz4 »

stevage wrote: post #93
simonw wrote: post #8

OK, I think I understand what this is about: the vulnerability that they reported (and Microsoft fixed) is that there was a trick you could use to run your own code with root privileges inside the container - when the system was designed to have you only execute code as a non-root user. It turned out not to really matter, because the container itself was still secured - you couldn't make network requests from it and…

It seems weird to me that copilot sometimes refuses to execute code but sometimes allows it. What exactly are they aiming for?

They're not. It's better to think of Copilot as a collaborative storytelling session with a text autocomplete system, which some other program is rudely hijacking to insert the result of running certain commands.

Sometimes the (completion randomly selected from the outputs of the) predictive text model goes "yes, and". Other times, it goes "no, because". As observed in the article, if it's autocompleting the result of many "yes, and"s, the story is probably going to have another "yes, and" next, but if a story starts off with a certain kind of demand, it's probably going to continue with a refusal.

Re: How we rooted Copilot

#102

Post by pbhjpbhj »

thfuran wrote: post #92
ajross wrote: post #90

Earlier quoted context omitted.

Not really the right metaphor. A $5 wrench isn't a "vulnerability" because it's $5! Tools that are accessible to everyone are part of the threat model, not something you can eliminate or avoid. This trick is novel and new. Like, consider your personal cult was built around an "unopenable" bolt-tighted box. Then someone invents the wrench in an attempt to open it. That would be a clear "security vulnerability", right?

Not a serious one if all the wrench actually gets you is access to the room that contains the box that no known tool can open, which is a closer analogy to what happened.

And an exploit that breaks out of the sandbox is not really anything if it needs root to work... so if a hacker had those two MS wouldn't care about them selling those bugs because both of them are not serious. See, perfect security and it didn't cost them anything.

Re: How we rooted Copilot

#103

Post by bigfatkitten »

stogot wrote: post #91
0xbadcafebee wrote: post #78

Earlier quoted context omitted.

I have to give Microsoft props here. Most companies don't bother to lock things down well enough, but they were thorough.

I would give the one engineer the credit for doing things better, not Microsoft. Microsoft overall culture of security is terrible. Look at the CISA report.

Microsoft has islands of security excellence in what these days is a sea of mediocrity.

Re: How we rooted Copilot

#104

Post by stevage »

wizzwizz4 wrote: post #101
stevage wrote: post #93

Earlier quoted context omitted.

It seems weird to me that copilot sometimes refuses to execute code but sometimes allows it. What exactly are they aiming for?

They're not. It's better to think of Copilot as a collaborative storytelling session with a text autocomplete system, which some other program is rudely hijacking to insert the result of running certain commands. Sometimes the (completion randomly selected from the outputs of the) predictive text model goes "yes, and". Other times, it goes "no, because". As observed in the article, if it's autocompleting the result o…

funny how it sounds kind of the opposite of how people might work. Get enough 'no's from someone and they might finally cave in. get enough 'yes'es and they might get sick of doing everything you ask.

Re: How we rooted Copilot

#105

Post by [deleted] »

stogot wrote: post #91
0xbadcafebee wrote: post #78

Earlier quoted context omitted.

I have to give Microsoft props here. Most companies don't bother to lock things down well enough, but they were thorough.

I would give the one engineer the credit for doing things better, not Microsoft. Microsoft overall culture of security is terrible. Look at the CISA report.

[deleted]

Re: How we rooted Copilot

#106

Post by 0xbadcafebee »

stogot wrote: post #91
0xbadcafebee wrote: post #78

Earlier quoted context omitted.

I have to give Microsoft props here. Most companies don't bother to lock things down well enough, but they were thorough.

I would give the one engineer the credit for doing things better, not Microsoft. Microsoft overall culture of security is terrible. Look at the CISA report.

Okay, so I give the team that put this together credit. Hopefully the parent company sees based on this that it's worth letting teams invest more in quality and security work, over features.

Re: How we rooted Copilot

#107

Post by kortilla »

trueismywork wrote: post #69
CharlesW wrote: post #41

Earlier quoted context omitted.

I applaud your choice! I just can't think of any free software examples that don't also benefit corporations.

You need to think it in a different manner. When you have AGPL code, then it benefits mankind more than corporations. There's a Harvard report on value of open source to society based on how much money corporations put in. Today linux is working nicely on desktops (even though it's not the year of linux) and is heavily dominated by corporations. The parts where linux doesn't do well are exactly parts without corporat…

That’s not an example of open source that doesn’t benefit corporations. Linux is amazing for corporations.

Re: How we rooted Copilot

#108

Post by NemosDemos »

kenjackson wrote: post #95
stogot wrote: post #91

Earlier quoted context omitted.

I would give the one engineer the credit for doing things better, not Microsoft. Microsoft overall culture of security is terrible. Look at the CISA report.

What CISA report?

Not OP, but guessing they were referencing this one:

https://www.cisa.gov/resources-tools/resources/CSRB-Review-S...

Re: How we rooted Copilot

#109

Post by tptacek »

ajross wrote: post #96
thfuran wrote: post #92

Earlier quoted context omitted.

Not a serious one if all the wrench actually gets you is access to the room that contains the box that no known tool can open, which is a closer analogy to what happened.

Again, though, you're taking "all that gets you" as a prior when (abandoning the metaphor) container and VM escapes are routine vulnerabilities . They just weren't the subject of this particular team who wanted to hack on AI. You don't do security analysis by presuming the absence of vulnerabilities! Modern security is defense in depth. The AI pre-prompting setup was the first layer, and it was escaped. The UID separ…

Sure, I guess, but a lot more is broken than Copilot if you assume arbitrary container escape. (I do!)

Re: How we rooted Copilot

#110

Post by what »

eddythompson80 wrote: post #80
blastonico wrote: post #70

How does he know that the response isn't just hallucinations? I'm telling it because I work there and I don't recognize any of those processes. In fact I found one script named keepAliveJupyterSvc.sh in a public repo: https://github.com/shivamkm07/code-interpreter/blob/load-tes...

That repo, and its contributors are MS/Azure employees working on the service for running python code in a container. I don't know why it's under a personal account. Though it says it's a fork from an Office repo that I can't find.

How do you figure? I don’t see anything that suggest they work for ms/azure?
Post reply on HN