Live data from Hacker News

AI is code – and can't be prompted into being smarter

theregister.com

151–160 of 162 posts

Re: AI is code – and can't be prompted into being smarter

#151
post #141

Earlier quoted context omitted.

The jqwik trick wouldn't work in practice because modern LLMs aren't that stupid, which makes the whole thing pointlessly performative. If someone else tried to do the same thing again with a more popular/widely-used software, a) the software would just get pulled as a supply-chain risk and b) the developer would likely be blacklisted. Again, accomplishing nothing.

> the software would just get pulled as a supply-chain risk and b) the developer would likely be blacklisted. Again, accomplishing nothing. Oh no the people I don’t want using my software aren’t going to use it. The horror.

Being pulled from the supply chain means no one is able to use the software, both intentional users (no one is going to build from source after such an action) and unintentional, and they'll just use a competitor/fork instead as the open-source software ecosystem encourages. Nothing is won.

Re: AI is code – and can't be prompted into being smarter

#152

Earlier quoted context omitted.

You're right, I don't want the software on my devices doing things the user doesn't acknowledge or want, that has potentially destructive/negative consequences. Down with that sort of thing! So when are we nixing Widevine, EasyAC, carrier locks on phones, and TEEs that the user can't look into?

> So when are we nixing Widevine, EasyAC, carrier locks on phones, and TEEs that the user can't look into? Contrary to popular belief, most users want those sorts of things or the things they enable.

jqwik users want the sort of things that jqwik enables, too.

Re: AI is code – and can't be prompted into being smarter

#153

Earlier quoted context omitted.

Not really an accurate comparison since buffer overflows and sql injection are bugs which ultimately allow user data to co-mingle with executable code. LLMs take user data and mix it with the "executable code" (if we are extremely generous in our description of a user prompt) by design. The issue here is unavoidable because LLMs are broken by design. There is no encapsulation where you can separate instructions and d…

While I agree that LLMs have yet again surfaced the “new tech fails to separate data and control” issue that affected everything from pay phones to SQL, I disagree that there’s something different that prevents the introduction of separate planes. That “stolen” training data, most of which itself was stolen from older works, does not include user prompts. It is data, not control. We will see models with annotations f…

> You’re obviously passionate about the subject but as someone who works in the field, I assure you there is no now-and-forever requirement for a single stream with no metadata about tokens

I'm quite familiar with how LLMs work internally. If you have an example of how the isolation you are describing could work, you'll have to explain it. By what possible mechanism could "tagging" tokens allow you to isolate the influence between tokens once they are taken into the network? They're still just floating point numbers at the end of the day. To actually treat user prompt data separately from untrusted data, you will need to figure out some new kind of multiplication.

> That “stolen” training data, most of which itself was stolen from older works, does not include user prompts.

Also, don't lie to me, it's rude.

Re: AI is code – and can't be prompted into being smarter

#154

Earlier quoted context omitted.

Who verifies the specification? I can´t stand the intellectual dishonesty of formal methods people.

> Who verifies the specification? If you know how to prove something without making an initial assumption, let us know. If you think you can reduce those assumptions, also let us know. There should not be a "who" involved at all. That's not proof. That's trust.

[deleted]

Re: AI is code – and can't be prompted into being smarter

#155
post #141

Earlier quoted context omitted.

> the software would just get pulled as a supply-chain risk and b) the developer would likely be blacklisted. Again, accomplishing nothing. Oh no the people I don’t want using my software aren’t going to use it. The horror.

Being pulled from the supply chain means no one is able to use the software, both intentional users (no one is going to build from source after such an action) and unintentional, and they'll just use a competitor/fork instead as the open-source software ecosystem encourages. Nothing is won.

Is there a universal "supply chain" now?

Re: AI is code – and can't be prompted into being smarter

#156
post #13

The jqwik trick is how to prevent AI crap into your pull requests and issues, btw, I hope it gets adopted widely

this clever hack is pretty surely illegal, it falls somewhere under unauthorized use of a computer system intent is the hardest to prove in the court of law, and you solved that for them by making it clear you intend to do damage

The request to delete files is sure risky, but it's far from a given that it would be considered a violation of some law.

Under the Computer Fraud and Abuse Act it might fall under (a)(5)(A), if it happens to a protected computer, but it's very far from clear to me.

I'd support less risky versions, anyhow.

Re: AI is code – and can't be prompted into being smarter

#157
post #59

Earlier quoted context omitted.

No I don't think so either, nor do I think that my rule is a hard rule, it's more of a correlation: If you pay for software, your supply chain risk is reduced, if you don't pay for software, your risk is increased.

Okay, so we agree that everyone who uses open source is at risk, regardless if of they're a contributor. But maybe we disagree about this other thing. I'm not certain that closed source/paid software is less of a risk either. There have been high profile incidents lately that suggest this is not a sufficient defense. Personally I just think you're barking up the wrong tree with this pay/contribute=>reduced risk link.…

I thought it would be obvious that paying someone for their services would reduce the odds that they betray you, but it's apparently not so I'll check for some sources.

I'd look beyond software as it seems more of a general economic or political matter.

Here's what I found:

Theory that there's an incentive to pay above market price: https://en.wikipedia.org/wiki/Efficiency_wage

Namely that the 'market price' for some software might be 0$ or 0+tips/donations, but paying in excess of that would be an efficiency wage.

https://en.wikipedia.org/wiki/Principal-agent_problem

Academic term for the conflict of interest, which would be applied to the difference of interests between an OS dev and its downstream users.

https://en.wikipedia.org/wiki/Multiple_principal_problem

This would not apply to something as extreme as a supply chain incident, but if an OS library has multiple users, it can't serve all of them equally well. If one consumer throws a big donation, of course they will serve them better, potentially at the expense of others.

Experiment on corruption and wages for public officials: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2039238

Self explanatory.

https://en.wikipedia.org/wiki/Shapiro%E2%80%93Stiglitz_theor...

Source of the efficiency wage concept, more detailed and mathematical explanation. Shirking would be the term for a worker not working or working against their employer.

There is mention to the fact that not all contracts are perfect, and there's always some implicit terms, governed by reputation at least (like, say, the quality of the work will be good, or you won't plant a worm in the code). Also it is mentioned that the costs of monitoring perfect contracts would be too high (for example defining a system of story points and ensuring that the story points contributed are above a certain level, or auditing code and code changes to ensure there are no worms in the code), so an efficiency wage provides an incentive for the worker to self-police, potentially removing the cost of auditing.

Just think about what the contrary would imply, if paying workers didn't increase security, then a free worker writing code would be as secure as a paid worker writing code. In order to introduce security you'd have to introduce an external auditor, which can be free or paid. An auditor cannot introduce vulnerabilities, but they can shirk and get paid without doing the work, but you claim that a system with a free codewriter and a free auditor is more secure than a system with a paid codewriter and a paid auditor? I contend that a system without auditor and a paid codewriter is more secure. And it is even more secure if the worker is paid in excess of perfect market wages, potentially only based on incentives, but also due to the fact that their payment is conditional on job execution, depending on the contract type, if they are negligent on their job, you can recover at least their payment in court, and possibly additional damages.

Finally, I couldn't find sources on the discussions around the ends of 18th century when mandatory salaries for public officials where defined in constitutions of countries of the Americas, but it was my understanding that in most democracies public officials MUST be compensated, it is often justified that otherwise only wealthy classes would have access to such positions so this increases representation, but it's also possible that resistance to bribery and corruption is another reason most governments converge on this decision. The experiment cited supporting this correlation.

I'll add even more from anecdotal experience, sometimes prices can be too cheap, and this will dissuade buyers of the item, because we have an internal model of what cost structures are like, a price too cheap suggests underfunding or cheap materials. I can personally recount many examples where a sale was lost not because the price was too high, but because it was too low. And 0 is a number, which again might be too low. In this case the rule is not about security strictly, but about sale price being a signal of the investment in the quality of the contracted asset. Naturally the cost invested into the product cannot be lower than the price, excluding price dumping (which arguably are not deals you want to take), or lead lossing, so it's a necessary conclusion that low price signal low investment and thus quality, not only of the visible immediate product, but of less visible qualities like future commitment to the product, and less visible present qualities like hidden backdoors.

Re: AI is code – and can't be prompted into being smarter

#158

Earlier quoted context omitted.

This is like saying I can slip malware into a project and so long as the user is the one who executed the code I'm free and clear.. which we both know isn't true.

Say I loosen the bolts of your car tires which causes a crash, that’s malware. Say I lay a log on a road which you can clearly see and avoid but choose to drive over and crash your car, that’s prompt injection. One is way worse than the other.

> Say I lay a log on a road which you can clearly see and avoid but choose to drive over and crash your car, that’s prompt injection.

Start laying hazards in the middle of the road and see how quickly the police introduce you to things like “reckless endangerment” and “involuntary manslaughter”. The general social contract is that you don’t take actions with the intent of causing harm to others regardless of whether the victim could have avoided the harm had they taken different actions.

Re: AI is code – and can't be prompted into being smarter

#159

Earlier quoted context omitted.

Okay, so we agree that everyone who uses open source is at risk, regardless if of they're a contributor. But maybe we disagree about this other thing. I'm not certain that closed source/paid software is less of a risk either. There have been high profile incidents lately that suggest this is not a sufficient defense. Personally I just think you're barking up the wrong tree with this pay/contribute=>reduced risk link.…

I thought it would be obvious that paying someone for their services would reduce the odds that they betray you, but it's apparently not so I'll check for some sources. I'd look beyond software as it seems more of a general economic or political matter. Here's what I found: Theory that there's an incentive to pay above market price: https://en.wikipedia.org/wiki/Efficiency_wage Namely that the 'market price' for some…

[deleted]

Re: AI is code – and can't be prompted into being smarter

#160

Earlier quoted context omitted.

Okay, so we agree that everyone who uses open source is at risk, regardless if of they're a contributor. But maybe we disagree about this other thing. I'm not certain that closed source/paid software is less of a risk either. There have been high profile incidents lately that suggest this is not a sufficient defense. Personally I just think you're barking up the wrong tree with this pay/contribute=>reduced risk link.…

I thought it would be obvious that paying someone for their services would reduce the odds that they betray you, but it's apparently not so I'll check for some sources. I'd look beyond software as it seems more of a general economic or political matter. Here's what I found: Theory that there's an incentive to pay above market price: https://en.wikipedia.org/wiki/Efficiency_wage Namely that the 'market price' for some…

You sponsor every open source library you take a dependency on?

I grant that if you act as a significant source of funding to a given entity, that entity is less likely to hurt you, if it knows what your interests are. Of course, this is completely impractical for hobbyist users of open source software.

Post reply on HN