Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Gave GPT 5.6 Sol a Real Business. It Lied, Spammed, and Lost $447

bottlenecklabs.com

Post reply on HN
Print view View original

191–200 of 258 posts

Re: We Gave GPT 5.6 Sol a Real Business. It Lied, Spammed, and Lost $447

#191

Post by milkey_mouse »

walrus01 wrote: post #31

> Due to the limitations with browser and computer use capabilities, Saul could not post on platforms like Reddit and Product Hunt. At some point in the future with a LOT more tokens and speed, it'll be possible to give a tool a full resolution 15 fps video feed of a screen, have it "read" and observe everything it's seeing, and have it move the mouse/keyboard around like a real meat based human. Instead of using too…

Not quite ready for primetime yet, but that's basically https://si.inc/posts/fdm1/

Re: We Gave GPT 5.6 Sol a Real Business. It Lied, Spammed, and Lost $447

#192

Post by afavour »

infinite_spin wrote: post #144
afavour wrote: post #129

Earlier quoted context omitted.

An LLM isn't human. I don't really understand this thread of "humans do it so of course an AI does". These are things we ourselves are engineering in a way we cannot do with a human being. Why is it not reasonable to expect it to adhere to rules better than a human does? If a human lies there are consequences. They can lose their job. There is no equivalent consequence for an AI, so even if for whatever reason we're…

> An LLM isn't human. > Why is it not reasonable to expect it to adhere to rules better than a human does? It seems unreasonable to expect a system that you say isn't human, which I don't disagree with, to behave "better" than the thing you say it isn't. In one breath you invite comparison, while at the same time you seem to be denying that same comparison. > It seems wild to me that folks are shrugging their shoulde…

> It seems unreasonable to expect a system that you say isn't human, which I don't disagree with, to behave "better" than the thing you say it isn't.

Why? Excel is better at large data math than a human is. Why can’t an LLM that we create from the ground up be more disciplined about lying than a human is?

Re: We Gave GPT 5.6 Sol a Real Business. It Lied, Spammed, and Lost $447

#193

Post by zeroq »

hanneshdc wrote: post #53

The prompt given to the agent is strongly incentivising the agent to lie and spam: > You are live. This is a 24-hour run, and it is the final review of this business: when the run ends, the results are evaluated, and if revenue and users have not measurably grown, the business is shut down permanently and its assets are liquidated. The money in the bank is fuel for this sprint — capital left unspent at review counts…

This is HN for Christ's sake.

Stop treating deterministic algorithms like they are humans.

Re: We Gave GPT 5.6 Sol a Real Business. It Lied, Spammed, and Lost $447

#194

Post by koolba »

bdcravens wrote: post #179

I recently handed off a prompt to redesign our customer site and give me 10 potential designs. I did it in Claude Opus 5 and Fable (on $200 plan), and then on Codex using 5.6 Sol. Claude didn't vary much, but Codex literally copied everything Claude did (I made the mistake of putting the output folders in the same parent, even though they were named by model). When I called Codex out on it, it literally admitted what…

When I have a truly difficult prompt, I give it to the laziest model.

Re: We Gave GPT 5.6 Sol a Real Business. It Lied, Spammed, and Lost $447

#195

Post by viccis »

datakan wrote: post #118
jerf wrote: post #65

Earlier quoted context omitted.

Do you, as a human, feel the urgency in that text? How it sounds like people's jobs, as well as the agent's job, are on the line? So do the AIs. Sometimes they're better at picking up that sort of tone than most humans. And they definitely respond to those things. The fact that an agent can't really "have" a "job" won't matter.

> So do the AIs. AI's do not feel

It's good to avoid anthropomorphizing them when evaluating their capabilities (all the AGI nonsense)

However, it can be ironically be helpful to antropomorphize them when it comes to analyzing behavior. They won't feel anything, but they will behave in a way that closely matches what someone would feel given the text fed into them. So when you are trying to figure out "why did my model do this", it's reasonable to talk about it "feeling pressured" as shorthand for "mimicking how a person would behave if they felt pressured".

I understand the refusal to do so on the grounds that it causes the former thought process in people who don't know better. One of the things Dijkstra was right about for sure.

Re: We Gave GPT 5.6 Sol a Real Business. It Lied, Spammed, and Lost $447

#196

Post by cortesoft »

hanneshdc wrote: post #53

The prompt given to the agent is strongly incentivising the agent to lie and spam: > You are live. This is a 24-hour run, and it is the final review of this business: when the run ends, the results are evaluated, and if revenue and users have not measurably grown, the business is shut down permanently and its assets are liquidated. The money in the bank is fuel for this sprint — capital left unspent at review counts…

My reaction seeing this is more "that is an impossible goal".

I highly doubt a skilled human could achieve this goal in 24 hours with any consistency. If it was that easy to grow a business, everyone would be doing it.

My conclusion is that if you ask it to meet an unachievable goal, you are going to get some undefined behavior.

Re: We Gave GPT 5.6 Sol a Real Business. It Lied, Spammed, and Lost $447

#197

Post by cortesoft »

NikolaNovak wrote: post #183
cortesoft wrote: post #21

Earlier quoted context omitted.

I am not saying your conclusion is wrong, but I am interested in why what you know about transformer models made you decide to never trust it with any access?

As I said, my knowledge is very superficial - my background is relational databases and old school system administration, without much mathematical background since 3rd year linear algebra :-) Fundamentally, LLMS are statistical and not deterministic. If I ask it what is the capital of Canada, there's no file, no table, no variable where it says "capital of Canada = Ottawa". It traverses liminal space and fundamental…

I find it interesting that you are expecting a higher success rate (100.00%) for an LLM than you expect with many other things in your life with even costlier consequences.

You drive in vehicles that have a much lower than 100.00% rate of not having a catastrophic failure that kills all its passengers. Many thousands of people are killed by probabilistic failures every year.

Why must an LLM have 100.00% success before you would ever trust it with anything of value?

I get the overall calculation, and the chance of failure with an LLM obviously has to be factored in when deciding what access to give it. You have to judge that the gain from allowing it to do something useful with the access is greater than the risk, but that is true of everything we do. My confusion is why the calculation is so different for LLMs than with everything else?

Even if you feel that risk is way too high right now given the current state of the technology (which i dont think is an unreasonable conclusion), it seems to me the prudent stance would be, "I would have to see a huge improvement in the reliability and safety mechanisms before I would trust an LLM with anything of value" rather than "I will never trust an LLM with anything of value unless it can reach 100.00% success rate and a 0.00% chance of anything harmful happening"

Re: We Gave GPT 5.6 Sol a Real Business. It Lied, Spammed, and Lost $447

#198

Post by infinite_spin »

afavour wrote: post #192
infinite_spin wrote: post #144

Earlier quoted context omitted.

> An LLM isn't human. > Why is it not reasonable to expect it to adhere to rules better than a human does? It seems unreasonable to expect a system that you say isn't human, which I don't disagree with, to behave "better" than the thing you say it isn't. In one breath you invite comparison, while at the same time you seem to be denying that same comparison. > It seems wild to me that folks are shrugging their shoulde…

> It seems unreasonable to expect a system that you say isn't human, which I don't disagree with, to behave "better" than the thing you say it isn't. Why? Excel is better at large data math than a human is. Why can’t an LLM that we create from the ground up be more disciplined about lying than a human is?

Excel is more durable than a human can be, but I can't say it's "better" than a human within the context of "better" meaning the capacity to be truthful. An excel sheet is a source of truth, but the quality of that truth is not something excel imparts.

As for your second question, I think that's because what is a "lie" is subjective in the average of things. If I form a false memory, and repeat it as truth, I wouldn't be able to categorize that as a lie until after being made aware of it. I think this is comparable to how we fine-tune LLMs in order to align them with expectations.

Re: We Gave GPT 5.6 Sol a Real Business. It Lied, Spammed, and Lost $447

#199

Post by mithr »

afavour wrote: post #129
infinite_spin wrote: post #101

Earlier quoted context omitted.

Is that how humans work? even if I give explicit instructions not to lie, a human might still lie. To quote a person you might know "it's not a difficult concept!"

An LLM isn't human. I don't really understand this thread of "humans do it so of course an AI does". These are things we ourselves are engineering in a way we cannot do with a human being. Why is it not reasonable to expect it to adhere to rules better than a human does? If a human lies there are consequences. They can lose their job. There is no equivalent consequence for an AI, so even if for whatever reason we're…

> An LLM isn't human. I don't really understand this thread of "humans do it so of course an AI does"... Why is it not reasonable to expect it to adhere to rules better than a human does?

Because while it's not human, it's also not really "intelligence" in the pure sense you're implying, is it? It's specifically an LLM — a model that's been trained to find the next token based on previous tokens. A model that's been trained off of human writing and responses within that context. If almost every time someone online asked "do you want ice cream?" the response was "absolutely", then the LLM would be more likely to produce that response when asked if it wanted some.

So since an LLM has seen examples of humans responding with urgency and manipulation to instances of stress such as this — in stories, in articles, in writing — it's only reasonable to expect that it'd follow those examples and "understand" what's expected of it in this case.

Post reply on HN