Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating a production AI agent to GPT-5.6: 2.2x faster, 27% cheaper

ploy.ai

Post reply on HN
Print view View original

111–120 of 144 posts

Re: Migrating a production AI agent to GPT-5.6: 2.2x faster, 27% cheaper

#111

Post by avianlyric »

SwtCyber wrote: post #97
lcampbell wrote: post #65

> The fix that worked is a schema transform at the provider boundary. For OpenAI-family models only, we rewrite every optional property to be required but nullable, using anyOf: [T, null], which gives the model an explicit way to say “not using this.” I admit, I've only used a bastardized form of MCP, but this smells... wrong? It's not clear to me why the Typescript type definitions would have any influence on (what…

The thing is this isn't a schema generation or Typescript bug at all. This is just how openai's function calling works under the hood. Their weights were fine-tuned for tool use to output the most complete data structures possible. If the model sees a parameter name in the system prompt context it will try to fill it with a value, even if it is not in the required array

If it isn’t a schema generation or typescript bug, then why did changing the API types they use result in different model behaviour?

Re: Migrating a production AI agent to GPT-5.6: 2.2x faster, 27% cheaper

#112

Post by Anon1096 »

vessenes wrote: post #106
kristianp wrote: post #6

> Numbers like that buy a model a real migration effort. Such a silly choice of words. I wish the human directing the LLM writing the article put some effort into rewriting the worst examples of LLM style. > But it did extremely well, and the promise was immediate and specific: builds finishing in less than half the wall-clock time, at 27% lower cost, scoring at or above our incumbent on completed work. The way the L…

My current claude.md bans the phrase “load-bearing”, and Claude HATES that. It will troll occasionally in comments by saying things like “load-be…most specific”. Like it REALLY loves saying load-bearing. Urgh.

You are poisoning the context and priming it to say load-bearing. I don't really think there is a good way to actually ban specific phrases other than have it do a second pass, it's something baked into the model from post training.

Re: Migrating a production AI agent to GPT-5.6: 2.2x faster, 27% cheaper

#113

Post by brryant »

throwa356262 wrote: post #99

As of today, Ploy’s agent runs on GPT-5.6 Sol, the flagship tier of the model family OpenAI released this morning. Wait a moment, did they make the switch based on half a days of playing with Sol? Are these companies ran by teenagers?

hah - we actually skew staff, senior staff.

We have been testing GPT 5.6 for about a week as a preview model through a YC relationship, providing them feedback on the model. Our evals run in github CI and we can run them all in about 15 minutes against our eval bench of 115+ web design and marketing related jobs that ploy.ai specializes in.

then after we toggled it on (through a posthog feature flag) we actively monitored for failures.

I came from running Webflow, which powers > 1% of the internet so trying my best to relay all of that knowledge to ploy to power more % of the internet!

Re: Migrating a production AI agent to GPT-5.6: 2.2x faster, 27% cheaper

#114

Post by cute_boi »

make_it_sure wrote: post #92
blfr wrote: post #2

> Ploy’s agent builds and edits real marketing websites. It plans a page, reads the codebase, writes components, generates imagery, screenshots its own work, and decides when it’s done. That job description sets a very high bar for a model, and we test every frontier release against it. For the four months Opus held the default slot (first Opus 4.7, then 4.8), nothing we tested beat it. Well, unlike OP I haven't run…

gpt 5.6 is so much better ar design than fable

Can you please provide evidence. It shouldn't be hard to give side by side comparison. I have not found any task where gpt is better than fable.

Re: Migrating a production AI agent to GPT-5.6: 2.2x faster, 27% cheaper

#115

Post by Planktonne »

mkovach wrote: post #110
kristianp wrote: post #6

> Numbers like that buy a model a real migration effort. Such a silly choice of words. I wish the human directing the LLM writing the article put some effort into rewriting the worst examples of LLM style. > But it did extremely well, and the promise was immediate and specific: builds finishing in less than half the wall-clock time, at 27% lower cost, scoring at or above our incumbent on completed work. The way the L…

> The way the LLMs write (Claude perhaps?) With short phrases separated by colons, commas, or full stops, is so poor and frustrating. This is exactly why I keep a WRITING.md file alongside AGENTS.md or CLAUDE.md. Most people spend time telling the model what to build, but very little time telling it how to write. LLMs are surprisingly good at following explicit style guidance if you bother to give it to them. Mine in…

Making a problem harder to sport is not a way to fix it.

Re: Migrating a production AI agent to GPT-5.6: 2.2x faster, 27% cheaper

#116

Post by RugnirViking »

mkovach wrote: post #110
kristianp wrote: post #6

> Numbers like that buy a model a real migration effort. Such a silly choice of words. I wish the human directing the LLM writing the article put some effort into rewriting the worst examples of LLM style. > But it did extremely well, and the promise was immediate and specific: builds finishing in less than half the wall-clock time, at 27% lower cost, scoring at or above our incumbent on completed work. The way the L…

> The way the LLMs write (Claude perhaps?) With short phrases separated by colons, commas, or full stops, is so poor and frustrating. This is exactly why I keep a WRITING.md file alongside AGENTS.md or CLAUDE.md. Most people spend time telling the model what to build, but very little time telling it how to write. LLMs are surprisingly good at following explicit style guidance if you bother to give it to them. Mine in…

Willing to share your writing.md somewhere? pastebin or the like? I'd love to take a look at how different people are doing this.

I disagree with the person below - I don't think its a wrongheaded goal in and of itself to get better ai generated writing. I read ai output half of my day, I may as well try and make it better for my own sake. Passing that onto other people is an entirely unrelated field of problems, and passing it off as stuff you authored yourself is just plagarism.

Re: Migrating a production AI agent to GPT-5.6: 2.2x faster, 27% cheaper

#117

Post by mkovach »

RugnirViking wrote: post #116
mkovach wrote: post #110

Earlier quoted context omitted.

> The way the LLMs write (Claude perhaps?) With short phrases separated by colons, commas, or full stops, is so poor and frustrating. This is exactly why I keep a WRITING.md file alongside AGENTS.md or CLAUDE.md. Most people spend time telling the model what to build, but very little time telling it how to write. LLMs are surprisingly good at following explicit style guidance if you bother to give it to them. Mine in…

Willing to share your writing.md somewhere? pastebin or the like? I'd love to take a look at how different people are doing this. I disagree with the person below - I don't think its a wrongheaded goal in and of itself to get better ai generated writing. I read ai output half of my day, I may as well try and make it better for my own sake. Passing that onto other people is an entirely unrelated field of problems, and…

Here is a sample I am using for testing on a personal project (I'm also working on using SpeechActs to do prompting, so yes, the repo is all AI-generated).

https://f.mek.cc/gombasic/file?name=WRITING.md&ci=tip

And in many cases, like you, I spend a good amount of time reading AI documentation. I'm not "stealing" anything, just having a model generate words in a specific way. I'm just being precious in how I want things, without having to do it myself.

Re: Migrating a production AI agent to GPT-5.6: 2.2x faster, 27% cheaper

#118

Post by mkovach »

Planktonne wrote: post #115
mkovach wrote: post #110

Earlier quoted context omitted.

> The way the LLMs write (Claude perhaps?) With short phrases separated by colons, commas, or full stops, is so poor and frustrating. This is exactly why I keep a WRITING.md file alongside AGENTS.md or CLAUDE.md. Most people spend time telling the model what to build, but very little time telling it how to write. LLMs are surprisingly good at following explicit style guidance if you bother to give it to them. Mine in…

Making a problem harder to sport is not a way to fix it.

I understand what you are saying, but inside the repo, I'm making it explicit.

I'm not trying to hide the problem, just offering a solution that people may find useful, or curiously odd.

Re: Migrating a production AI agent to GPT-5.6: 2.2x faster, 27% cheaper

#119

Post by peab »

throwa356262 wrote: post #99

As of today, Ploy’s agent runs on GPT-5.6 Sol, the flagship tier of the model family OpenAI released this morning. Wait a moment, did they make the switch based on half a days of playing with Sol? Are these companies ran by teenagers?

at this point, it's pretty easy to create evals/benchmarks, and then run the latest model on them.

LLMs are so easy to swap out, so having good benchmarks/evals are pretty useful.

Even then, a lot of the time the model improvements are so obvious that you don't even need an eval.

Post reply on HN