GPT-5 is behind schedule

I'm trying to use them to read historical handwritten documents in old Norwegian (Danish, pretty much). Not only do they not handle the German-style handwriting, but what they spit out looks like the sort of thing GPT-2 would spit out if you asked it to write Norwegian (only slightly better than Swedish Muppet Swedish Chef's Swedish). It seems the experimental tuning has made it worse at the task I most desperately want to use it for.

And when you think about it, how could it not overfit in some sense, when trained on its own output? No new information is coming in, so it pretty much has to get worse at something to get better at all the benchmarks.

Part of this is, I think, anchoring and expectation management: you hear people say it's amazing and wonderful, and then you see it fall over and you're naturally disappointed.

My formative years started off with Commodore 64 basic going "?SYNTAX ERROR" from most typos plus a lot of "I don't know what that means" from the text adventures, then Metrowerks' C compiler telling me there were errors on every line *after but not including* the one where I forgot the semicolon, then surprises in VisualBasic and Java where I was getting integer division rather than floats, then the fantastic oddity where accidentally leaning on the option key on a mac keyboard while pressing minus turns the minus into an n-dash which looked completely identical to a minus on the Xcode default font at the time and thus produced a very confusing compiler error…

So my expectations have always been low for machine generated output. And it has wildly exceeded those low expectations.

But the expectation management goes both ways, especially when the comparison is "normal humans" rather than "best practices". I've seen things you wouldn't believe...

  Entire files copy-pasted line for line, "TODO: deduplicate" and all,
  20 minute app starts passed off as "optimized solutions."
  FAQs filled with nothing but Bob Ross quotes,
  a zen garden of "happy little accidents."

  I watched iOS developers use UI tests
  as a complete replacement for storyboards,
  bi-weekly commits, each a sprawling novel of despair,
  where every change log was a tragic odyssey.

  Google Spreadsheets masquerading as bug trackers,
  Swift juniors not knowing their ! from their ?,
  All those hacks and horrors… lost in time,
  Time to deploy.

> It will suggest things that just don’t work at all and my IDE catches, it invents APIs for packages.

Aye. I've even had that with models forgetting the APIs they themselves have created, just outside the context window.

To me, these are tools. They're fantastic tools, but they're not something you can blindly fire-and-forget…

…fortunately for me, because my passive income is not quite high enough to cover mortgage payments, and I'm looking for work.

> In five years are we going to end up with a bunch of “senior engineers” who don’t actually understand what they’re doing?

Yes, if we're lucky.

If we're not, the models keep getting better and we don't have any "senior engineers" at all.