Ask HN: Is it just me or GPT-4's quality has significantly deteriorated lately?

Or, more straightforwardly, with "beginner's luck", which can be seen as a form of survivor bias. Most people, when they start gambling, win and lose close to the average. Some people, when they start gambling, lose more than average -- and as a result are much less likely to continue gambling. Others, when they start gambling, win more than average -- and as a result are much more likely to continue gambling. Most long-term / serious gamblers did win more than average when starting out, because the ones who lost more than average didn't become long-term / serious gamblers.

Almost certainly a similar effect would happen w/ GPT-4: People who had better-than-average interactions to begin with became avid users, and really are experiencing a lowering of quality simply by statistics; people who had worse-than-average interactions to begin with gave up and never became avid users.

One could try to re-run the benchmarks that were mentioned in the OpenAI paper, and see how they fare; but it's not unlikely that OpenAI themselves are also running those benchmarks, and making efforts to keep them from falling.

Probably the best thing to do would be to go back and find a large corpus of older GPT-4 interactions, attempt to re-create them, and have people do a blind comparison of which interaction was better. If the older recorded interactions consistently fare better, then it's likely that ongoing tweaks (whatever the nature of those tweaks) have reduced effectiveness.

Reason #1 is that those languages are unreadable line noise to humans too. Fundamentally, almost all of the code written in array languages is made purposefully obtuse. Single-letter identifiers, no or little comments, dense code with minimal structure, etc...

Reason #2 is that there are very few examples of these languages on the web, and even more importantly: vanishingly few examples with inline comments and/or explanations. This isn't just because they're rare -- see reason #1 above.

Reason #3 is that LLMs can only write left-to-right. They can't edit or backtrack. Array-based languages are designed to be iterated on, rapidly modified, and even "code golfed" to a high degree.[1]

I've noticed that LLMs struggle with things my coworkers also struggle with: the "line noise" languages like grep, sed, and awk. Like humans, LLMs do well with verbose languages like SQL.

PS: I just tested GPT 4 to see if it can parse a short piece of K code that came up in a thread[2] on HN and it failed pretty miserably. It came close, but on each run it came up with different explanations of what the code does, and none of them matched the explanations in that thread. Conversely, it had no problems with the Rust code. And, err... it found a bug in one of my Rust snippets. Outsmarted by an AI!

[1] You can have an LLM generate code, and then ask it to make it shorter and more idiomatic. Just like a human touching up hastily written messy code, the LLM can fix its own mistakes!

[2] https://news.ycombinator.com/item?id=27220613

Ouch

>(...) our civilization will continue to be bottlenecked by problems of culture. OpenAI will succeed/fail for cultural reasons, not technical ones. Humanity will benefit from or be harmed by ai for cultural reasons, not technical ones.

It's capitalism. You can say it, it's OK. Warren Buffet isn't going to crawl out of your mirror and stab you.

All of the "cultural" problems around AI come down to profit being the primary motive driving and limiting innovation.