Testing GPT 4's code-writing capabilities with some real world problems

I'm actually taken back by how well it's doing; including providing me some refreshers on stuff I forgot how it should work.

I can see it failing at solving complex problems, but like the blog post mentions, most programming isn't new or hard problems.

This is particularly powerful when you're producing something you've done before, but in a completely different language/stack. You just guide GPT-4 towards the goal, you roughly know the methods needed to get to the end goal and just watch your assistant do all the dirty work.

Looking back, I came from a world of floppy disks; I left them behind for zip disks and CDs, then portable disks and cloud storage. I also came from dialup Internet, I left it behind for ADSL then fibre. I feel this is a tangential point here too, where AI, whatever it ends up being called, will become a fulltime assistant making our lives easier; so that we can focus on the hard parts and the creative problem solving. What are we leaving behind? For me, mostly Stack Overflow and Google.

You'd be silly to ignore it and palm it off. It's a big deal.

Each A* location stores where it comes from, how long it takes to get to it, and how many fires it passed through to get there. The algorithm only considers fire cells neighbors if the current number of fires passed through is less than the current fireWillingness global.

1. count fire tiles within movement range

2. run A* from src to dst completely avoiding fire

3. if we can reach then that's the solution

4. if we can't reach, increase fireWillingness to 1, re-run A* on the board

5. keep increasing fire-willingness until the A* results don't change, or we can now reach the dst.

This works because a low fire path is always better than a high fire path. And increasing fire-tolerance will only shorten the paths from src to dst.

The most difficult problem that I have asked GPT-4 to solve was writing a parser for the Azure AD query language in a niche programming language and it did that just fine (I did have to copy paste some docs into the prompt).

I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating.

Thinking back to my first job out of college as a solid entry level programmer. ChatGPT couldn’t have done what I was doing on day 2. Not because it’s so hard or I’m so special. Just because programming is never just a snippet of code. Programming is an iterative process that involves a CLI, shell, many runtimes, many files, a REPL, a debugger, a lot of time figuring out a big codebase and how it all links together, and a ton of time going back and forth between designers, managers, and other programmers on your team, iterating in problems that aren’t fully clear, getting feedback, testing it across devices, realizing it feels off for reasons, and then often doing it and redoing it after testing for performance, feel, and feedback.

Often it’s “spend a whole day just reading code and trying to replicate something very tricky to find” and you only produce a single tiny change deep in the code somewhere. GPT is absolutely terrible at stuff like this.

And yes, often it is finding new solutions that aren’t anywhere on the internet. That’s the most valuable programming work, and a significant % of it.

Feel like there’s 10 more points I could make here but I’m on my phone and don’t like wasting too much time on HN. But man, what a disappointment of critical thinking I’ve seen in this specific topic.

I think the author is onto something – while AI might not be able to program per se, it can certainly be handed a code snippet and then use its huge corpus of Internet Learning™ to tell you things about it, code that looks like it, and ways (people on the Internet think) it might be solved better.

In that sense, it isn't replacing the programmer; it's replacing IDE autocomplete.