I've spoken about my troubles getting as much mileage out of LLMs as before while being vague about my industry and, well, this hits the nail in the head.
> because the tooling used in the gaming industry is as unfriendly to LLM coding as it gets: Heavily visual scripting based
At the start of the LLM craze, we (as a team) put ChatGPT to test with Godot. It wasn't very successful in that, IIRC, GDScript 2 was just released and ChatGPT's training corpus was so obviously based on GDScript 1.
We could make ChatGPT outline steps of how to accomplish things in Godot, sort of like getting a customized tutorial. When code isn't involved, ChatGPT was okay and Gemini seemed to fare better. Based on vibes, I think Gemini had a marginally better handle on GDScript 2 for some reason.
I've been trying it lately with Claude still with mixed results. I had to install a few skills/extensions for it (can't tell you which as I basically just blindly followed what our AI advocate recommended). Sometimes it works but when it doesn't it's harder to put a finger on why. Overall I prefer the DX of generating customized tutorials with ChatGPT.
> extremely reflection heavy
Big time. And IME we don't even have to deal with textbook reflection here. Game entities are so convoluted (remember the Fallout 3 Train NPC, the stories of how Skyrim works, etc.) that it is really pushing inheritance and OO to contortions it shouldn't be doing.
Dirty confession: in our game we have this GIANT switch-statement dealing with game objects. It happens in a handful of places, for different game object types. LLMs (Copilot and Codex) could generate the monkey code of adding a switch case and even writing the body but sooner than later, when the new objects have to interact with others, LLMs just can't reason around it. Not to mention the hundred edge cases you have to consider!
And before some smart-ass comments: in my almost-decade of dealing with this code base there have been a handful of attempts to "refactor" these switch-statements, always a newcomer's enthusiastic effort. I'm proud to say, I've managed to slay one of the giants, the only successful effort to my knowledge, and this only happened last year. But I did so by basically delegating to another of the giants; they turned out to be twins and we could do without one. The dirty way is the clean way because the alternative contains Lovecraftian geometries.
Claude has been great for finding edge-case bugs but that's only once the code has been written properly. Generally if QA reports a bug on a pre-release feature, it's at most 50/50 if Claude can debug it. But if it's a player report/live incident, I'd say Claude's chances goes up to around 80%.
All that said,
> if you are in the game industry, you probably have nothing to worry about in terms of LLM coding replacing you
Hah-hah. Look, I agree with you but please get in touch with upper management. As an engineer I'm confident on the value I bring to the table but I'm not sure management "gets" that. Like, no matter how I tell them what features I shipped, what infra I built, etc., it doesn't come across to them that LLMs would not have been able to automate that output!