I was able to get ChatGPT 4 to produce a working websocket server in rust fairly quickly I know rust but had no experience with the networking crates or async runtimes.
Getting it to also serve HTTP, it fell into quite a few issues. Part of it was not telling me I needed to enable a feature, and part of it was that it's knowledge was quite a bit out of date.
I actually filmed that whole interaction here:
https://www.youtube.com/watch?v=TFsbMGSOeCY
I also was able to get it to make a working (though extremely basic/naive) SAT solver in J. J is pretty far out of the mainstream, so I had to go through MANY rounds of correcting it. (That was the only time I used up all my ChatGPT4 prompt quota for the 3-hour period.)
Since then, I've stumbled on the technique of presenting it with a rough plan or idea and then iteratively having it ask ME questions about what I posted, and summarizing everything we've agreed so far, rather than just immediately writing code. I find that it's actually pretty good at pointing out things I hadn't considered (security and scaling questions, for example), and asking for clarification.
Most recencly, I've started using it to help me get past the learning curve in languages where I'm not fluent at all (making an animation in Mathematica, and discovering how to do some simple things in smalltalk).
In general, I try to ask it for the most minimal/general example it can give me that shows what I actually want to know. For example, building on the rust web server thing, I asked it to give me the structure for building a restful API with certain endpoints (which "we" worked out "together" using the iterative design discussion method) but just leave the implementations blank, because they would be un-necessary detail for it, and I already knew how to do that part.
Aside from that, I've used ChatGPT 3 in a non-chatting context through GitHub Copilot, and that is a whole other ballgame: it's basically a plugin for Visual Studio Code that acts like a super-smart autocomplete.
It doesn't always guess what I'm about to type correctly right, and the wrong suggestions are occasionally annoying when I'm pausing to think through how to word a comment... But very often now when I start to write a function, several whole lines that were just a vague idea in my head suddenly appear on my screen exactly as I would have written them. (And I mean exactly, including my sometimes unusual code formatting choices...)
I'm still on the waitlist for copilot chat, which presumably is just ChatGPT but insta-trained on your codebase... I'm very much looking forward to trying it, though.