31–40 of 82 posts
The blog attempts to solve 3 of 24 (thats 12.5 %) of advent of code 2022, and if you read along you'll see OP only had success on the first task of day 1, which would make a more correct title as "AI solves 2% of Avent of Code 2022" (assuming 2 tasks each day).
Do note that AoC tends to start with hello-world style tasks and increase in difficulty.
I think it must have seen the solution somewhere already on the web. I find it extremely hard to believe that such a general purpose chatbot would just be able to solve programming problems. Deepmind had a paper [1] on solving programming problems a couple of months ago and they had to apply quite specialized heuristics in order to solve these problems. Obviously ChatGPT does nothing of the sorts and it just seems ex…
I’ve had this skepticism until I started asking the bot to do things like “create a limerick using some words that have never existed before and then explain the definition of those words.” I don’t believe the AI is generalized by any means. But I think it’s more than a Eliza-style parlour trick. It does really well with facts but misses subjectivity and nuance. Like I ask it to write stories for Seinfeld or Raymond…
rappers disagree
Title is mildly misleading, to say the least. The blog attempts to solve 3 of 24 (thats 12.5 %) of advent of code 2022, and if you read along you'll see OP only had success on the first task of day 1, which would make a more correct title as "AI solves 2% of Avent of Code 2022" (assuming 2 tasks each day). Do note that AoC tends to start with hello-world style tasks and increase in difficulty.
The world in 10 years will be hard to believe for many of us. Only issue I see now is that the mindshare today is more towards computing. Materials science, robotics, biotech are lagging behind compared to the advances in computing.
What advances in computing? As we approached physical limits we've seen cpu and gpu stopped scaling for a couple of years already [1]. The new models just run on higher frequencies and consume unpropornally more wattage.
Quantum computing is a joke [2]. AI is just a overhyped rephrasing of machine learning.
This is rather hinting about the next decade of no technological progress.
And don't get me started on the effects of recession.
1: https://arstechnica.com/gaming/2022/09/do-expensive-nvidia-g...
Earlier quoted context omitted.
I’ve had this skepticism until I started asking the bot to do things like “create a limerick using some words that have never existed before and then explain the definition of those words.” I don’t believe the AI is generalized by any means. But I think it’s more than a Eliza-style parlour trick. It does really well with facts but misses subjectivity and nuance. Like I ask it to write stories for Seinfeld or Raymond…
> I’ve had this skepticism until I started asking the bot to do things like “create a limerick using some words that have never existed before and then explain the definition of those words.” Sounds interesting! Mind sharing the results?
This day I asked it not too fundamental questions about Clojure and it was able to provide impressive, accurate answers and provide correct code examples. However if you continue the dialogue and ask it to do more advanced stuff, it will just make up stuff out of thin air. For instance it will use functions that don't exist and claim that they can be imported from packages that don't exist or don't have them. Once yo…
This was perhaps a very hard problem for an LLM, as the Packer tool’s nature is to manage layers of context. Environment variables passed through templates then passed to scripts which themselves might be in other frameworks. So in this case it to be confused about what was Ansible syntax and what was Packer.
So the bot seems to have different failure modes than humans. Distinguishing context layers seems to be a weak point. And an answer that is a wild guess looks as authoritative as a solid answer. But it’s still extremely impressive.
While I did say 80%, the 20% is most crucial and without it, the code is useless. For example, it doesn't understand scope and assignment in Elixir. Getting it to write in more pure functional style is close to impossible (or I just haven't found a good prompt).
I spent a good 30 minutes trying to get it to generate a working code for Day 1 Part 1. No nudging, just errors and AoC answers (too high, too low) and it never got there. Even after I started to correct its mistakes, like "your Enum.reduce/3 return is not assigned anywhere", it couldn't get a solution and started reverting to previous answers.
I think what's going to happen here, is that these models will shift a meaning of "boilerplate". If I can write the scaffolding and basic architecture easily, I'm happy to use them.
Also, I do wonder how is all of this going to play out if it has access to Input, REPL and just learns.
Earlier quoted context omitted.
I’ve had this skepticism until I started asking the bot to do things like “create a limerick using some words that have never existed before and then explain the definition of those words.” I don’t believe the AI is generalized by any means. But I think it’s more than a Eliza-style parlour trick. It does really well with facts but misses subjectivity and nuance. Like I ask it to write stories for Seinfeld or Raymond…
> things that are objective, like code or rhymes rappers disagree
I think it must have seen the solution somewhere already on the web. I find it extremely hard to believe that such a general purpose chatbot would just be able to solve programming problems. Deepmind had a paper [1] on solving programming problems a couple of months ago and they had to apply quite specialized heuristics in order to solve these problems. Obviously ChatGPT does nothing of the sorts and it just seems ex…
This was the first solution, submitted very quickly after the problem was published.
Replit included so you can verify: https://twitter.com/thiteanish/status/1598217824392351744?t=...
I plan on going back and catching up on the other days