Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Advent of Code 2023's new AI/LLM Policy

adventofcode.com

Post reply on HN
Print view View original

11–20 of 133 posts

Re: Advent of Code 2023's new AI/LLM Policy

#11

Post by postalrat »

SimplyUnknown wrote: post #5
postalrat wrote: post #4

"(If you want to use AI to help you solve puzzles, I can't really stop you, but I feel like it's harder to get better at programming if you ask an AI to do the programming for you.)" I'd say that whatever "better at programming" is quickly evolving. AI is a tool like high level languages, IDEs, autocomplete or all the different types of code analyzers. How is ignoring a powerful tool like AI going to get you better a…

Maybe better programmer not so much, but I'd say the spirit of AoC is to crack a puzzle and translate the solution into code. Solving this using LLM will not help you getting better at solving puzzles, solving puzzles, or grasp the main concepts of a programming language. In that sense, ChatGPT is not a pedagogical asset in this particular instance.

Obviously they get to defined whatever better means in the context of AoC. But better at AoC does not mean a better programmer in any way.

Re: Advent of Code 2023's new AI/LLM Policy

#12

Post by Grambles »

postalrat wrote: post #4

"(If you want to use AI to help you solve puzzles, I can't really stop you, but I feel like it's harder to get better at programming if you ask an AI to do the programming for you.)" I'd say that whatever "better at programming" is quickly evolving. AI is a tool like high level languages, IDEs, autocomplete or all the different types of code analyzers. How is ignoring a powerful tool like AI going to get you better a…

I've always liked the premise that you should be competent one level below where you do your daily work. The tools you use should be more efficient for you, not necessarily the only way you can get your work done. If you don't know how to use a drill, you shouldn't use a drill press.

By that token, using AI to solve AoC really seems like one would be doing themselves a disservice. You might tell yourself you fully understand what the AI spits out, and maybe you do, but you won't build the same sort of comprehension that you would from solving it yourself. Nor will you retain it for as long.

Re: Advent of Code 2023's new AI/LLM Policy

#14

Post by trescenzi »

eddtries wrote: post #9

If LLMs are mostly working by predicting tokens from previously seen problems (am I wrong? An AI noob here!), could an LLM even tackle such weirdly written novel problems? It's hard to know because the solutions for previous AoC problems posted on Github/blogs could be the reason it's capable of solving them nowadays, but could it solve 2023 on day 1? Really keen to find out!

Yea honestly I think there would be value in them adding a checkbox that says "I used AI to mostly solve this problem" and maybe even list the AI. Could be an interesting way to see how the different AIs perform against such problems.

Re: Advent of Code 2023's new AI/LLM Policy

#15

Post by lsedgwick »

eddtries wrote: post #9

If LLMs are mostly working by predicting tokens from previously seen problems (am I wrong? An AI noob here!), could an LLM even tackle such weirdly written novel problems? It's hard to know because the solutions for previous AoC problems posted on Github/blogs could be the reason it's capable of solving them nowadays, but could it solve 2023 on day 1? Really keen to find out!

There are certainly emergent capabilities that come out of the training process of being able to predict next tokens, and it's shown that LLMs often can solve logic and math problems, riddles, etc. which are not in it's training data because deep in the internal layers of the model it has something resembling logic and reasoning abilities. This can be seen in the "benchmark" results of papers like the GPT-3 and InstructGPT papers, among many others, where they tried pretty hard to test the models against questions which were not part of the training.

This is also shown in the Codex paper, where they trained an LLM to write code and then watched it solve a number of code problems they handwrote originally to make sure the problems could not have been in the training data.

Try it out yourself, make up some little math word problems and ask chatGPT or something.

Of course, advent of code will be much more challenging problems, but to get help with some subcomponents of the problem a motivated participant would likely try to use the most recent, powerful, and advanced models which outperform the results from papers written a few years ago, and outperform the free chatGPT.

Re: Advent of Code 2023's new AI/LLM Policy

#16

Post by falsenapkin »

postalrat wrote: post #4

"(If you want to use AI to help you solve puzzles, I can't really stop you, but I feel like it's harder to get better at programming if you ask an AI to do the programming for you.)" I'd say that whatever "better at programming" is quickly evolving. AI is a tool like high level languages, IDEs, autocomplete or all the different types of code analyzers. How is ignoring a powerful tool like AI going to get you better a…

Programming != software engineering

> Software engineers who think they are better without AI may quickly be replaced with "lesser" software engineers who get things done much quicker.

Using AI to program is just ctrl-c ctrl-v software engineering, very clearly not the point of programming competitions. High level tools as you mention don't make you a better programmer, they make you a better software engineer.

I think the discussion you're raising is a good one it's just not relevant to a programming competition.

Re: Advent of Code 2023's new AI/LLM Policy

#17

Post by beepbooptheory »

postalrat wrote: post #4

"(If you want to use AI to help you solve puzzles, I can't really stop you, but I feel like it's harder to get better at programming if you ask an AI to do the programming for you.)" I'd say that whatever "better at programming" is quickly evolving. AI is a tool like high level languages, IDEs, autocomplete or all the different types of code analyzers. How is ignoring a powerful tool like AI going to get you better a…

I know we have seen some recent advancements with the Large Language Models, but I'm pretty sure they still can't, say, generate a working python Flask server for my specific REST API through only some prompts, can they? Because that would be incredible.

Re: Advent of Code 2023's new AI/LLM Policy

#18

Post by Lewton »

eddtries wrote: post #9

If LLMs are mostly working by predicting tokens from previously seen problems (am I wrong? An AI noob here!), could an LLM even tackle such weirdly written novel problems? It's hard to know because the solutions for previous AoC problems posted on Github/blogs could be the reason it's capable of solving them nowadays, but could it solve 2023 on day 1? Really keen to find out!

People we're solving AoC problems with ChatGPT (with no human interaction) last year so your questions have already been answered

Re: Advent of Code 2023's new AI/LLM Policy

#19

Post by rpearl »

reedf1 wrote: post #6

I can't help but feel part of the meaning drops out of it when there are carve outs for tools. If an AI can do it instantly, it starts to feel like pointless busy work, isn't that why we become programmers in the first place? If your intent is to make it a pure human-keyboard-code challenge, then I would have thought this would be a great opportunity to make the underlying questions harder, or to find the edge cases…

Computers completely dominate humans in, say, chess. A game between humans is still interesting, though--and isn't nearly as interesting if one person cheats with an engine.

In the same way, a human competition in advent of code is an interesting challenge--and wouldn't be, if people cheat with AI solvers.

Re: Advent of Code 2023's new AI/LLM Policy

#20

Post by simonw »

postalrat wrote: post #4

"(If you want to use AI to help you solve puzzles, I can't really stop you, but I feel like it's harder to get better at programming if you ask an AI to do the programming for you.)" I'd say that whatever "better at programming" is quickly evolving. AI is a tool like high level languages, IDEs, autocomplete or all the different types of code analyzers. How is ignoring a powerful tool like AI going to get you better a…

"I feel like it's harder to get better at programming if you ask an AI to do the programming for you"

I've been thinking about this for a full year now, and I'm ready to present my opinion: I think AI assistance makes it easier, not harder to get better at programming.

Sure, if you're lazy you won't learn anything because you'll get the AI to do all the work for you. But if you're that lazy I imagine you wouldn't have even tried to learn programming without access to AI assistance, so you're still doing better here!

If you're curious and dedicated to self-education, LLM assistance is a gift that never stops giving. You have to learn HOW to learn with it first, but once you've figured that out you can really fly with it.

Post reply on HN