Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Advent of Code 2023's new AI/LLM Policy

adventofcode.com

Post reply on HN
Print view View original

51–60 of 133 posts

Re: Advent of Code 2023's new AI/LLM Policy

#51

Post by reedf1 »

lsy wrote: post #22

Maybe I'm a dinosaur but I'm utterly mystified by folks who are interested in making the leaderboard or even solving AoC puzzles with AI assistance. To me it feels like going to a 5k charity run, taking a motorcycle to the finish line, and parading your gold medal around. Like, who are you impressing? And are you even proud of yourself? I understand the appeal of using the best and fastest means when there are real-w…

I guess it's not really an apt analogy. It's more like if a the challenge was to get from city A to city B in the fastest time by any means necessary. People would run it until the horse was domesticated, and then people would ride them until the motorcycle or car was invented, and so on.

That's not to say that you can't still do a foot race between A and B. It's just now you have to constrain it, it has become something different.

Re: Advent of Code 2023's new AI/LLM Policy

#52

Post by yunwal »

foooorsyth wrote: post #35
Art9681 wrote: post #21

Seems like they need to adapt the level of difficulty to the new world of developer tooling. An AI may be able to write a starter template and used wisely can move the project forward, but it still requires knowledge and expertize to build a system of systems even with the AI. This is an opportunity to change our expectations of developer potential. Up the stakes and innovation will soon follow. Edit: This is also an…

>Seems like they need to adapt the level of difficulty "They" is one guy - Eric Wastl. He has a very specific style and hey may not want to change the entire style of the challenge just because some dorks want to cheat. I hope that he doesn't change it -- AoC is very approachable, especially inside the first ~15 or so days. It's a charming set of puzzles that people look forward to. I don't want it to become an elite…

> With that said, I don't think LLMs will be helpful with some of the tougher problems. It's a hard thing to test, because I assume CoPilot would do well with previous year challenges due to the massive amount of public code available on the same problems.

I'm not 100% sure on this, but I think copilot isn't trained on anything past January 2022 (the gpt-3.5 and 4 cutoff dates), so I think it still shouldn't do well with AoC questions from 2022

Re: Advent of Code 2023's new AI/LLM Policy

#53

Post by miki123211 »

postalrat wrote: post #4

"(If you want to use AI to help you solve puzzles, I can't really stop you, but I feel like it's harder to get better at programming if you ask an AI to do the programming for you.)" I'd say that whatever "better at programming" is quickly evolving. AI is a tool like high level languages, IDEs, autocomplete or all the different types of code analyzers. How is ignoring a powerful tool like AI going to get you better a…

This same sentiment we now have against AIs has existed in the past, mostly against what was then called "high-level" languages, that is C, Pascal and maybe C++. Similar opinions were once held against memory-wasting languages (those with garbage collection) and languages that are too easy (the ones with dynamic typing and no pointers).

Programmers that write "slow C" instead of "fast assembly" were once regarded as lazy and incompetent, same for programmers that write "slow Java instead of fast C" or "slow Python instead of fast Java."

Re: Advent of Code 2023's new AI/LLM Policy

#54

Post by setgree »

lsy wrote: post #22

Maybe I'm a dinosaur but I'm utterly mystified by folks who are interested in making the leaderboard or even solving AoC puzzles with AI assistance. To me it feels like going to a 5k charity run, taking a motorcycle to the finish line, and parading your gold medal around. Like, who are you impressing? And are you even proud of yourself? I understand the appeal of using the best and fastest means when there are real-w…

> To me it feels like going to a 5k charity run, taking a motorcycle to the finish line, and parading your gold medal around. Like, who are you impressing? And are you even proud of yourself?

Weirdly enough, basically this happens all the time at races [0]. Not a motorcycle, but a runner in my local competitive scene used a bicycle to fake her strava data for a HM in 2017 [1].

I don't get it either.

[0] https://www.wired.com/story/marathon-investigation-cheaters-...

[1] https://washingtonpost.com/news/morning-mix/wp/2017/02/23/ho...

Re: Advent of Code 2023's new AI/LLM Policy

#55

Post by Herbstluft »

reedf1 wrote: post #6

I can't help but feel part of the meaning drops out of it when there are carve outs for tools. If an AI can do it instantly, it starts to feel like pointless busy work, isn't that why we become programmers in the first place? If your intent is to make it a pure human-keyboard-code challenge, then I would have thought this would be a great opportunity to make the underlying questions harder, or to find the edge cases…

> If an AI can do it instantly, it starts to feel like pointless busy work

If you can't solve the problem in a few seconds/minutes you will be able to just look up or copy/paste the answer for every AoC day from the associated solutions thread.

So in other words if you view it like that there was already a better "tool" to very easily "solve" any AoC problem: internet search.

If all you care about is getting a number to put into the website to get a checkmark that is.

Re: Advent of Code 2023's new AI/LLM Policy

#56

Post by Tainnor »

lsedgwick wrote: post #15
eddtries wrote: post #9

If LLMs are mostly working by predicting tokens from previously seen problems (am I wrong? An AI noob here!), could an LLM even tackle such weirdly written novel problems? It's hard to know because the solutions for previous AoC problems posted on Github/blogs could be the reason it's capable of solving them nowadays, but could it solve 2023 on day 1? Really keen to find out!

There are certainly emergent capabilities that come out of the training process of being able to predict next tokens, and it's shown that LLMs often can solve logic and math problems, riddles, etc. which are not in it's training data because deep in the internal layers of the model it has something resembling logic and reasoning abilities. This can be seen in the "benchmark" results of papers like the GPT-3 and Instr…

> Try it out yourself, make up some little math word problems and ask chatGPT or something.

I have, and chatGPT simply sucks at logical reasoning in the way that is required for mathematics. It's extremely easy to make it trip up by just formulating questions in a slightly nonstandard way (so that it doesn't exactly match what it has been trained on), e.g.

> User: prove that there are only finitely many primes

> ChatGPT: Actually, it is not possible to prove that there are only finitely many prime numbers because this statement is false. There are infinitely many prime numbers, and this has been proven by the ancient Greek mathematician Euclid over 2,000 years ago. Euclid's proof is based on the principle that if you assume there are only finitely many prime numbers, you can always find a new prime by multiplying all known primes together and adding 1. This new number will either be prime itself or have a prime factor not present in the original list of primes, which contradicts the assumption of finiteness.

> User: prove that there exists a finite list of integers that contains all integers which are only divisible by itself and 1

> ChatGPT: You're referring to prime numbers. The statement is essentially the definition of prime numbers. A prime number is a positive integer greater than 1 that is only divisible by 1 and itself. There is no need to prove this statement because it defines what prime numbers are. While there are infinitely many prime numbers, they can be listed, but this list will be finite in the sense that there are infinitely many prime numbers, but they can all be enumerated within an infinite sequence. Some of the first prime numbers are 2, 3, 5, 7, 11, 13, and so on.

ChatGPT gets the first question right because something like that is undoubtedly in its training corpus, but it gets utterly confused by the second question (which is the same statement), to the point that its answer belongs to the "not even wrong" category.

I don't know where this myth comes from that LLMs are magically good at maths. They're not.

Re: Advent of Code 2023's new AI/LLM Policy

#57

Post by [deleted] »

yunwal wrote: post #47
lsy wrote: post #22

Maybe I'm a dinosaur but I'm utterly mystified by folks who are interested in making the leaderboard or even solving AoC puzzles with AI assistance. To me it feels like going to a 5k charity run, taking a motorcycle to the finish line, and parading your gold medal around. Like, who are you impressing? And are you even proud of yourself? I understand the appeal of using the best and fastest means when there are real-w…

> Is it fun to hit over and over? Last year, AI was only able to solve the first 3 or 4 problems, so it's really not like using AI takes all human ingenuity out of the equation. What it does is enable you to autocomplete sub-problems that are already solved problems so you can focus on the good stuff. I'd personally prefer if there were 2 leaderboards - one with AI, and one without. Motorcycle races are fun too :)

[deleted]

Re: Advent of Code 2023's new AI/LLM Policy

#58

Post by mckn1ght »

yunwal wrote: post #47
lsy wrote: post #22

Maybe I'm a dinosaur but I'm utterly mystified by folks who are interested in making the leaderboard or even solving AoC puzzles with AI assistance. To me it feels like going to a 5k charity run, taking a motorcycle to the finish line, and parading your gold medal around. Like, who are you impressing? And are you even proud of yourself? I understand the appeal of using the best and fastest means when there are real-w…

> Is it fun to hit over and over? Last year, AI was only able to solve the first 3 or 4 problems, so it's really not like using AI takes all human ingenuity out of the equation. What it does is enable you to autocomplete sub-problems that are already solved problems so you can focus on the good stuff. I'd personally prefer if there were 2 leaderboards - one with AI, and one without. Motorcycle races are fun too :)

I agree that two leaderboards would be a fine middle ground. Just put a checkbox next to the submission box.

Re: Advent of Code 2023's new AI/LLM Policy

#59

Post by adverbly »

I wonder how long until they drop the leaderboard entirely.

Most people are not realistically shooting for a spot on the leaderboard.

It adds novelty value to know that some geniuses are out there who can solve a problem in 20 minutes which takes me over 4 hours, but the main value is my private leaderboard, and the enjoyment of solving the puzzles.

Re: Advent of Code 2023's new AI/LLM Policy

#60

Post by BananaaRepublik »

lsy wrote: post #22

Maybe I'm a dinosaur but I'm utterly mystified by folks who are interested in making the leaderboard or even solving AoC puzzles with AI assistance. To me it feels like going to a 5k charity run, taking a motorcycle to the finish line, and parading your gold medal around. Like, who are you impressing? And are you even proud of yourself? I understand the appeal of using the best and fastest means when there are real-w…

Because AoC has a leaderboard? Whenever you have a leaderboard of any kind, there will be people trying to get up there.

It's also likely that a lot of AI users will be skilled developers too. They already know they can solve the problems, they also think they deserve the leaderboard rankings, so using AI is ok.

Post reply on HN