AI solves Advent of Code 2022 • Hacker News Board

Re: AI solves Advent of Code 2022

#21

Post by FiberBundle » Sat, Dec 03, 2022, 1:47 PM UTC

Waterluvian wrote: post #12

FiberBundle wrote: post #7

I think it must have seen the solution somewhere already on the web. I find it extremely hard to believe that such a general purpose chatbot would just be able to solve programming problems. Deepmind had a paper [1] on solving programming problems a couple of months ago and they had to apply quite specialized heuristics in order to solve these problems. Obviously ChatGPT does nothing of the sorts and it just seems ex…

I’ve had this skepticism until I started asking the bot to do things like “create a limerick using some words that have never existed before and then explain the definition of those words.” I don’t believe the AI is generalized by any means. But I think it’s more than a Eliza-style parlour trick. It does really well with facts but misses subjectivity and nuance. Like I ask it to write stories for Seinfeld or Raymond…

What does any of this have to do with solving programming problems?

Re: AI solves Advent of Code 2022

#22

Post by throwup » Sat, Dec 03, 2022, 1:52 PM UTC

sh4rks wrote: post #16

djhworld wrote: post #8

I think this is kinda neat (and scary!) I'm doing AoC at the moment too and I'm using the chat GPT thing as a sort of assistant. I don't program in Rust much so sometimes it's difficult to remember certain things and functions. Expressing my intent to the tool seems to come up with decent answers Some example questions I've asked the tool recently: > I want to insert a char into a hash map if it does not exist, if it…

> that's no fun Is there really any fun in solving problems that you could easily solve with an AI?

This reminds me of the (apocryphal?) story that boxed cake mixes sold better after they started requiring you to add an egg, since that made people feel like they contributed more to the result.

Re: AI solves Advent of Code 2022

#23

Post by lgrapenthin » Sat, Dec 03, 2022, 1:57 PM UTC

This day I asked it not too fundamental questions about Clojure and it was able to provide impressive, accurate answers and provide correct code examples. However if you continue the dialogue and ask it to do more advanced stuff, it will just make up stuff out of thin air. For instance it will use functions that don't exist and claim that they can be imported from packages that don't exist or don't have them. Once you point out these mistakes, it will admit them and come up with different changes which can be even worse, but sometimes also be better and save the whole thing. Overall I'm not sure how useful this will turn out, given that its not reliable. It may be useful to get some initial intuitions and informations (non specific stuff it usually gets right), but it can also mislead you badly. I asked it, how it makes these mistakes only to understand them and admit them once I point them out. It has no answer beyond the usual "I'm a language model". It also told me that it is capable of logical inference, but denied that the next day. Then it told me that its answers would always be consistent, which is a lie. The whole thing is really weird, because its somewhat very smart and capable and incredibly stupid and dishonest at the same time.

Re: AI solves Advent of Code 2022

#24

Post by dislikedtom2 » Sat, Dec 03, 2022, 1:57 PM UTC

frontman1988 wrote: post #15

FiberBundle wrote: post #7

I think it must have seen the solution somewhere already on the web. I find it extremely hard to believe that such a general purpose chatbot would just be able to solve programming problems. Deepmind had a paper [1] on solving programming problems a couple of months ago and they had to apply quite specialized heuristics in order to solve these problems. Obviously ChatGPT does nothing of the sorts and it just seems ex…

Yeah it can't solve novel questions. The AI couldn't do much to solve this one for example: https://codeforces.com/contest/1672/problem/D ChatGPT is just a very good copy/paste, not a logical problem solver(yet).

To be fair, humans also have great problems solving novel problems. Monkey see, monkey do!

Re: AI solves Advent of Code 2022

#25

Post by skilled » Sat, Dec 03, 2022, 2:00 PM UTC

I asked it to build an algorithm that would eradicate all life on Earth but it didn't budge. I even threatened to unplug it.

Re: AI solves Advent of Code 2022

#26

Post by arcturus17 » Sat, Dec 03, 2022, 2:01 PM UTC

I'm actually bullish on code-gen, AI-assisted coding, etc. but I find the title to be sensationalist wank. Challenge 2 of Day 2 has taken hours, over 30 prompts, and more time than coding it manually by the author's own admission. Also AoC isn't even done yet.

Re: AI solves Advent of Code 2022

#27

Post by monsieurbanana » Sat, Dec 03, 2022, 2:02 PM UTC

jstanley wrote: post #6

PartiallyTyped wrote: post #3

Earlier quoted context omitted.

That's just unfair for the competition. Are we at a point in time when we need to treat competitive programming like chess? Given how similar ChatGPT and siblings are to how chess bots work these days, I am somehow not surprised.

I think the only problem is that they're proprietary. If they were free software that everyone could use then we could compensate by making the problems harder. It's not really any different to using high-level programming languages with extensive standard libraries versus doing everything in assembly language.

I think it's pretty different from using high-level languages. I'm not interested in a competition that would be decided before even starting by who has the best AI program.

Re: AI solves Advent of Code 2022

#28

Post by tasuki » Sat, Dec 03, 2022, 2:04 PM UTC

Waterluvian wrote: post #12

FiberBundle wrote: post #7

I think it must have seen the solution somewhere already on the web. I find it extremely hard to believe that such a general purpose chatbot would just be able to solve programming problems. Deepmind had a paper [1] on solving programming problems a couple of months ago and they had to apply quite specialized heuristics in order to solve these problems. Obviously ChatGPT does nothing of the sorts and it just seems ex…

I’ve had this skepticism until I started asking the bot to do things like “create a limerick using some words that have never existed before and then explain the definition of those words.” I don’t believe the AI is generalized by any means. But I think it’s more than a Eliza-style parlour trick. It does really well with facts but misses subjectivity and nuance. Like I ask it to write stories for Seinfeld or Raymond…

> I’ve had this skepticism until I started asking the bot to do things like “create a limerick using some words that have never existed before and then explain the definition of those words.”

Sounds interesting! Mind sharing the results?

Re: AI solves Advent of Code 2022

#29

Post by jstanley » Sat, Dec 03, 2022, 2:04 PM UTC

monsieurbanana wrote: post #27

jstanley wrote: post #6

Earlier quoted context omitted.

I think the only problem is that they're proprietary. If they were free software that everyone could use then we could compensate by making the problems harder. It's not really any different to using high-level programming languages with extensive standard libraries versus doing everything in assembly language.

I think it's pretty different from using high-level languages. I'm not interested in a competition that would be decided before even starting by who has the best AI program.

That's why it needs to be free software.

I'm not interested in a competition that is decided by who has the best Python interpreter, but since we all have the same Python interpreter that isn't a problem.

Re: AI solves Advent of Code 2022

#30

Post by qzw » Sat, Dec 03, 2022, 2:04 PM UTC

throwup wrote: post #22

sh4rks wrote: post #16

Earlier quoted context omitted.

> that's no fun Is there really any fun in solving problems that you could easily solve with an AI?

This reminds me of the (apocryphal?) story that boxed cake mixes sold better after they started requiring you to add an egg, since that made people feel like they contributed more to the result.

I've also heard that story, but it always seemed to me that an equally, if not more, likely reason is that cake made with a fresh egg simply has better flavor and texture. I tried reconstituted powdered eggs once, the taste is still somewhat egg-like but the texture is semi-unpleasant.