Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Testing GPT 4's code-writing capabilities with some real world problems

tylerglaiel.substack.com

Post reply on HN
Print view View original

281–290 of 677 posts

Re: Testing GPT 4's code-writing capabilities with some real world problems

#281

Post by LeanderK »

training ML model to code is a very interesting challenge. I am surprised by GPTs ability to code, given that it, as I understood it, has basically no tools at the ready. I am convinced that it is way harder to code without debugging and other interactive features both for a human and for a machine. Keep in mind that GPT could not have learned to simulate the code internally given its fixed runtime.

I think ML models need to learn how to interact with our tools (compiler, debugger etc.) to really be effective at coding. That's hard.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#282

Post by danuker »

danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

We could have this already.

- Have a cron job that checks email from a certain sender (CRM?). - Instruct an OpenAI API session to say a magic word to reply to an e-mail: "To reply, say REPLY_123123_123123 followed by your message." - Pipe received email and decoded attachments as "We received the following e-mail: " to the OpenAI API. Make it a 1-click action to check if there is a reply and confirm sending the message. If it does not want to send a message, read its feedback.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#283

Post by jorgeleo »

jmuguy wrote: post #256
danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

This is exactly why I'm so tired of these "can AI write code" think pieces. I assume people writing this crap aren't actual developers. Maybe its management fan-fiction.

"Maybe its management fan-fiction"

:D Priceless!

Re: Testing GPT 4's code-writing capabilities with some real world problems

#284

Post by jobstijl »

tinco wrote: post #181
scg wrote: post #19

As a human programmer I didn't quite understand the problem statement until I read the whole article and the tests. I believe the goal is to find a path with the fewest possible "fire" cells and the minimum cost as a tie breaker. The cost of a path is the sum of its cells' cost and it can't be greater than 5. If I understood the assignment correctly, I don't think the problem statement is equivalent to what's include…

By the time you've formulated the problem as: "Give me the shortest route with a cost of 5 or lower that doesn't go through fire, and if that doesn't exist, the shortest route with a cost or 5 or lower that goes through fire." Then you've basically formulated the algorithm as well. That's also precisely where one of the programmer's greatest challenges lies, to carefully translate and delineate the problem. I agree i…

Wouldn't a better formulation be: "Give me the shortest route with a cost of 5 or lower with the minimum fire tiles in the path necessary" Since your formulation doesn't care about the amount of fire tiles in case there is no other solution?

Re: Testing GPT 4's code-writing capabilities with some real world problems

#285

Post by [deleted] »

danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

[deleted]

Re: Testing GPT 4's code-writing capabilities with some real world problems

#286

Post by sequoia »

I had a teacher with a sign on her door that read:

    Technology will not replace teachers
    But teachers who use technology will replace those who don't
s/teachers/programmers/ and s/technology/AI/ and this sounds about right. It may become typical or even required to leverage AI to write code more efficiently.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#287

Post by dmm »

yucky wrote: post #169
nwienert wrote: post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

> I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Equally fascinating is all of the "this is fine" posts from programmers suddenly realizing they are not the gods they once thought. But fret not, programming is not the first industry that has been automated into a shell of itself. Yes, the industr…

> Just as farmers had to learn industrial jobs and then miners and autoworkers had to "learn to code"

The transition from agrarian to industrial societies was extremely painful and arguably it was centuries before the people affected were better off.

> Humans are resilient and will adapt.

Based on recent events I think it's more likely people will elect extremist politicians who promise quick fixes while blaming a convenient out-group for everything.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#288

Post by cosmodisk »

danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

So I recently finished a job, where I had to create a custom POST endpoint on Salesforce,so it'd take a simple JSON payload, apply some logic and save to the database. The job itself was a few hours with tests,etc. Well guess what, almost 100 emails and two months later,the project is still not finished, because of the simple middleware that was supposed to send the JSON to my endpoint and is as basic as my endpoint. ChatGPT can write the code, but all the BS inbetween will need humans to deal with.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#289

Post by sgarland »

lionkor wrote: post #252
sgarland wrote: post #241

There are plenty of edge cases where it fails. However, the one thing that made me think it actually knows (for certain definitions of the word "knows") what it's doing was asking it to re-write a non-trivial SQL query into equivalent relational algebra. I created a simplified schema from Northwind [0], gave it the CREATE TABLE statements for tables, and then some sort-of-TSV files for the values. It was able to not…

Could you share the results?

Here's a gist [0]. You can see that it made a couple of minor semantic mistakes, which it corrected. It also regressed a bit at the end, but it _was_ able to correctly parse the (admittedly simple) math in a WHERE predicate and return a partially valid tuple, with respect to pricing.

[0]: https://gist.github.com/stephanGarland/ed18f8f8fdc63a0b997f9...

Re: Testing GPT 4's code-writing capabilities with some real world problems

#290

Post by sequoia »

One other question: can GPT-4 reliably modify code? In A Philosophy of Software Design the author points out that code is written once and modified possibly dozens of times, so ease of maintainability/reading is more important than ease of writing.

I wonder whether a) AI can reliably modify code and b) whether AI can reliably write code that is able to be easily modified by humans. If AI starts spitting out machine code or something, that's not useful to me even if "it works."

Post reply on HN