Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Testing GPT 4's code-writing capabilities with some real world problems

tylerglaiel.substack.com

Post reply on HN
Print view View original

231–240 of 677 posts

Re: Testing GPT 4's code-writing capabilities with some real world problems

#231

Post by mtlmtlmtlmtl »

kykeonaut wrote: post #205
danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

To paraphrase Harold Abelson: Computer Science is not really very much about computers. And it’s not about computers in the same sense that physics isn’t really about particle accelerators, and biology is not really about microscopes and petri dishes. It is about formalizing intuitions about process: how to do things [0]. [0]: https://www.driverlesscrocodile.com/technology/the-wizard-1-... .

Watching Abelson give that lecture(on video I mean, first lecture of SICP IIRC) made a lot of things click in my head as a neophyte programmer. Even after one lecture my understanding of the nature of computation had grown immensely.

He and Sussman are great at distilling and explaining abstract concepts in a clear and precise way.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#232

Post by SuoDuanDao »

caddemon wrote: post #68
dragonwriter wrote: post #40

Earlier quoted context omitted.

> I think most people see where the puck is going Overinflated hype about “where the puck is going” being wrong is…not a new phenomenon. And the non-tech media (traditional and social, and particularly a whole lot of the elite commentariat that spans both more than the “hard” news side of traditional media, though that is influenced too) perspective on this is driven quite disproportionately by the marketing message…

You know the shoeshine boy indicator story [1]? That's the vibe I've been getting with ChatGPT lately, I have multiple old high school teachers with 0 computational background who hardly ever spoke about ML before posting regularly on LinkedIn about ChatGPT the last couple months. The part that gets me is they aren't just aimlessly making posts, they're getting involved with webinars targeted at educators in their re…

Teaching is going to be a career affected early by chatgpt because students will use it (are using it) to do assignments. A good teacher doesn't need to know much about machine learning to think about how that will affect his job, he just needs to know his students and have an idea how to work with them given the new tools they have.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#233

Post by suslik »

DaiPlusPlus wrote: post #154
anonzzzies wrote: post #121

Earlier quoted context omitted.

HN echo chamber. Generally the better/best programmers hang around here; your day 2 was probably better than many coders hope to achieve in their whole lives. It is replacing people already; I personally know about 10 people who have been fired or assigned to a completely different position because gpt did a faster and better or equal job. So ‘not anyone’ is simply nonsense; I am only one person, there will be many m…

> I personally know about 10 people who have been fired or assigned to a completely different position because gpt did a faster and better or equal job Please elaborate. And, if true, this would be a major news story that Vox or any number of major newspapers would love to write about - so have you approached the media about this? If not, why not?

Manager here. Did not fire anyone (and hope not to), but I am starting to look into implementing GPT-4 as a part of our development practice and, potentially, reduce future hiring. This might have a positive impact on our engineering budgets, which are always tight. Many of my colleagues are doing the same - this has been a cooler topic with other managers for weeks now (strange how progress is now measured in weeks).

At this point, this would only affect engineers who don't understand our subject area (biotech) and are relatively junior (in the sense that their output is not much better than a GPT4 output reviewed by a more senior dev).

Re: Testing GPT 4's code-writing capabilities with some real world problems

#234

Post by hackinthebochs »

MrDresden wrote: post #80
pewpew2020 wrote: post #47

Earlier quoted context omitted.

I feel it's the average programmer who is enamoured with GPT. The programmers working on hard stuff know that AI writing programs is still far away.

It is not just those who are working on hard problems who are of that opinion. Good luck having a ML model understand a 20 year old undocumented dataformat developed inhouse at a specific research lab to be used in their proprietary systems which are also undocumented and are a part of a spiderweb of interconnected systems at that lab (I have worked at this particular lab). It will be a long time (if ever) until a ML…

>It will be a long time (if ever) until a ML model will be able to handle these situations (and I hazard a guess that most of the worlds active code is something akin to this).

ChatGPT can do that right now. Just provide it example data in the input and it can extrapolate the format and interact with this abstraction (i.e. describe the format, write code to it, etc). LLMs don't just remix data it has seen before, they perform in-context learning. This means abstracting out patterns in the input and then leveraging it in generating output.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#235

Post by the_af »

danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

> Basically, 99% of the effort is NOT WRITING CODE

I've come to realize this is true in more contexts than I would like. I've encountered way too many situations where "sitting on your hands and not doing anything" was the right answer when asked to implement a project. It turns out that often there is radio silence for a month or so, then the original requester says "wait, it turns out we didn't need this. Don't do anything!"

Re: Testing GPT 4's code-writing capabilities with some real world problems

#236

Post by caleb-allen »

anonyfox wrote: post #195
guax wrote: post #189

Earlier quoted context omitted.

I have the same feeling, people are very concentrated on the ability of this AI generators to create working code from super specific and well formed prompts. When in reality, figuring out what the prompt should be accounts for 80% of the job.

don't fall into this mental trap. you can get into recursion quite easily here, and figuring out what to prompt can start from simple general questions - and there is no need for a developer at all, aside from the current limitations of copy/paste/run workflow has to be done manually

It's astonishing to see the goalposts move so quickly. The cope of "well, okay, it can do that, but that's not even the hard part!" when just a year ago this entire product was almost unimaginable.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#237

Post by amrb »

danwee wrote: post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

..what if it was two AI's deciding XD

Re: Testing GPT 4's code-writing capabilities with some real world problems

#238

Post by theshrike79 »

joenot443 wrote: post #217
theshrike79 wrote: post #106

Earlier quoted context omitted.

I'm still waiting for the no-code tools that can replace me, the API Glue Coder. They've been promising those since the early 2000's, but I'm still employed. Some stuff has been automated by Zapier, IFTTT and the like, but most of the systems I'm integrating with don't have public APIs.

"I'm still waiting for the automobile that can replace me, the Horse Carriage Driver. They've been promising those since the 1880s, but I've still got passengers in my carriage every day!" Obviously it's a cheeky example, but this would not be the first time in history a previously well-established career was upended in a (relatively) short amount of time. I'm a FAANG dev, I've got skin in the game too and I'm trying…

> All that being said, if you've been working since the 2000s, you've got nearly a 20 year head-start on me, so perhaps it makes sense for me to be a bit more worried.

Yea, that's mostly why I get hired. Experience gives people a certain intuition on what kind of solutions work for which cases.

And when you've been working long enough, you don't (hopefully) feel the need to do cool bleeding edge shit at work, you just want the work code to ... work. You pick the simplest and most boring solution possible so you can clock out at 1600 and not think about services crashing because the newfangled thingamageek coded with the latest language du jour failed because of an edge case nobody has seen before.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#239

Post by SuoDuanDao »

brookst wrote: post #187
dr_dshiv wrote: post #147

Earlier quoted context omitted.

When I say singularity, I mean “changes the nature of human life irreversibly and to a greater degree than any prior technology through a rapid progression of self-reinforcing growth.” That rapid progression can be modeled with an exponential growth curve, like Moore’s law, even though nothing material can have a sustained exponential growth. Regardless of how the curve is modeled, the steepness is such that it serve…

I think you just declared radio, television, the internet, the macaroon, and mobile phones to be singularities. People generally use the term to mean the point at which progress is automatic, unbounded, and instant. E.g. AIs that can produce smarter AIs themselves, in short order.

'step change' sounds less exciting than 'singularity' but in terms of the original definition there's very little distinction.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#240

Post by lionkor »

nwienert wrote: post #8

In before all the comments about how “most code is trivial” or “most programming is stuff that already exists” or “you’re missing the point look how it’s getting better”. I really am in awe of how much work people seem willing to do to justify this as revolutionary and programmers as infantile, and also why they do that. It’s fascinating. Thinking back to my first job out of college as a solid entry level programmer.…

For me, its mostly that I have used GPT-3.5 a little for programming C++, and I wasnt impressed.

For one, it made horrible, glaring mistakes (like defining extern functions which dont exist, using functions which are specific to a platform im not using, etc.), stuff beginners would do.

It also decided to sneak in little issues, such as off-by-one errors (calling write() with a buffer and a size that is off by one in a place where its very hard to tell), missing edge cases (such as writing a C++ concept which worked, but actually did everything in slightly the wrong way to actually ensure the concept was requiring exactly what I asked).

Even when asked to correct these mistakes, it often struggled, made me read paragraph after paragraph of "im sorry, ive been such a bad little machine" garbage, and didnt even correct the issue (or, in some cases, introduced new bugs).

Im utterly unimpressed by this. GPT is great for a lot of things, but not writing code better than I would, in the same time.

The time it took me to massage it to solve a nontrivial problem (write hello world with just syscalls) was way longer than reading the manual and writing it myself (and has less bugs).

Not everyone unfazed by these articles is simply in denial. I feel sorry for people who write copy paste code and find that ChatGPT or Clippy from 2000 can relace them, but not everyone writes trivial code.

Post reply on HN