Live data from Hacker News

Testing GPT 4's code-writing capabilities with some real world problems

tylerglaiel.substack.com

561–570 of 677 posts

Re: Testing GPT 4's code-writing capabilities with some real world problems

#561

Earlier quoted context omitted.

Basically. I'm not sure I'm losing much efficiency, my previous approach was contacting friends who know how to do math in programming, sometimes it could days before I could move past the issue. And the result is the same, ending up with a piece of code I mostly understand, but struggled to write myself so the knowledge is fleeting at best. Just to be clear, the context here is me writing games for fun, while strugg…

Has using it ended up teaching you anything? Like from reading the produced solutions?

Yes, for sure. Some applications of procedural texture primitives (like Perlin Noise) that I wasn't super familiar with and for example "quaternion" which I never heard about before trying to write a game.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#562

Earlier quoted context omitted.

> you should be concerned about the rate of progress Kind of agree? On the one hand we don't even have a roadmap toward reliable AI. On the other, if we ever plug an LLM into something that has memory, acquires experiences, does experiments, observes the outcome and adjusts its worldview in response, consciousness might fall out of that. And writing good code might not even require consciousness.

Epistemologically speaking, I think we can roughly break down the potential nature of consciousness into three categories: - as a function of an independent human soul - as the fundamental substrate on which the rest of the universe is built - as a byproduct/secondary phenomenon of physical processes In the latter two cases I believe that the question of whether GPT is conscious is immaterial. In either case it is fu…

Consciousness in this context is often used as an imprecise but important bundle of very material concepts, including whether something can have wants (and therefore warrants our anticipation of them) and whether it deserves ethical status.

One can debate whether either those is necessarily a consequence of consciousness, but nonetheless those kinds of qualities are what people are aiming at when they wonder about conscious AI.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#563

Earlier quoted context omitted.

> Companies are already bloated, imagine when they realize one overworked highly paid senior can replace 10 juniors. Yep. This is where I'm at in terms of personal armchair predictions of the future. I expect the labor market will be tough for more junior software engineers in the coming years. This might indeed cause backpressure in the supply of new grads/new labor force entrants in this family of fields ("software…

Disagree -- I think chatGPT will make juniors more palatable to hire. ChatGPT will basically give juniors a free pair programmer to baby-sit their work/progress. Why pay extra for senior devs when juniors can become much more efficient thanks to ChatGPT becoming stack-overflow on steroids. I think the wage gap between junior and senior will actually drop massively. I predict teams will keep 1-3 architect level positi…

I really doubt that ChatGPT will be able to give the kind of guidance that turns juniors into seniors.

Getting juniors un-stuck on "simple" problems, maybe. Stack Overflow already does this. Image doesn't build? Search for the error message online. Don't know how to build the old software that only works in Ubuntu 14.10? Sure, you'll find that.

Suggestions on how to refactor, proper design of interfaces, what skill to acquire next? Maybe, but that will be a bigger surprise.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#564
post #352

Earlier quoted context omitted.

I take care not to feed it secrets ; this is just boring ERP stuff without the magic numbers (they are not needed to create or test; we use test data normally as well, as we cannot give that to outsourcing companies either, so there is no difference in work).

And the output? Are you allowed to use their code for your own projects or company?

Yes, we don’t all live in the US.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#565
post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

   If we use GPT-X in our company, it will help us with 1% of our workload
I think there are many such cases. Another one that comes to mind is adding features to a large/legacy code-base. Writing the new code/function is a small part of the work. The main part of the work is first understanding and agreeing on how/where to implement the changes, sometimes across multiple teams, and the implications/knock-on effects to other software components, potential API changes, updating test suites, etc...

Re: Testing GPT 4's code-writing capabilities with some real world problems

#566

Earlier quoted context omitted.

> Companies are already bloated, imagine when they realize one overworked highly paid senior can replace 10 juniors. That is already possible without AI and has been the case for a long time... the issue is nobody will stay to be that highly paid senior running entire projects because at that point you can just run your own shop and pocket the full profits.

It also causes a "single point of failure". If that senior gets hit by a bus then what? Can the company afford to bring in another senior that will take ~6 months to become productive? I'm not disagreeing with you. Im thinking of going solo myself.

The biggest problem going solo is getting the projects/contracts. Now it will become even harder so one has to charge less and less to compete.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#567
post #297

There are plenty of edge cases where it fails. However, the one thing that made me think it actually knows (for certain definitions of the word "knows") what it's doing was asking it to re-write a non-trivial SQL query into equivalent relational algebra. I created a simplified schema from Northwind [0], gave it the CREATE TABLE statements for tables, and then some sort-of-TSV files for the values. It was able to not…

It's not just edge cases where it fails. It fails all the time at all kinds of things. I've been using chatgpt in my work, but I have to essentially know the answer it's going to give me because I have to catch all of its mistakes. It really, really nice for certain kinds of drudge work. Using northwind is probably not a good thing to use to evaluate chatgpt's general capability. It is very commonly used for examples…

> Using northwind is probably not a good thing to use to evaluate chatgpt's general capability.

I did think of that, which is why I modified the schema, and removed any reference to Northwind (normally products are prefixed with "Northwind Traders"). That said, it isn't a particularly complex schema in my example, but it's a reasonable starting point for something a small business might use.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#568
post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

And in infra there's figuring out what new approaches you can use to replace stuff in your infrastructure. Then figuring out the migration costs, evaluating its usability, and dealing with a director that's been sweet-talked by a vendor into using some other solution that sucks. Then deciding whether to just build in house b/c none of the solutions quite work and would require additional stuff to build on top. Then when you finally decide on something the back and forth with the vendor because you need to handle some unique thing they hadn't thought of.

The complexity in software engineering is almost never coding. Coding is easy, almost anyone can do it. Some specialized aspects of coding (ultra low latency realtime work, high performance systems, embedded) require deep expertise but otherwise it's rarely hard. It's dealing in ambiguity that's hard.

The hype around GPT-* for coding generally confirms my suspicions that 70+% of folks in software engineering/development are really "programmers" and 30% are actually "engineers" that have to worry about generating requirements, worrying about long term implications, other constraints, etc.

And every time that comes up those folks in the 70% claim that's just a sign of a poorly managed company. Nope. It's good to have these types of conversations. Not having those conversations is the reason a lot of startups find themselves struggling to stay afloat with a limited workforce when they finally start having lots of customers or high profile ones.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#569

Earlier quoted context omitted.

don't fall into this mental trap. you can get into recursion quite easily here, and figuring out what to prompt can start from simple general questions - and there is no need for a developer at all, aside from the current limitations of copy/paste/run workflow has to be done manually

It's astonishing to see the goalposts move so quickly. The cope of "well, okay, it can do that, but that's not even the hard part!" when just a year ago this entire product was almost unimaginable.

The goalposts haven't really moved though? The reality that a lot of people have thrown "engineer" in their title but just write code given a spec or tightly scoped requirements or design has been true for a while now.

But for quite a few folks, the coding is the easy part. When building larger systems you're not even that often implementing things from scratch. The decision making, dealing with existing tech debt, tooling, etc. is the hard part. Ambiguity is the hard part and it's always been that way.

Don't get me wrong GPT-* is impressive. Heck, I pay for a subscription. But it won't really make a lot of folks at my company notably more productive.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#570
post #50

Earlier quoted context omitted.

I don’t think you’ve tried GPT-4. If you have, I don’t think you are like majority of devs (maybe not on HN, but in real life). You sound lucky to have true, novel problems to solve each day. I’m with many here commenting that this is quite powerful stuff, especially when my day-to-day is writing simple CRUD apps, or transforming data from one format to another within an API, or configuring some new bit of infra or C…

If all you're doing is very simple crud apps and transforming API responses, your coding is already replaceable with "no code" tools.

> If all you're doing is very simple crud apps and transforming API responses,

I think all of us here conflate simple with easy. It's simple in theory yes - you get some JSON from service X, maybe tweak service Y that talks to X and then feed it into some front end. In practice even very experienced engineers can take days writing or changing a simple end point or some front end because unclear requirements/bugs/micro service hell/unclear existing code/etc etc.

If it was that easy the pace and quality would have been much higher than what I'm seeing in tech companies.

Post reply on HN