Live data from Hacker News

Testing GPT 4's code-writing capabilities with some real world problems

tylerglaiel.substack.com

491–500 of 677 posts

Re: Testing GPT 4's code-writing capabilities with some real world problems

#491
I've used chatgpt and while it's mediocre, even bad at writing code, it's very good at reading code and explaining it in a way that's easier to understand. It's also good at giving hints and starting point for when you don't quite familiar with the language feature / library.

From writing code, they're good at bootstrapping unit tests and skeleton code, also useful at transpiling dto / entities between languages.

Overall if you're willing to learn and not just treat a gpt as code monkey, they're very useful.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#492
post #183

I want to see GPT-4 dealing with this situation: - they: we need a new basic POST endpoint - us: cool, what does the api contract look like? URL? Query params? Payload? Response? Status code? - they: Not sure. Third-party company XXQ will let you know the details. They will be the ones calling this new endpoint. But in essence it should be very simple: just grab whatever they pass and save it in our db - us: ok, cool…

My two cents is that the parts of the job that are more like product management will become more dominant but still not exclusive, and the parts that were more like coding will become less dominant but still not vanish. Many of us, as you describe, already do jobs that look a lot like this. But for me, it's not consistently that way; there are periods where I'm almost entirely coding, and periods where I'm almost entirely doing communication. I do expect a shift in this balance over time.

The other thing that I spend a huge amount of my time doing - consistently more than writing code - is debugging. Maybe these models really will get to the point where I can train one on our entire system (in a way that doesn't hand over all our proprietary code to another company...), describe a bug we're seeing, and have it find the culprit with very high precision, but this seems very far from where the current models are. Every time I try to get them to help me debug, it ends in frustration. They can find and fix the kinds of bugs that I don't need help debugging, but not the ones that are hard.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#493

Earlier quoted context omitted.

How much more efficient is programming in Python with modern tools vs assembly? I would bet more than 5x

That's an interesting case and use of the word efficient. Developer time reduced by 5X CPU time increased by 500x When CPU cycles are cheap it's a logical tradeoff but it still grates against my sensibilities.

Depends on the field kinda. For a really extreme case, something like big dense linear algebra is going to just use BLAS calls anyway. For big enough matrices, all of the flops are coming from the library, the cost of calling from Python vs the cost of calling from C or Fortran is amortized anyway, and probably most people won’t beat a tuned BLAS however much developer time they throw at it.

It makes more sense to tune libraries excessively, we could say it is less of a trade off, more of an allocation of the finite low-level tuning developer resources to high-impact libraries.

Anyway it turns out that the only way to get most people to link to a good BLAS is to use Numpy and distribute it through some giant Anaconda Rube Goldberg machine, so, I dunno, people are weird.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#494

Earlier quoted context omitted.

That's an interesting case and use of the word efficient. Developer time reduced by 5X CPU time increased by 500x When CPU cycles are cheap it's a logical tradeoff but it still grates against my sensibilities.

We should stop thinking about the trade-off as between developer time and CPU time. The CPU is an inanimate object which doesn't mind pushing around more electrons. What we're really throwing under the bus when we optimize for developer time is the customer's time . If an inefficiency saves a developer 10 days of work, but results in an operation taking 100 milliseconds longer, and you have 50M customers who do that…

So we just have to figure out if developer-hours or customer-hours are the tighter bottleneck.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#495
I'm not an AI as far as I know, but I would try a classic programming competition technique for this and observe that 6 isn't a very big number.

Step 0: Let's try to find a path without walking through fire. Run Dijkstra's or A* to find the shortest path with no fire, up to distance 6. If it succeeds, that's the answer.

Step 1: Okay, that didn't work. We need to go through at least 1 fire tile. Maybe we can do at most 1. Define distances to be a tuple (fire, cost) where fire is the number of fire tiles used and cost is the cost. Comparison works the obvious way, and Dijkstra's algorithm and A* work fine with distances like this. Look for a solution with cost at most (1, 6). Implemented straightforwardly will likely explore the whole grid (which may be fine), but I'm pretty sure that the search could be pruned when the distance hits values like (0, 7) since any path of cost (0, 7) cannot possibly be a prefix of a (1, c) path for any cStep f: We know we need to go through at least f fire tiles. If f > 6, then just fail -- no path exists. Otherwise solve it like step 1 but for costs up to (f, 6). Prune paths with cost (f', c') with c' > 6.

This will have complexity 6D where D is the cost of Dijkstra's or A or whatever the underlying search is. Without pruning, D will be the cost of search with no length limit but, with pruning, D is nicely bounded (by the number of tiles with Manhattan distance 6 from the origin times a small constant).

For a large level and much larger values of 6, this could be nasty and might get as large as t^2 * polylog(t) where t is the number of tiles. Fortunately, is upper-bounded by 6 and doesn't actually get that large.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#496
post #229

Earlier quoted context omitted.

I think the fear should be less about AI taking 100% of jobs but it should be AI making a single programmer do the job of 5, which would wipe a majority of the market out and make it a non-viable career option for most. Companies are already bloated, imagine when they realize one overworked highly paid senior can replace 10 juniors.

> Companies are already bloated, imagine when they realize one overworked highly paid senior can replace 10 juniors. Yep. This is where I'm at in terms of personal armchair predictions of the future. I expect the labor market will be tough for more junior software engineers in the coming years. This might indeed cause backpressure in the supply of new grads/new labor force entrants in this family of fields ("software…

I see it the other way around the senior engineer is expensive and costs a lot 2 juniors are cheap. Who cares if their code is crap they produce it really fast so they find a bug, just fix it quickly. If anything the time a Sr. spends thinking about things to do things "right" is seen as a waste of time. Whereas the jr. will produce an enormous amount of buggy code but they can fix the bugs quickly by just throwing another prompt to ChatGPT to solve.

Now some might say that the code will be terrible quality and buggy and full of holes and the users will hate it, it is never economically viable to build enormous systems on a house of cards like that. To which I respond, you just described every piece of enterprise software I've ever used ever.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#497

Earlier quoted context omitted.

That's irrelevant though, because actual programming is very much about computers, and about writing things the majority of which have already been formalized by computer scientists, and gluing code and programs inti pipelines...

I just adore the hubris on full display here to say what one of the world's best computer science educators has to say is irrelevant.

"Science advances one funeral at a time"

- Max Planck

Are you saying Max Planck is wrong here? There is plenty of evidence that he was right, that statements made by famous scientists holds fields back because they were reductive or hurting different perspectives. Putting their words at an altar is holding us back.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#498

Earlier quoted context omitted.

> Those parts would still be handled with GPT-X talking to another GPT-X in the other company. Even the requirements might be given by a GPT-X. What happens if one (or more) of the GPT-Xs starts having hallucinations while they're busy working on this project? > Second, even if that's not the case, the part of doing the communication can be handled by non-programmers. I was in a meeting with a sales manager and tech…

Are hallucination is a systemic for-ever problem that will not be solved, mitigated or in akne other way rendered inconsequential? Also, having conversations about things you don't understand with a machine, where you don't have to keep up social decorum and can ask the dumbest questions should help a lot with improving the decision making of non-technical personel

> Are hallucination is a systemic for-ever problem that will not be solved, mitigated or in akne other way rendered inconsequential?

In the same way that wrong results in a search engine will not be solved or mitigated in its entirety, yes. These new generative AIs are big search engines that bring results from several content points and combine them into a single, plausible stream of words.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#499
post #253

Earlier quoted context omitted.

If increased productivity equaled job loss, there would be two programmers alive today, doing the same job as the fewer than 10000 programmers using punch cards as we entered the year 1950. A lot of projects today are not even greenlit because they would be too expensive to make. For instance, there are a lot processes in almost every country that require you to file paper forms, even though we have had web forms and…

> If increased productivity equaled job loss there would be two programmers alive today, doing the same job as the fewer than 10000 programmers using punch cards as we entered the year 1950. The only reason it's not the case in this example is because computers at the time were a tiny early adopter niche, which massively multiplied and expanded to other areas. Like, only 1 in 10,000 businesses would have one in 1950,…

[deleted]

Re: Testing GPT 4's code-writing capabilities with some real world problems

#500
post #253
post #229

Earlier quoted context omitted.

I think the fear should be less about AI taking 100% of jobs but it should be AI making a single programmer do the job of 5, which would wipe a majority of the market out and make it a non-viable career option for most. Companies are already bloated, imagine when they realize one overworked highly paid senior can replace 10 juniors.

If increased productivity equaled job loss, there would be two programmers alive today, doing the same job as the fewer than 10000 programmers using punch cards as we entered the year 1950. A lot of projects today are not even greenlit because they would be too expensive to make. For instance, there are a lot processes in almost every country that require you to file paper forms, even though we have had web forms and…

If increased productivity equaled job loss, there would be two programmers alive today

Increased productivity doesn't necessarily lead to overall job loss, but it will eventually in the area where the productivity is realized. Agricultural employment is a very clear example.

https://cdn2.vox-cdn.com/assets/4565243/Ag_workforce.png

Post reply on HN