Live data from Hacker News

Testing GPT 4's code-writing capabilities with some real world problems

tylerglaiel.substack.com

651–660 of 677 posts

Re: Testing GPT 4's code-writing capabilities with some real world problems

#651
post #72

Earlier quoted context omitted.

the big thing to me is programming isn't hard I've worked on adtech, crypto, fintech, gamedev, startup founder, BigCo. Not once was programming something that was a time sink. Makes me feel like GPT is marketing to the incompetent or something.

Exactly, we know how much code gets written per programmer hour, its in the single digits on average. Speeding that up wont make a big difference.

Single digits what? What are you measuring here? Files, lines, KLOCs?

Re: Testing GPT 4's code-writing capabilities with some real world problems

#652

Earlier quoted context omitted.

Epistemologically speaking, I think we can roughly break down the potential nature of consciousness into three categories: - as a function of an independent human soul - as the fundamental substrate on which the rest of the universe is built - as a byproduct/secondary phenomenon of physical processes In the latter two cases I believe that the question of whether GPT is conscious is immaterial. In either case it is fu…

Consciousness in this context is often used as an imprecise but important bundle of very material concepts, including whether something can have wants (and therefore warrants our anticipation of them) and whether it deserves ethical status. One can debate whether either those is necessarily a consequence of consciousness, but nonetheless those kinds of qualities are what people are aiming at when they wonder about co…

[dead]

Re: Testing GPT 4's code-writing capabilities with some real world problems

#653

Earlier quoted context omitted.

This is perhaps one of the most impressive responses I've read. It truly seems like there is some reasoning happening. I don't understand how this can be the output of a generative LLM.

> It truly seems like there is some reasoning happening. I don't understand how this can be the output of a generative LLM Right - this seeming "cognition" is exactly what's so spooky about the whole thing. Here's what spooked me out from yesterday: https://news.ycombinator.com/item?id=35167685 - specifically how it determines the divide-by-zero error in this code: https://whatdoesthiscodedo.com/g/6a8f359 ...which de…

Whoa, that explanation is really cool.

> capable of at-least C++ "constexpr"-style compile-time computation, which shouldn't even be possible if one presumes GPT is "just" a giant database storing only multidimensional word similarity scores and sequence distribution from text inference

I don't see how being a giant word-DB necessarily disqualifies compile-time computation. You can view computation as applying a series of term rewrite rules to an input until some sort of termination condition (or indefinite loop). In the case of these AI, the input is the prompt and predicting the next token is a limited form of term rewriting (where the rules are probabilistic and based off the network), and because code and explanations were probably included in the training data, it seems reasonable to me that the "rewrite rules" of Python bled a little bit into the AI.

It makes me insanely curious about the internal structures though. I gave that site 2 similar examples: one produces a correct explanation while another produces an incorrect explanation. The difference: a deleted line of insignificant whitespace

* https://whatdoesthiscodedo.com/g/dd2af89

* https://whatdoesthiscodedo.com/g/45ea060

From those 2 examples, I think its pretty clear that the AI's "rewrite rules" don't always coincide with Python's, but I would expect this to be mitigated by targeted training (like Copilot).

Re: Testing GPT 4's code-writing capabilities with some real world problems

#654
post #639

Earlier quoted context omitted.

> You meant "expand massively" i think. Nope, I meant what I said. > There wont be any unprecedented mass layoffs Baghdad Bob strikes again.

Then youre wrong. There is no market incentive for layoffs due to GPT-like technology, as I have demonstrated above. Similar breakthroughs "replacing jobs" have happened before in the field of software engineering, this is nothing new or unprecedented. Its merely another tool that will become in widespread use to increase production. What i think will lead to mass layoffs is the current recession rather.

  > this is nothing new or unprecedented
This is both new and unprecedented.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#655

Earlier quoted context omitted.

For me, its mostly that I have used GPT-3.5 a little for programming C++, and I wasnt impressed. For one, it made horrible, glaring mistakes (like defining extern functions which dont exist, using functions which are specific to a platform im not using, etc.), stuff beginners would do. It also decided to sneak in little issues, such as off-by-one errors (calling write() with a buffer and a size that is off by one in…

Two points: GPT4 is significantly better in this regard, and you should be concerned about the rate of progress more than it’s actual capabilities today.

[dead]

Re: Testing GPT 4's code-writing capabilities with some real world problems

#656
post #592

Earlier quoted context omitted.

I'm impressed. But I want it to be better, so I don't have to spend so much time coding and can trust it to give me good code so I can offload a lot of my efforts on to it. Right now I have to check and verify or go back and forth with prompts so much I almost would have been better off writing the damn code in the first place. Meanwhile I've got a backlog of about 100 personal projects I've never gotten around to ma…

Do you find AI to be good with art generation? I can't use any of the art in the projects that I do without extensive inefficient editing since the stuff it spits out isn't how a human would draw. It's good for concept browsing, but not much more for me at the moment.

I haven't messed with it much since I find the process for the 'good one' (Midjourney) annoying (go onto a Discord channel, type your request in public for a bunch of people, wait a while, hunt for it amongs the long channel of requests, etc).

I'm assuming the process has gotten better since, but I don't know. I'm mostly just using free vector art and simple colors/shapes or whatever simple things I can make in Blender for my art still, in part because there's such a backlash against using any A.I. art right now.

Most of it is judging by what people have been saying in groups online. Some people have found it very useful and use it extensively, like for their board game art.

It doesn't even have to get 100% of the way there (for coding games based on rulesets). Even 75% would probably save me a lot of time and allow me to find the energy to close the gap as opposed to not even starting since I have so many other projects going on.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#657

Earlier quoted context omitted.

The most challenging part of software development (and the reason we have well-paying jobs) is not understanding syntax, it's analysing and abstracting a problem domain into a set of cleanly separated modules that interact to solve those problems. That being the case then, actually none of us is getting replaced by GPT-n any time soon - 'prompt engineering' will just become the new Javascript, only more abstract; jus…

Correct. But once everyone has that tool in their toolbox everyone will become more productive, meaning skills scarcity will be greatly reduced. In turn that will lead to massive wage depression.

You mean such as when high level languages became mainstream and we no longer needed to code in assembly language? Or when IDEs became widely available? The underlying design skills are still difficult to acquire and not displaced by new tools - that is at least until SoftwareArchitectGPT comes along...

Re: Testing GPT 4's code-writing capabilities with some real world problems

#658
post #606

Earlier quoted context omitted.

> Why should it happen now? What are S-curves.

Perhaps the quantity of labour utilizing each new technology through time is a (n-shaped) parabola that intersects with the technology it replaced. The fear that technological advances will cause mass unemployment and destroy labour markets has been common throughout history. Yet here we are at full employment. Maybe this time is different?

It is different this time. In the past automation effected some domains more and some domains not at all. People moved to those other domains. AI can run all the domains humans do and more that they can't.

https://www.youtube.com/watch?v=7Pq-S557XQU

Re: Testing GPT 4's code-writing capabilities with some real world problems

#659

Earlier quoted context omitted.

> Especially when a higher-ranking person comes back from vacation and says "no, everything you nailed down in the last two weeks is completely wrong". Yes, and at some point this high-ranking person is fed up with this now-inefficient use of time and money enough that they will just sort this out using an LLM tuned to handle this situation better if not today then tomorrow. Maybe they will pay someone to coach them…

Imagine how angry this hypothetical person will be when they get the same problem from the LLM, after all those extra steps.

Imagine an LLM tuned to eliminate misunderstanding and ask why at least 5 levels deep… Without fearing to irritate the boss or to create an impression of being not smart, both possibly harmful for human career but irrelevant to unthinking software tool.

Re: Testing GPT 4's code-writing capabilities with some real world problems

#660

Earlier quoted context omitted.

Imagine how angry this hypothetical person will be when they get the same problem from the LLM, after all those extra steps.

Imagine an LLM tuned to eliminate misunderstanding and ask why at least 5 levels deep… Without fearing to irritate the boss or to create an impression of being not smart, both possibly harmful for human career but irrelevant to unthinking software tool.

I too like science fiction. People keep acting like it will be easy to bolt on things like eliminate misunderstandings onto LLMs and quite frankly I would be incredibly surprised if that happens any time soon.
Post reply on HN