101–110 of 229 posts
I'm sympathetic to the viewpoint that GPT-4 is prone to mistakes when writing code. Unfortunately, the analysis in this paper is pretty bad and doesn't support that conclusion. The authors assume that for any given method under consideration, it must only occur within a particular pattern of other method calls and control flow instructions. But the templates they have chosen are clearly only applicable in certain sit…
I feel like AI for programming has been so overhyped. I've attempted to use ChatGPT for programming so many times, and almost every time it's just wasted my time. Giving me outright lies or generating stuff that looks right but doesn't work. And just throwing out it's answer and starting from scratch is faster than fixing its output. I've only found it useful for explaining basic terminology or concepts for a topic I…
The over-hype is causing it to look under-hyped. 48% of the code being freaking correct from 0% about 5 years ago is huge. With everyone and all the news talking about it constantly inevitably people are going to start rolling their eyes. This is bias. 48% is half way to 100%. Once it reaches 100% you don't have a job. You realize that right? It's halfway their to taking your job and you're underwhelmed. The bias is…
Many programmers also do work that doesn’t just involve stitching APIs together; GPT consistently fails badly when forced to reason, making it useless for many tasks. For example, ask it to implement a common algorithm like SHA - it will do a decent job. Then ask it to do it with an arbitrary limitation which no implementation it was trained on would have (e.g. use no integer type wider than 8 bits). In my experience, it cannot achieve this task with a correct result, even with significant hand-holding.
I feel like AI for programming has been so overhyped. I've attempted to use ChatGPT for programming so many times, and almost every time it's just wasted my time. Giving me outright lies or generating stuff that looks right but doesn't work. And just throwing out it's answer and starting from scratch is faster than fixing its output. I've only found it useful for explaining basic terminology or concepts for a topic I…
I've heard this from many people now but I cannot relate at all. Copilot has changed my entire strategy of programming (which I have been doing for 20 years) It writes lots of code before I even think about what I was going to do in that file. Half the time it just works outright, and if not, with minor changes. It is fantastic when I have to build functionality in languages I am not great at. I regularly copy and pa…
I can understand using chatgpt 4 but copilot??? copilot can barely autocomplete... and half the time it does it incorrectly when variables and other stuff are involved
Earlier quoted context omitted.
I've heard this from many people now but I cannot relate at all. Copilot has changed my entire strategy of programming (which I have been doing for 20 years) It writes lots of code before I even think about what I was going to do in that file. Half the time it just works outright, and if not, with minor changes. It is fantastic when I have to build functionality in languages I am not great at. I regularly copy and pa…
copilot? lol I can understand using chatgpt 4 but copilot??? copilot can barely autocomplete... and half the time it does it incorrectly when variables and other stuff are involved
Ok but what are we calling API misuses? We collect 1208 coding questions from StackOverflow on 24 representative Java APIs. We summarize thecommon misuse patterns of these APIs and evaluate them oncurrent popular LLMs. The evaluation results show that evenfor GPT-4, 62% of the generated code contains API misuses,which would cause unexpected consequences if the code isintroduced into real-world software. Should have u…
I just don’t see how it could save time. Programmers nearly universally agree that reading code is harder than writing it. So when you have chat GPT writing your code, you have to read and understand it to ensure it’s actually doing what you need it to without awkward bugs or problems. Given that it’s harder to read than write code, it seems to stand to reason that the “time savings” must be nil.
I feel like AI for programming has been so overhyped. I've attempted to use ChatGPT for programming so many times, and almost every time it's just wasted my time. Giving me outright lies or generating stuff that looks right but doesn't work. And just throwing out it's answer and starting from scratch is faster than fixing its output. I've only found it useful for explaining basic terminology or concepts for a topic I…
I've heard this from many people now but I cannot relate at all. Copilot has changed my entire strategy of programming (which I have been doing for 20 years) It writes lots of code before I even think about what I was going to do in that file. Half the time it just works outright, and if not, with minor changes. It is fantastic when I have to build functionality in languages I am not great at. I regularly copy and pa…
> It writes lots of code before I even think about what I was going to do in that file.
If you haven't even thought of what you want to do in the file, how are you prompting Copilot to write the code you need? Am I understanding correctly that Copilot not only generates the correct code, but it's also able to deduce what code you meant to ask for without any input from you? I don't see how this is possible except in the uncommon case where you are bringing a file to be more in line with other files in your code base that have a similar structure.
I do think Copilot can be useful, but I'm confused at how different my experience is from what others are getting out of the tool.
Earlier quoted context omitted.
The over-hype is causing it to look under-hyped. 48% of the code being freaking correct from 0% about 5 years ago is huge. With everyone and all the news talking about it constantly inevitably people are going to start rolling their eyes. This is bias. 48% is half way to 100%. Once it reaches 100% you don't have a job. You realize that right? It's halfway their to taking your job and you're underwhelmed. The bias is…
> 48% if half way to 100% … It's halfway their to taking your job and your underwhelmed. You’re implying 100% reliability is actually attainable. If that were the case, wouldn’t that mean the halting problem would have been solved by AI? I’m not an expert but I’ve heard that’s like one of those fundamental laws of information theory that really can’t be broken. > It's typical. It's like an indie band is only popular…
Nah even 100% reliability isn't attainable by a human. 100% obviously doesn't imply solving the halting problem.
100% as in 100% as reliable as a human. Even surpassing a human. Sort of in the same way as a computer now beats humans at chess.
>I think applying the sociology of hipster band fans to LLMs is a mistake
Why would it be a mistake? Human psychology is similar across all spectrums. What happens in one area is likely possible in another area. If it happens among hipster bands it can happen among hipster technology fads.
>I think there’s a temptation to apply the analogy of something like the early internet or smartphone to LLMs but those are networked products. LLMs don’t really improve with the various applications built on top of them if the LLM is itself fundamentally broken or faulty to the point that it is unsafe to use in practice.
This is a valid speculation. But it's speculation. Basically your saying that LLMs are fundamentally broken and stuck at 38% forever because of fundamental and permanent flaws. The jury on that one is still out. And your point is highly, highly speculative.
We see quantitative improvements on LLMs constantly AND this is a nascent technology we don't completely understand yet. The most probably and logical conclusion is to follow the technological trendline. That trendline is pointing up.
To speculate on fundamental flaws of the LLM when people don't even fully understand what's going on with the LLM is illogical because you can't derive conclusions from something you don't understand. We can only generalize the trendline and constant improvements we've seen in AI for the past decade. Again that trendline is pointing to further break throughs in the future.
>Obviously, I, like probably everyone else on this website would love LLMs to be reliable to a high degree.
No this is not obvious to me. I disagree. I think some people are like you but other people, for example Geoffrey Hinton are in the apocalyptic camp. Personally I'm in the middle, I think it could go either way. It will definitely harm a segment of our society by taking over work, but whether the benefits of AGI outweighs the harm remains to be seen.
Earlier quoted context omitted.
The over-hype is causing it to look under-hyped. 48% of the code being freaking correct from 0% about 5 years ago is huge. With everyone and all the news talking about it constantly inevitably people are going to start rolling their eyes. This is bias. 48% is half way to 100%. Once it reaches 100% you don't have a job. You realize that right? It's halfway their to taking your job and you're underwhelmed. The bias is…
You assume that getting from “48% as good” as a human programmer (on an undoubtedly biased sample) to “100% as good” is about equally hard as getting from 0% to 48%. That seems a pretty big assumption. Many programmers also do work that doesn’t just involve stitching APIs together; GPT consistently fails badly when forced to reason, making it useless for many tasks. For example, ask it to implement a common algorithm…
I meant 38% aka 40% made a mistake on my math.
It's not a big assumption. It's the most reasonable assumption. When you drive 50% of 10 miles the next 50% takes the same amount of time. It's the default assumption.
I understand where you're coming from though. Many products made by entrepreneurs follow that model where the remaining percentage is always harder than the beginning green fielded tech that was built. But can we honestly say that this is what AGI will be like? The LLM was an unexpected jump forward by an unprecedented amount. It could be we fill the gap to 100% with another such jump.
Earlier quoted context omitted.
Meh. My coworkers do what you’re doing, and then I end up rewriting their stuff. (AI generated stuff generally passes code reviews because it gets the benefit of the doubt.)
Meh. I started coding professionally in the age of AI. My productivity is way higher than it should be. I don't let the AI "write" the code for me. I use it as a companion that I asked question of and suggestions. It suggests solutions to problems, I add the complexity. So far my code has been both clear and performant. This should not be the case at my level.
What does this mean?
Copilot was released in March of this year. ChatGPT 4, which the community considers to be the only version of ChatGPT to be competent enough at coding tasks, was also released in March.
My guess is that you didn't mean that statement to be a fancy way of saying "I've been coding professionally for around 6 months", but I don't know what else you mean by the "age of AI".
Earlier quoted context omitted.
I've heard this from many people now but I cannot relate at all. Copilot has changed my entire strategy of programming (which I have been doing for 20 years) It writes lots of code before I even think about what I was going to do in that file. Half the time it just works outright, and if not, with minor changes. It is fantastic when I have to build functionality in languages I am not great at. I regularly copy and pa…
It's comments like these that make me wish I could stand behind the chair of the people who get these results because I feel like there's something lost in translation. I don't know if I'm taking your comment too literally but this part confuses me: > It writes lots of code before I even think about what I was going to do in that file. If you haven't even thought of what you want to do in the file, how are you prompt…