121–130 of 229 posts
Earlier quoted context omitted.
38%. And 38% is not "a third way to 100%". I'm sure you know what diminishing return is.
my mistake. 40% of the way there, point still stands.
No?
I assume you actually don't know the concept of diminishing return, and it's totally fine. Every other comment here is explaining it to you already. I won't bother to repeat. Please read the sibling comments carefully.
Earlier quoted context omitted.
I've heard this from many people now but I cannot relate at all. Copilot has changed my entire strategy of programming (which I have been doing for 20 years) It writes lots of code before I even think about what I was going to do in that file. Half the time it just works outright, and if not, with minor changes. It is fantastic when I have to build functionality in languages I am not great at. I regularly copy and pa…
It's comments like these that make me wish I could stand behind the chair of the people who get these results because I feel like there's something lost in translation. I don't know if I'm taking your comment too literally but this part confuses me: > It writes lots of code before I even think about what I was going to do in that file. If you haven't even thought of what you want to do in the file, how are you prompt…
Otherwise, even just creating a function, generally they will be filled out automatically.
I suppose I'm quite big on naming (vars and functions)
Maybe if your language or code is more terse, it might struggle more.
I'm sympathetic to the viewpoint that GPT-4 is prone to mistakes when writing code. Unfortunately, the analysis in this paper is pretty bad and doesn't support that conclusion. The authors assume that for any given method under consideration, it must only occur within a particular pattern of other method calls and control flow instructions. But the templates they have chosen are clearly only applicable in certain sit…
The paper actually explains that it targets at the API misuse problem, not the semantic alignment or bugs. Semantic bugs are difficult to detect, which is already a consensus in software engineering field and still many ongoing work on it. And when we say 'semantic' in this special problem it means more than 'semantic' in programs but also how developers express their semantics, which is definitely a problem bigger t…
"Misuse" is not a formally defined thing in Java, and the authors never define what they mean by it.
> If adding the bugs you mentioned, the buggy code generated by LLMs could even exceed the number claimed in the paper.
Or it could be much fewer. Aside from a few scarce examples, the paper gives no evidence that most of the patterns that were selected are actually associated with "misuse". They just declare it to be so. (The 2018 paper they cite for their dataset also provides little such evidence. It just says that the authors reviewed the patterns, reviewed documentation, and decided which ones they considered to be poor code quality.)
It is easy to come up with valid situations where a piece of code violates those patterns, but behaves as intended and is not misusing the API. So why should I assign any meaning to the fact that some percentage of code snippets violate the patterns?
I could make a list of adjectives that frequently appear in comments near buggy code, and then count how many LLM outputs contain those outputs, and then say that means the LLM output is buggy. But I would not be saying anything meaningful about the LLM's quality by doing so.
Earlier quoted context omitted.
> Once it reaches 100% you don't have a job. You realize that right? It's halfway their to taking your job and you're underwhelmed. The bias is on your side. What this study looked at was feeding StackOverflow questions to LLMs and then looking at the quality of the code. If you think a programmer's job is just turning an english-language description of a function into isolated code that never gets modified, I don't…
Not true. They are well past 1%. That's not looking at reality. You can lower it below 38% but 1% is pure fantasy.
To answer your complaint, I am absolutely looking at reality. Please point us to an actual real-world professional programming job, matching the criteria I list above, for which a current LLM can economically replace a human for over a 5-year period.
My believe is that there are exactly zero such jobs. If your claim is that we're at least at 1%, then you're claiming that there are at least 269k fully automatable programmer jobs [1]. It shouldn't be hard for you to find at least one to start.
[1] Wikipedia says there are an estimated 26.9m professional programmers: https://en.wikipedia.org/wiki/Software_engineering_demograph...
Run the code it writes, if it gives an error, paste it into the chat and 90% of the time the LLM can fix the issue. People are missing the point here - it's not about writing code in one-shot. An LLM-enabled loop can generate the code and test and refine it until it works
Someone posted a transcript of having chatgpt write them a bash script with the comment "see how much easier this was than figuring it out" and it took like a dozen tries of pasting error messages back in. I was infuriated just reading it. I cannot imagine trying to develop this way. But if people like it, whatever blows your hair back. Allegedly the benefit is it will do the boring stuff like write the error handlin…
That being said, better workflows do exist. I imagine IDE-integrated LLM tools like Copilot and CodeGPT are much more productivity-enhancing than copy-pasting code between a browser and an editor.
There was a good video posted here yesterday about a (for me) seemingly typical experience writing code with chatGPT https://www.youtube.com/watch?v=U2Q3KAOSAEY The combination of hallucination and just bad advice makes it time consuming and not clearly worth it. I think it's a step in the right direction for conversational interfaces. The fact that you really can have a back and forth to clarify intent, make correct…
Ok but what are we calling API misuses? We collect 1208 coding questions from StackOverflow on 24 representative Java APIs. We summarize thecommon misuse patterns of these APIs and evaluate them oncurrent popular LLMs. The evaluation results show that evenfor GPT-4, 62% of the generated code contains API misuses,which would cause unexpected consequences if the code isintroduced into real-world software. Should have u…
I have been trying to use ChatGPT to make a VueJS website (I’m a backend dev). What I have noticed with VueJS specifically is that it has little to no knowledge of the composition API, and when I ask it to help build a component it is pretty good at the scaffolding. The component will generally work; however, it is pretty useless for CSS styling. I probably should try Copilot as I bet that would work better. Also the…
Ok but what are we calling API misuses? We collect 1208 coding questions from StackOverflow on 24 representative Java APIs. We summarize thecommon misuse patterns of these APIs and evaluate them oncurrent popular LLMs. The evaluation results show that evenfor GPT-4, 62% of the generated code contains API misuses,which would cause unexpected consequences if the code isintroduced into real-world software. Should have u…
Evidently, THEY are calling the following API misuses: "To evaluate the API usage correctness in code, RO- BUSTAPI detects the API misuses against the API usage rules by extracting call consequences and control structures from the code snippet, as shown in Figure 2. The code checker firstly check the code snippets to see whether it is a snippet of a method or a method of a class, so that it can enclose this code snip…
The validity of this entire paper would seem to depend on how robust and agreed upon these "API usage rules" are.
Ok but what are we calling API misuses? We collect 1208 coding questions from StackOverflow on 24 representative Java APIs. We summarize thecommon misuse patterns of these APIs and evaluate them oncurrent popular LLMs. The evaluation results show that evenfor GPT-4, 62% of the generated code contains API misuses,which would cause unexpected consequences if the code isintroduced into real-world software. Should have u…
I just don’t see how it could save time. Programmers nearly universally agree that reading code is harder than writing it. So when you have chat GPT writing your code, you have to read and understand it to ensure it’s actually doing what you need it to without awkward bugs or problems. Given that it’s harder to read than write code, it seems to stand to reason that the “time savings” must be nil.
Have you tried it much?
It's saved me a ton of time over the past 8 months. I don't know what else to say - I'm not the kind of person to deceive myself into thinking something is saving me time when it isn't.
The time saved is mainly in the micro-research you no longer have to do - the bits where you have to go and look up how to write a for loop in Bash, or how to call super() in a Python class, or which exceptions you need to catch for a call to httpx.get().
GPT-4 writes the code right 90% of the time. The 10% of the time it doesn't you catch in the same testing you would have done against code you had written yourself the long way.
Earlier quoted context omitted.
I have been trying to use ChatGPT to make a VueJS website (I’m a backend dev). What I have noticed with VueJS specifically is that it has little to no knowledge of the composition API, and when I ask it to help build a component it is pretty good at the scaffolding. The component will generally work; however, it is pretty useless for CSS styling. I probably should try Copilot as I bet that would work better. Also the…
Can you confirm if you used gpt-4? This was not my experience