Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Did GitHub Copilot increase my productivity?

trace.yshui.dev

Post reply on HN
Print view View original

251–260 of 326 posts

Re: Did GitHub Copilot increase my productivity?

#251

Post by BlueTemplar »

selcuka wrote: post #59
simonw wrote: post #8

Earlier quoted context omitted.

I've been finding this stuff genuinely useful for two years now, across Copilot and ChatGPT and Claude 3 Opus and similar tools. Either I'm a dimwit, easily conned by hype and shiny tools to the point that I can imagine benefits for two years that simply aren't there... or there's something to them.

I basically say the same when people ask me if the AI bubble is going to burst. While I agree that the current push for LLMs should be taken with a grain of salt, I don't think it's a bubble either. I have been finding it useful too, and I don't think I will stop using it because it will look less shiny in the future.

The dot com was a bubble that burst spectacularly enough, yet we still use the World Wide Web, and even some of those websites.

Re: Did GitHub Copilot increase my productivity?

#252

Post by DanielHB »

marcus_holmes wrote: post #87

Years ago, over a decade ago now, I was a .Net developer. Microsoft introduced Entity Framework, their new way of handling data in .Net applications. Promises made, promises believed, we all used it. I was especially glad of Lazy Loading, where I didn't have to load data from the database into my memory structures; the system would do that automatically. I could write my code as if all my memory structures were popul…

> I was especially glad of Lazy Loading, where I didn't have to load data from the database into my memory structures; the system would do that automatically.

oh god, I have used Java with Hibernate a lot and once I read "Lazy Loading" I didn't even need to finish reading the post.

Re: Did GitHub Copilot increase my productivity?

#253

Post by causal »

couchand wrote: post #207
klauserc wrote: post #148

I find GitHub Copilot close to useless for production code. The worst, most obscure bugs I've had to debug in the last year were all in Copilot-written code. It _looks_ plausible, but it makes extremely subtle mistakes. Occasionally, you have repetitive sections of code where it can copy&adapt lines from the context, but that's about it. It's a different story for test code. Test code is often formulaic and "standard…

Test code is code. It's as much of a burden as every other piece of code you are troubled with, so you must make it count. If you're finding it repetitive and formulaic, take that opportunity to identify the next refactoring. Just churning out more near copies is not a good answer.

LLMs are pretty good at anything that follows a pattern, even a really complex pattern. So unit tests often take a form similar to the n-shot testing we do with LLMs, a series of statements and their answers (or in the case of unit tests, a series of test names and their tests). It makes sense to me that LLMs would excel here and my own experience is that they are great at taking care of the low-hanging fruit when it comes to testing.

Re: Did GitHub Copilot increase my productivity?

#254

Post by zarathustreal »

klauserc wrote: post #148

I find GitHub Copilot close to useless for production code. The worst, most obscure bugs I've had to debug in the last year were all in Copilot-written code. It _looks_ plausible, but it makes extremely subtle mistakes. Occasionally, you have repetitive sections of code where it can copy&adapt lines from the context, but that's about it. It's a different story for test code. Test code is often formulaic and "standard…

If you’re not using a language that can properly support algebraic structures and randomized property-based testing you’re essentially getting no guarantees about your code from tests. You wrote the code, you wrote the tests, they’re equally likely to be incorrect.

Re: Did GitHub Copilot increase my productivity?

#255

Post by causal »

mring33621 wrote: post #248

I am a 20+ year polyglot developer. I just cancelled my Github Copilot sub. Sometimes, it was magical, but more often it was mediocre and just got in the way. I generally have better luck with a back-and-forth chat with an LLM on the web and then cutting/pasting the snippets into my IDE. I can do those for free. Will try again in a year, probably.

+1 on being more productive with chat than Copilot.

I find copilot mostly to be a useful Google replacement, e.g. if I forget the syntax for a for loop in JavaScript I can just write a comment like `// For loop over X` and it will spit out the boilerplate I need without breaking my flow.

In either case it's really important not to get stuck "trying to make it do what I want". If an LLM can't solve your problem in 3 tries, you're probably asking too much of it.

Re: Did GitHub Copilot increase my productivity?

#256

Post by claytongulick »

skywhopper wrote: post #165
muglug wrote: post #29

GitHub touts a 55% improvement in coding speed based on a study conducted in-house that tested the participants’ ability to paste a pre-written prompt and then check the output: https://github.blog/2022-09-07-research-quantifying-github-c... The effectiveness of a Copilot-like tool trialed at FB showed that 8% of code contributed by participants was sourced from suggestions, but the latter study made no promise about…

The last entity I would trust for trustworthy data on how much Copilot helps programmers is GitHub. They obviously skew the scenarios and metrics to find the best possible number to report. I think the truth as revealed in your comment and many others is that it all depends on the context. For repetitive code or boilerplate, or starting new projects in well-known frameworks and languages, it probably does increase ve…

It also depends on what tools a developer is already using as a productivity booster.

For example, I use vanilla web components which have some boilerplate for new components.

I already have a simple vscode snippet that does the job well, with no hallucinations [1]. I've experimented with llms doing the same thing with not great results.

It took me longer to explain what I wanted than it did for me to just write that snippet. Doing it repeatedly and waiting for the results definitely didn't increase my speed, though I was impressed that it was eventually able to figure it out (vanilla web components with lit-html renderer isn't a super common technique). Also, I prefer the pythonic approach of snake-case local variable names. Getting the llm to do that in a js project where it's not super common was another whole iteration.

We've had code generation tools around for decades to deal with repetitive boilerplate tasks. Maybe they aren't quite as capable as when the llm "gets it right" - but I wonder with these productivity claims how they are measured.

Are they starting from zero and a new developer? Or comparing against an experienced developer with proficiency with lots of workflow enhancers like templates, snippets, vim macros, etc...

[1] https://gist.github.com/claytongulick/e4251c1b27b22c25fb68f3...

Re: Did GitHub Copilot increase my productivity?

#257

Post by nasmorn »

couchand wrote: post #207
klauserc wrote: post #148

I find GitHub Copilot close to useless for production code. The worst, most obscure bugs I've had to debug in the last year were all in Copilot-written code. It _looks_ plausible, but it makes extremely subtle mistakes. Occasionally, you have repetitive sections of code where it can copy&adapt lines from the context, but that's about it. It's a different story for test code. Test code is often formulaic and "standard…

Test code is code. It's as much of a burden as every other piece of code you are troubled with, so you must make it count. If you're finding it repetitive and formulaic, take that opportunity to identify the next refactoring. Just churning out more near copies is not a good answer.

I agree. A very high impact change I made for an application my team is working on was allowing easy creation of test cases from production data. We deal with almost unknowable upstream data and cheaply testing something that was not working out has reduced the time to find bugs tremendously

Re: Did GitHub Copilot increase my productivity?

#258

Post by DangitBobby »

couchand wrote: post #204
lozenge wrote: post #137

Earlier quoted context omitted.

ORM is not for writing analytics queries. It's for your CRUD operations. Something like Django Admin would be impossible without an ORM. You create tables for your business logic and customer support or whoever can just browse and populate them.

Wouldn't standard ANSI SQL's information_schema be sufficient to build such an interface? I'm struggling to see how an ORM is necessary.

I consider an ORM to be any SQL generating API, without which it would indeed be impossible to have a generic Admin class to make Admin views in Django.

Re: Did GitHub Copilot increase my productivity?

#259

Post by DangitBobby »

blksv wrote: post #147
kaba0 wrote: post #124

Earlier quoted context omitted.

How do you map rows to objects? How do you insert into/update rows in your databases? These are the basic problems ORMs solve splendidly. They are for OLTP workloads, and have deliberate escape hatches to SQL (or some abstraction over it, like JPQL in java-land). I just fail to see what else would you do, besides implementing a bug-ridden, half-ORM yourself.

Rows are tuples, not objects, and treated as such throughout the code. Only the needed data is selected in the form most appropriate to the task at hand, constructed in a hand-written sql query, maybe even taylored to the DB/task specifics. Inserts/updates are also specific to the task, appropriately grouped, and also performed using plain sql. Data pipelines are directly visible in the code, all DB accesses are expl…

Maybe we need to use a different acronym than ORM, because to me the thing we can all agree we need is code that emits SQL. If you can't agree that projects need generated SQL because SQL is dog water for composition, then we can't really agree on anything.

Re: Did GitHub Copilot increase my productivity?

#260

Post by DangitBobby »

sgarland wrote: post #197
kaba0 wrote: post #124

Earlier quoted context omitted.

How do you map rows to objects? How do you insert into/update rows in your databases? These are the basic problems ORMs solve splendidly. They are for OLTP workloads, and have deliberate escape hatches to SQL (or some abstraction over it, like JPQL in java-land). I just fail to see what else would you do, besides implementing a bug-ridden, half-ORM yourself.

Every RDBMS has multiple connector libraries that solve this for you, without requiring the overhead of a full ORM.

If the connector library solves this problem then the connector library is an ORM.
Post reply on HN