GitHub Copilot Workspace: Technical Preview • Hacker News Board

Re: GitHub Copilot Workspace: Technical Preview

#261

Post by H12 » Tue, Apr 30, 2024, 12:57 AM UTC

candiddevmike wrote: post #150

singluere wrote: post #112

While I've not used this product, I've created somewhat similar setup using open source LLMs that runs locally. After having used it for about three months, I can say that debugging LLM prompts was far more annoying than debugging code. Ultimately, I ended up abandoning my setup and going in favor of writing code the good old fashioned way. YMMV

I had ChatGPT output an algorithm implementation in Go (Shamir Secret Sharing) that I didn't want to figure out. It kinda worked, but everytime I pointed out a problem with the code it seemed more bugs were added (and I ended up hating the "Good catch!" text responses...) Eventually, figuring out why it didn't work made me have to read the algorithm spec and basically write the code from scratch, throwing away all of…

The only time I've had success with using AI to drive development work is for "writers block" situations where I'm staring at an empty file or using a language/tool with which I'm out of practice or simply don't have enough experience.

In these situations, giving me something that doesn't work (even if I wind up being forced to rewrite it) is actually kinda helpful. The faster I get my hands dirty and start actually trying to build the thing, the faster I usually get it done.

The alternative is historically trying to read the docs or man pages and getting overwhelmed and discouraged if they wind up being hard to grok.

Re: GitHub Copilot Workspace: Technical Preview

#262

Post by coffeebeqn » Tue, Apr 30, 2024, 12:59 AM UTC

incorrecthorse wrote: post #21

studenthrow3831 wrote: post #3

Student here: I legitimately cannot understand how senior developers can dismiss these LLM tools when they've gone from barely stringing together a TODO app to structuring and executing large-scale changes in entire repositories in 3 years. I'm not a singulatarian, but this seems like a brutal S-curve we're heading into. I also have a hard time believing that there is enough software need to make such an extreme prod…

> they've gone from barely stringing together a TODO app to structuring and executing large-scale changes in entire repositories in 3 years. No they didn't. They're still at the step of barely stringing together a TODO app, and mostly because it's as simple as copying the gazillionth TODO app from GitHub.

I’ve used copilot recently in my work codebase and it absolutely has no idea what’s going on in the codebase. At best it’ll look at the currently open file. Half the time it can’t seem to comprehend even the current file fully. I’d be happy if it was better but it’s simply not.

I do use chatgpt most recently today to build me a GitHub actions yaml file based on my spec and it saved me days of work. Not perfect but close enough that I can fill in some details and be done. So sometimes it’s a good tool. It’s also an excellent rubber duck- often better than most of my coworkers. I don’t really know how to extrapolate what it’ll be in the future. I would guess we hit some kind of a limit that will be tricky to get past because nothing scales forever

Re: GitHub Copilot Workspace: Technical Preview

#263

Post by hanniabu » Tue, Apr 30, 2024, 12:59 AM UTC

lispisok wrote: post #194

imiric wrote: post #170

Earlier quoted context omitted.

> And I suspect that kind of story will continue quite a bit as the tech matures. Don't you think that this tech can only get better? And that there will come a time in the very near future when the programming capabilities of AI improve substantially over what they are now? After all, AI writing 300 line programs was unheard of a mere 2 years ago. This is what I think GP is ignoring. Spreadsheets couldn't to do ever…

You are extrapolating from when we saw huge improvements 1-2 years ago. Performance improvements have flatlined. Current AI predictions reminds me of self-driving car hype from the mid 2010s

We've entered the acceleration age

Re: GitHub Copilot Workspace: Technical Preview

#264

Post by hanniabu » Tue, Apr 30, 2024, 1:04 AM UTC

lispisok wrote: post #147

msoad wrote: post #78

This should've been done in VSCode. Most developers don't want to write code in a browser. Once VSCode extension is built they could generalize it to other editors. I'm sure they are thinking of VSCode integration and I am hopping for that to be available soon

I wonder if AI tools is going to kill editor diversity. Every one of these tools VS Code is the first class citizen and other editors are an afterthought. Sure people can write their own Emacs package but that's only if the tool developers enable it and the experience is usually not as good as the official VS Code version. I can also see the future where not using VS Code is a signal you are less reliant on AI tools…

I'm still chugging along on sublime text 3

Re: GitHub Copilot Workspace: Technical Preview

#265

Post by no_carrier » Tue, Apr 30, 2024, 1:14 AM UTC

goriv wrote: post #251

mvdtnz wrote: post #246

Earlier quoted context omitted.

How do you not let it read the buggy code but also take it into a new session?

I assume just copy and paste it

I'm just as confused... If you're copy/pasting the code into a new session, isn't that reading the code?

Re: GitHub Copilot Workspace: Technical Preview

#266

Post by wrl » Tue, Apr 30, 2024, 1:17 AM UTC

studenthrow3831 wrote: post #3

Student here: I legitimately cannot understand how senior developers can dismiss these LLM tools when they've gone from barely stringing together a TODO app to structuring and executing large-scale changes in entire repositories in 3 years. I'm not a singulatarian, but this seems like a brutal S-curve we're heading into. I also have a hard time believing that there is enough software need to make such an extreme prod…

after the first two or three times i got asked to code-review something that another developer "didn't know how to write so just asked copilot/chatgpt/etc and it produced this, could you tell me if it's right?" i got pretty tired of it. obviously it's useless to ask questions about how the code was written because the person asking for the code review didn't actually write it and they don't have any answers about why it was written how it was.

especially on the back of the xz supply chain attack and, y'know, literally any security vulnerability that slipped through code review, i refuse to have unaccountable, unreviewed code in projects i work on.

somewhat recently, there was the case with air canada's LLM-based support bot making a false statement and then a judge forcing air canada to honour it. i think we're setting the stage for something like that happening with LLM-written code – it's going to be great for a while, everyone's going to be more productive, and then we'll all collectively find out that copilot spat out a heartbleed-level flaw in some common piece of software.

Re: GitHub Copilot Workspace: Technical Preview

#267

Post by thomasfromcdnjs » Tue, Apr 30, 2024, 1:19 AM UTC

idan wrote: post #259

steve_adams_86 wrote: post #244

Watching the examples and other people's demos, I get the sense that this product completely ignores what makes good software. It has no "big picture" plan or contextual awareness required to make those nuanced yet critical decisions in the far ends of applications. Those cases where you badly need domain expertise, understanding of how the user interfaces with the product, awareness of existing technical debt, and s…

That hasn't been our experience using it in-house! It's not perfect, but not every software engineering task is some galaxy-brain architectural shit. Sometimes, you have to lay down some bricks. And sometimes, to lay down bricks, you need to touch more than one tiny patch of code. Having something round up the likely areas of the codebase that needs touching feels magical. It doesn't always succeed! But it feels pret…

A part of building great products is allowing your engineers to focus on things that really matter. But sometimes you need to change copy, add small features, from my experience with some of these tools, it is incredible. A PM can just create a repo issue, and the bots will go away with high accuracy and quality (if your codebase is setup well), and submit a PR.

This level of velocity for teams cannot be understated.

Re: GitHub Copilot Workspace: Technical Preview

#268

Post by SushiHippie » Tue, Apr 30, 2024, 1:24 AM UTC

no_carrier wrote: post #265

goriv wrote: post #251

Earlier quoted context omitted.

I assume just copy and paste it

I'm just as confused... If you're copy/pasting the code into a new session, isn't that reading the code?

The way I understand it:

First Variant:

1. User: Asks coding question

2. Ai: Outputs half functioning code

3. User: Asks to fix specific things

4. Ai: Creates buggy code

5. User: asks again to fix things

6. Ai: writes even more buggy code

Proposed second variant with copying code:

Until step 4 everything stays the same, but instead of asking it to fix the code again you copy it into another session, this way, you'll repeat step 3 again, without the LLM "seeing" the code it previously generated for step 4.

Re: GitHub Copilot Workspace: Technical Preview

#269

Post by esperent » Tue, Apr 30, 2024, 1:27 AM UTC

candiddevmike wrote: post #150

singluere wrote: post #112

While I've not used this product, I've created somewhat similar setup using open source LLMs that runs locally. After having used it for about three months, I can say that debugging LLM prompts was far more annoying than debugging code. Ultimately, I ended up abandoning my setup and going in favor of writing code the good old fashioned way. YMMV

I had ChatGPT output an algorithm implementation in Go (Shamir Secret Sharing) that I didn't want to figure out. It kinda worked, but everytime I pointed out a problem with the code it seemed more bugs were added (and I ended up hating the "Good catch!" text responses...) Eventually, figuring out why it didn't work made me have to read the algorithm spec and basically write the code from scratch, throwing away all of…

The bits GPT4 always gets wrong - and as you say, more and more wrong the further I try to work with it to fix the mistakes - are exactly the bits I want it to do for me. Tedious nested loops that I need to calculate on paper in particular.

What it's good for is high level overview and structuring of simple apps, which saves me a lot of googling, reviewing prior work, and some initial typing.

After my last attempts to work with it, I've decided that until there's another large improvement in the models (GPT5 or similar), I won't try to use it beyond this initial structure creation phase.

The issue is that for complex apps that already have a structure in place - especially if it's not a great structure and I don't have the rights or time to do a refactoring - the AI can't really do anything to help. So in this case, for new, simple, or test projects it'll seem like an amazing tool and then in the real world it's pretty much useless or even just wastes time, except for brainstorming entirely new features that can be reasoned about in isolation, in which case it's useful again.

A counterpoint is that code should always be written in a modular way so that each piece can be reasoned about in isolation. Which doesn't often happen in large apps that I've worked on, unfortunately. Unless I'm the one who writes them from scratch.

Re: GitHub Copilot Workspace: Technical Preview

#270

Post by halfmatthalfcat » Tue, Apr 30, 2024, 1:33 AM UTC

ajbt200128 wrote: post #220

candiddevmike wrote: post #150

Earlier quoted context omitted.

I had ChatGPT output an algorithm implementation in Go (Shamir Secret Sharing) that I didn't want to figure out. It kinda worked, but everytime I pointed out a problem with the code it seemed more bugs were added (and I ended up hating the "Good catch!" text responses...) Eventually, figuring out why it didn't work made me have to read the algorithm spec and basically write the code from scratch, throwing away all of…

> Shamir Secret Sharing > ChatGPT please don't roll your own crypto, and PLEASE don't roll your own crypto from a LLM. They're useful for other kinds of programs, but crypto libraries need to be to spec, and heavily used and reviewed to not be actively harmful. Not sure ChatGPT can write constant time code :)

People always say this but how else are you going to learn? I doubt many of us who are "rolling our own crypto" are actually deploying it into mission critical contexts anyway.