Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Watching AI drive Microsoft employees insane

old.reddit.com

Post reply on HN
Print view View original

161–170 of 580 posts

Re: Watching AI drive Microsoft employees insane

#161

Post by bonoboTP »

Fixing existing bugs left in the codebase by humans will necessarily be harder than writing new code for new features. A bug can be really hairy to untangle, given that even the human engineer got it wrong. So it's not surprising that this proves to be tough for AI.

For refactoring and extending good, working code, AI is much more useful.

We are at a stage where AI should only be used for giving suggestions to a human in the driver's seat with a UI/UX that allows ergonomically guiding the AI, picking from offered alternatives, giving directions on a fairly micro level that is still above editing the code character by character.

They are indeed overpromising and pushing AI beyond its current limits for hype reasons, but this doesn't mean this won't be possible in the future. The progress is real, and I wouldn't bet on it taking a sharp turn and flattening.

Re: Watching AI drive Microsoft employees insane

#162

Post by Quarrelsome »

jeswin wrote: post #85

I find it amusing that people (even here on HN) are expecting a brand new tool (among the most complex ever) to perform adequetely right off the bat. It will require a period of refinement, just as any other tool or process.

its more that the AI-first approach can be frustrating for senior devs to have to deal with. This post is an example of that. We're empathising with the code reviewers.

Re: Watching AI drive Microsoft employees insane

#163

Post by cyanydeez »

balazstorok wrote: post #38

At least opening PRs is a safe option, you can just dump the whole thing if it doesn't turn out to be useful. Also, trying something new out will most likely have hiccups. Ultimately it may fail. But that doesn't mean it's not worth the effort. The thing may rapidly evolve if it's being hard-tested on actual code and actual issues. For example it will be probably changed so that it will iterate until tests are actual…

Unfortunately,if you believe LLMs really can learn to code with bugs, then the nezt step would be to curate a sufficiently bug free data set. Theres no evidence this has occured, rather, they just scraped whayecer

Re: Watching AI drive Microsoft employees insane

#164

Post by RajT88 »

cebert wrote: post #2

Do we know for a fact there are Microsoft employees who were told they have to use CoPilot and review its change suggestions on projects? We have the option to use GitHub CoPilot on code reviews and it’s comically bad and unhelpful. There isn’t a single member of my team who find it useful for anything other than identifying typos.

The push for copilot usage is being driven by management at every level.

Re: Watching AI drive Microsoft employees insane

#165

Post by sbarre »

rsynnott wrote: post #41

Beyond every other absurdity here, well, maybe Microsoft is different, but I would never assign a PR that was _failing CI_ to somebody. That that's happening feels like an admission that the thing doesn't _really_ work at all; if it worked even slightly, it would at least only assign passing PRs, but presumably it's bad enough that if they put in that requirement there would be no PRs.

I feel like everyone is applying a worse-case narrative to what's going on here..

I see this as a work in progress.. I am almost certain the humans in the loop on these PRs are well aware of what's going on and have their expectations in check, and this isn't just "business as usual" like any other PR or work assignment.

This is a test. You can't improve a system without testing it on real world conditions.

How do we know they're not tweaking the Copilot system prompts and settings behind the scenes while they're doing this work?

Can no one see the possibility that what is happening in those PRs is exactly what all the people involved expected to have happen, and they're just going through the process of seeing what happens when you try to refine and coach the system to either success or failure?

When we adopted AI coding assist tools internally over a year ago we did almost exactly this (not directly in GitHub though).

We asked a bunch of senior engineers to see how far they could get by coaching the AI to write code rather than writing it themselves. We wanted to calibrate our expectations and better understand the limits, strengths and weaknesses of these new tools we wanted to adopt.

In most of those early cases we ended up with worse code than if it had been written by humans, but we learned a ton. We can also clearly see how much better things have gotten over time, since we have that benchmark to look back on.

Re: Watching AI drive Microsoft employees insane

#166

Post by serial_dev »

eterevsky wrote: post #18
margorczynski wrote: post #8

With how stochastic the process is it makes it basically unusable for any large scale task. What's the plan? To roll the dice until the answer pops up? That would be maybe viable if there was a way to automatically evaluate it 100% but with a human in the loop required it becomes untenable.

The plan is to improve AI agents from their current ~intern level to a level of a good engineer.

This looks much worse than an intern. This feels like a good engineer who has brain damage.

When you look at it from afar, it looks potentially good, but as you start looking into it for real, you start realizing none of it makes any sense. Then you make simple suggestions, it does something that looks like what you asked, yet completely missing the point.

An intern, no matter how bad it is, could only waste so much time and energy.

This makes wasting time and introducing mind-bogglingly stupid bugs infinitely scalable.

Re: Watching AI drive Microsoft employees insane

#167

Post by baalimago »

gf000 wrote: post #125
baalimago wrote: post #64

Well, the coding agent is pretty much a junior dev at the moment. The seniors are teaching it. Give it a 100k PRs with senior developer feedback and it'll improve just like you'd anticipate a junior would. There is no way that FANG aren't using the comments by the seniors as training data for their next version. It's a long-term play to have pricey senior developers argue with an llm

A junior dev is (most often) a bright human being, with not much coding experience yet. They can certainly execute instructions and solve novel problems on their own, and they most certainly don't need 100k PRs to pick up new skills. Equating LLMs to humans is pretty damn.. stupid. It's not even close (otherwise how come all the litany of office jobs that require far less reasoning than software development are not r…

A junior dev may also swap jobs, require vacation days, perks and can't be scaled up at a the click of a button. There are no such issues with an agent. So, if I were a FANG higher-up, I'd invest quite a bit into training LLM-agents who make pesky humans redundant.

Doing so has low risk, the senior devs may perhaps get fed up and quit, and the company might be a laughing stock on public PRs. But the potential value for is huge.

Re: Watching AI drive Microsoft employees insane

#168

Post by Hamuko »

sbarre wrote: post #143
globalise83 wrote: post #21

Malicious compliance should be the order of the day. Just approve the requests without reviewing them and wait until management blinks when Microsoft's entire tech stack is on fire. Then quit your job and become a troubleshooter on x3 the pay.

I know this is meant to sound witty or clever, but who actually wants to behave this way at their job? I'll never understand the antagonistic "us vs. them" mentality people have with their employer's leadership, or people who think that you should be actively sabotaging things or be "maliciously compliant" when things aren't perfect or you don't agree with some decision that was made. To each their own I guess, but I…

Considering that there's daily employee protests against Microsoft now, probably a lot of Microsoft employees want to behave like that.

Re: Watching AI drive Microsoft employees insane

#169

Post by diggan »

yubblegum wrote: post #142
diggan wrote: post #5

Interesting that every comment has "Help improve Copilot by leaving feedback using the or buttons" suffix, yet none of the comments received any feedback, either positive or negative. > This seems like it's fixing the symptom rather than the underlying issue? This is also my experience when you haven't setup a proper system prompt to address this for everything an LLM does. Funniest PRs are the ones that "resolves" t…

This field (SE - when I started out back in late 80s) was enjoyable. Now it has become toxic, from the interview process, to imitating "big tech" songs and dances by small fry companies, and now this. Is there any joy left in being a professional software developer?

> This field (SE - when I started out back in late 80s) was enjoyable. Now it has become toxic

I feel the same way today, but I got started around 2012 professionally. I wonder how much of this is just our fading optimism after seeing how shit really works behind the scenes, and how much the industry itself is responsible for it. I know we're not the only two people feeling this way either, but it seems all of us have different timescales from when it turned from "enjoyable" to "get me out of here".

Re: Watching AI drive Microsoft employees insane

#170

Post by preisschild »

surgical_fire wrote: post #109
diggan wrote: post #5

Interesting that every comment has "Help improve Copilot by leaving feedback using the or buttons" suffix, yet none of the comments received any feedback, either positive or negative. > This seems like it's fixing the symptom rather than the underlying issue? This is also my experience when you haven't setup a proper system prompt to address this for everything an LLM does. Funniest PRs are the ones that "resolves" t…

> I can't imagine how the people who have to deal with this are feeling. It's like you have a junior developer except they don't even read what you're telling them, and have 0 agency to understand what they're actually doing. That comparison is awful. I work with quite a few Junior developers and they can be competent. Certainly don't make the silly mistakes that LLMs do, don't need nearly as much handholding, and te…

> That said, I really don't see how it could meaningfully replace an intern however

And even if it could, how do you get senior devs without junior devs? ^^

Post reply on HN