Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AutoDev: Automated AI-driven development by Microsoft

arxiv.org

Post reply on HN
Print view View original

61–70 of 232 posts

Re: AutoDev: Automated AI-driven development by Microsoft

#61

Post by mkleczek »

davedx wrote: post #11
Havoc wrote: post #9

[flagged]

Yeah. The shocking speed of progress on AI - especially for coding - tells me quite clearly what the future holds for much of software engineering. Despite the human factors (yes, requirements gathering and specification is messy) our profession is intrinsically automatable at the end of the day. Being in denial is pointless.

I am quite confident my job is not in danger. Sure - scaffolding and boilerplate can be automated using ML - but problem solving - not at all.

The reason is simple - to find a solution is to create a program that has a specific set of properties. This - by Rice's Theorem - is not possible to realise using a Turing Machine. So as long as the solution search space is small (ie. we are talking about very similar programs to generate) - ML based methods are fine.

But once we are talking about finding solutions to non-trivial problems - I am sure human is needed.

Re: AutoDev: Automated AI-driven development by Microsoft

#62

Post by amlib »

pjmlp wrote: post #35
Kuinox wrote: post #20

Earlier quoted context omitted.

Management ? I'm not sure, if the AI makes engineers 10-20x more efficient, you will need way less engineers, and way less managers.

There is always a spot for more managers, welcome to big corps.

Its Managers all the way down...

Re: AutoDev: Automated AI-driven development by Microsoft

#63

Post by flir »

password54321 wrote: post #46
davedx wrote: post #11

Earlier quoted context omitted.

Yeah. The shocking speed of progress on AI - especially for coding - tells me quite clearly what the future holds for much of software engineering. Despite the human factors (yes, requirements gathering and specification is messy) our profession is intrinsically automatable at the end of the day. Being in denial is pointless.

"The shocking speed of progress on AI" Not sure what is shocking. Some of us used Codex in 2021 and nothing has changed since that has revolutionised software development by any capacity. Feels like people are just saying whatever some tech CEO or worse, some guru says. Progress isn't some infinite exponential either where you can just keep increasing the capacity of models and expect better results. Also not sure wh…

Yeah, to me it feels like a step change to a new level, followed by a lot of exploration of that new level. It feels like there's lots of exploring left to do, but we're not going to reach "magic box that turns English into flawless programs" on the current level.

Speaking just to programming, yes there's probably lots of AI-generated code in the training data now. But the generation of that code was guided by, and (presumably) confirmed to be working by, a human. So it's been filtered.

I suspect that AI-and-human produces better code than human, on average. I know that if I throw any twenty lines of my code into it, it can usually suggest some improvement, even if it's just a variable name.

Re: AutoDev: Automated AI-driven development by Microsoft

#64

Post by UK-Al05 »

flir wrote: post #60
UK-Al05 wrote: post #15

Earlier quoted context omitted.

I think software engineering will just go up the stack to requirements engineering. Those requirements will be specified as tests or in a very high level language that doesn't allow for ambiguous meanings. Which is essentially another programming language just more high level.

If the hypothetical high level language doesn't allow for ambiguous meanings, you can write a compiler for it and you don't need the AI. (Not saying you're wrong about "higher and higher levels of abstraction" - that's the way it's always been - but the unique promise of AI is the ability to deal with ambiguity in the requirements).

Ambiguity in requirements is where the majority of bugs come from now. So I don't think AI would improve things.

Re: AutoDev: Automated AI-driven development by Microsoft

#65

Post by smokel »

UK-Al05 wrote: post #8
smokel wrote: post #7

Great achievement, but what a horrible future we are facing. Instead of progressing towards more powerful programming interfaces with less cause for misinterpretation, we are going to automate the silly process of writing redundant unit tests to check if the behavior that we wanted was encoded properly. Why not skip this nonsense and have the code generated from the behavior in the first place?

Because test define the behaviour. The hardest part of requirements is describing exactly what you want. Tests are great way of doing that. I think AI software development is going to involve writing tests, which the AI agents then get passing. Or some other requirements language that allows for exactness where English can fall down.

It all depends of course, but I tend to disagree.

A requirement such as "A web interface to play the game of chess, but with all the pieces replaced by photos of my family members" is fairly adequate if I were to make a Christmas family game.

I am totally uninterested in specifying whether it is possible to have two white bishops on black squares. Also, I don't want to test whether my uncle's moustache is the proper size.

I'd much prefer to iteratively build the game by prompting than by specifying thousands of details. It just seems the wrong way around.

Re: AutoDev: Automated AI-driven development by Microsoft

#66

Post by margorczynski »

WanderPanda wrote: post #59
PodgieTar wrote: post #34

I guess I'd be interested to see how this performs against the same benchmark Devin was using. It's hard to deny that this isn't impressive. But I think there's two interesting parts to it. Claude 3 Opus already scored around 85-86% on these benchmarks, without an "AutoDev" style agentic approach. And all the same problems with HumanEval remain, the limitations in terms of what style of problems are chosen, and real…

I’ve been using Github copilot daily for two years and ChatGPT for 1 year now. And I think the tide lifts all the boats. I’ve seen a (perceived) 2-3x productivity increase. I think these tools slightly favor people in front of the learning curve of a particular field. I’ve been dabbling in all sorts of things so if you’re a focused expert (who doesn’t need to explore but just exploit) you probably get less than a 2x…

But how many experts do you need? Most dev jobs are mostly repetitive plumbing and those might disappear very fast because 1 dev + LLM >= 5 devs without. So what we'll see is an increase in company margins and an elimination of a large swathe of the middle class.

The alternative theory is that if everyone can now quickly create systems multiple companies and competing products will pop off which will drive down the margins instead but creating a compelling product requires much more than just software engineering skills.

Either way this doesn't look great for devs, especially the ones that are entering the workforce now or will be in the nearest future.

Re: AutoDev: Automated AI-driven development by Microsoft

#67

Post by mountainriver »

PodgieTar wrote: post #34

I guess I'd be interested to see how this performs against the same benchmark Devin was using. It's hard to deny that this isn't impressive. But I think there's two interesting parts to it. Claude 3 Opus already scored around 85-86% on these benchmarks, without an "AutoDev" style agentic approach. And all the same problems with HumanEval remain, the limitations in terms of what style of problems are chosen, and real…

Where are you seeing that Claude 3 scored 85%? That would be a massive jump

Re: AutoDev: Automated AI-driven development by Microsoft

#68

Post by ThalesX »

I'm a software and product guy; what I don't get, is how we're going to replace us lowly engineers and not a substantial percentage of a company's Human stack.

Close to engineering, what about the SCRUM masters? The testers. What about the product owners? What about devops? Further from us, what about the people signing up on our vacation? Or the ones signing up on our daily budgets when traveling, or hell, even the ones we interview with.

In my closest group of friends (we're all seniors in our domains, and very honest with eachother), I find that only the construction worker's job should be safe. And compared to myself and the devops guy, most others have what they themselves describe as trivial and bullshit jobs. Join a meeting, do some paper pushing, some document signing, a little coffee and the day is over by 1PM.

Am I seeing all these AIs replacing programming because I'm on a board where maybe a lot of us are programmers? Is it the same for other roles? Wouldn't it make more sense to have the interview process automated by LLMs if they're capable of building great software, before we replace those hired?

I'm very confused by all the hype when matched with my experience of using LLMs daily for the past years.

Re: AutoDev: Automated AI-driven development by Microsoft

#69

Post by PodgieTar »

mountainriver wrote: post #67
PodgieTar wrote: post #34

I guess I'd be interested to see how this performs against the same benchmark Devin was using. It's hard to deny that this isn't impressive. But I think there's two interesting parts to it. Claude 3 Opus already scored around 85-86% on these benchmarks, without an "AutoDev" style agentic approach. And all the same problems with HumanEval remain, the limitations in terms of what style of problems are chosen, and real…

Where are you seeing that Claude 3 scored 85%? That would be a massive jump

https://paperswithcode.com/sota/code-generation-on-humaneval
Post reply on HN