Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AutoDev: Automated AI-driven development by Microsoft

arxiv.org

Post reply on HN
Print view View original

51–60 of 232 posts

Re: AutoDev: Automated AI-driven development by Microsoft

#51

Post by fhd2 »

onion2k wrote: post #48
fhd2 wrote: post #37

Earlier quoted context omitted.

Good take! I think one specific development we might see would be an end of the SaaS micro tool era. Right now, most companies I advise have an absolute _jungle_ of different tools and a bunch of interns manually bridging the gap between them. If software development becomes a lot cheaper, wouldn't companies be more likely to just roll their own versions of what they need, to meet their needs exactly and have everyth…

I think one specific development we might see would be an end of the SaaS micro tool era. Possibly, but if SaaS tool providers make APIs so AI tools can consume the data, that would open up a huge ecosystem of AI 'plugins' that would be immensely valuable. I think OpenAI have something like that already.

Could be! Most SaaS tools I can think of cover some specific work flow in a way that it kinda works for 80% of the customers. Their UI is quite an intrinsic part of the offering, an API-only version of it wouldn't be as valuable. But if nobody has to put up with only getting 80% of their needs met, why would they? I'd imagine we'd have a few very general purpose systems that can indeed be glued together in a largely automated fashion. But getting the business processes into them in a meaningful way would then be the hard part, the programming.

I can encourage everyone to read good old "Domain Driven Design", there's a world of interesting problems and work out there beyond the purely technical side of things. And I think that part is a lot more difficult to automate than creating a WordPress site from a .psd.

Re: AutoDev: Automated AI-driven development by Microsoft

#52

Post by torginus »

jerpint wrote: post #47
PodgieTar wrote: post #34

I guess I'd be interested to see how this performs against the same benchmark Devin was using. It's hard to deny that this isn't impressive. But I think there's two interesting parts to it. Claude 3 Opus already scored around 85-86% on these benchmarks, without an "AutoDev" style agentic approach. And all the same problems with HumanEval remain, the limitations in terms of what style of problems are chosen, and real…

I agree that these benchmarks don’t mean as much anymore because it’s highly likely they were already present in the training set, but also believe it’s likely these tools will be significantly better in a few research cycles

A significant number of bugs just end in 'stupid mistake I didn't notice' or 'weird behaviour with a fix described on SO/docs/forum post'. Current day LLMs are much better positioned to solve these issues than humans are.

Re: AutoDev: Automated AI-driven development by Microsoft

#53

Post by EForEndeavour »

__loam wrote: post #31
schappim wrote: post #5

[flagged]

And I suppose you pasted this from some ai summary.

The post-ChatGPT world has made me hypersensitive to the "AI smell" of bullet lists where each item ends with a fullstop. (Insert standard discussion points about whether it really even matters who wrote it, I only notice the obvious cases, humans have been spewing low quality content since the invention of language, ...)

Re: AutoDev: Automated AI-driven development by Microsoft

#54

Post by wobblyasp »

throwaway22001 wrote: post #44
wobblyasp wrote: post #30

Earlier quoted context omitted.

Isn't this just wrapping existing agents with a "secure" coding environment? Does this represent a meaningful improvement in AI?

> Isn't this just wrapping existing agents with a "secure" coding environment? It is. The paper has this paragraph where autodev requires rules for it's setup. > The user initiates the process by configuring rules and actions through yaml files. These files define the available commands (actions) that AI agents can perform. Users can leverage default settings or fine-grained permissions by enabling/disabling specific…

Thank you for the clarification; I read through the paper and thought I missed something.

Definitely any interesting area of development with papers like this and tools like Devin being released. Interesting times.

Re: AutoDev: Automated AI-driven development by Microsoft

#55

Post by EForEndeavour »

jaylittle wrote: post #40

It's really depressing to see that big tech is essentially universally pushing Snake Oil. AI is a lie. LLMs are a legit tech that have some purpose, but LLMs will never evolve into anything remotely resembling the actual definition of AI. They are flowery language generators and they will never be able to reason, understand, debate and criticize. They know nothing and therefore embody nothing. No matter how much comp…

rebuttals:

- LLM capability has grown spectacularly fast. Google published the transformers paper in 2017. Remember when OpenAI said GPT-2 was too dangerous to release? That was five years ago. Now look where they've gone and where trillions of dollars are going. https://news.ycombinator.com/item?id=20912556

- what is the "actual definition of AI"? Do you mean AGI? Machine learning and deep learning have been subsets of AI since their inception.

- LLMs or any AI tech does not need to "reason, understand, debate and criticize" to be useful, disrupt the economy, and change how people work and live.

Re: AutoDev: Automated AI-driven development by Microsoft

#56

Post by bigyikes »

bregma wrote: post #27
saran945 wrote: post #2

If LLM/Agents contribute to software development, how does the role of software engineers evolve? SE should focus on : - System design - Integrations - Project management etc or the job will disappear in 10 years ?

Someone will still need to review the reams and reams of bullshit code generated by these Markov chains. It'll be like all your code gets written by first-term co-op students who have discovered alcohol for the first time and think it's pretty neat, but you're still responsible for the final output and don't get to do any of the fun and interesting stuff. Just constant, mind-numbing code reviews of something that mig…

You described v1. Just wait for v2.

You can’t possibly think these things will remain clowns forever.

These things aren’t “Markov chains” - the architecture is significantly more scalable, which is exactly why this time is different.

Re: AutoDev: Automated AI-driven development by Microsoft

#57

Post by datavirtue »

fhd2 wrote: post #25
Havoc wrote: post #9

[flagged]

I'm not on the hype train myself, but I'd never say never. I can certainly picture much of what devs do today become automated. Heck, we already have plain old deterministic tools that are ridiculously productive compared to common tech stacks in the industry. That said, it is the very nature of software development (or any endeavor in building complex systems) that we discover a lot of the detail problems later down…

You still have to do the hard part. Writing all the prompts. That's what we do in all those meetings that sap our time, energy, and patience. Even if we complete all that we rely on devs to identify and/or fill in gaps along the way.

If you think your job is on the chopping block by way of AI then you are confused about the value you offer. I'm often disgusted that even senior devs are confused about what their job is.

AI is an amazing tool for software development--the perfect immediate application. Pick it up and use it and don't look back.

Re: AutoDev: Automated AI-driven development by Microsoft

#59

Post by WanderPanda »

PodgieTar wrote: post #34

I guess I'd be interested to see how this performs against the same benchmark Devin was using. It's hard to deny that this isn't impressive. But I think there's two interesting parts to it. Claude 3 Opus already scored around 85-86% on these benchmarks, without an "AutoDev" style agentic approach. And all the same problems with HumanEval remain, the limitations in terms of what style of problems are chosen, and real…

I’ve been using Github copilot daily for two years and ChatGPT for 1 year now. And I think the tide lifts all the boats. I’ve seen a (perceived) 2-3x productivity increase. I think these tools slightly favor people in front of the learning curve of a particular field. I’ve been dabbling in all sorts of things so if you’re a focused expert (who doesn’t need to explore but just exploit) you probably get less than a 2x boost from using LLMs.

I can see LLMs eating into the expert regime IF they get another 5-10x better. But even in that case human (expert) knowledge will be required to know what is possible and hence what to ask (kind of like reward function design in reinforcement learning)

Re: AutoDev: Automated AI-driven development by Microsoft

#60

Post by flir »

UK-Al05 wrote: post #15
davedx wrote: post #11

Earlier quoted context omitted.

Yeah. The shocking speed of progress on AI - especially for coding - tells me quite clearly what the future holds for much of software engineering. Despite the human factors (yes, requirements gathering and specification is messy) our profession is intrinsically automatable at the end of the day. Being in denial is pointless.

I think software engineering will just go up the stack to requirements engineering. Those requirements will be specified as tests or in a very high level language that doesn't allow for ambiguous meanings. Which is essentially another programming language just more high level.

If the hypothetical high level language doesn't allow for ambiguous meanings, you can write a compiler for it and you don't need the AI.

(Not saying you're wrong about "higher and higher levels of abstraction" - that's the way it's always been - but the unique promise of AI is the ability to deal with ambiguity in the requirements).

Post reply on HN