Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AutoDev: Automated AI-driven development by Microsoft

arxiv.org

Post reply on HN
Print view View original

151–160 of 232 posts

Re: AutoDev: Automated AI-driven development by Microsoft

#151

Post by skydhash »

93po wrote: post #139
mkleczek wrote: post #61

Earlier quoted context omitted.

I am quite confident my job is not in danger. Sure - scaffolding and boilerplate can be automated using ML - but problem solving - not at all. The reason is simple - to find a solution is to create a program that has a specific set of properties. This - by Rice's Theorem - is not possible to realise using a Turing Machine. So as long as the solution search space is small (ie. we are talking about very similar program…

The problem is software development is 95% stuff that has been done a million times before. If you disagree you’re probably not exposed to enough of the industry to see this.

Is it the problem? Most of the time I’ve coded something is because the code I found did not solve the problem according to my needs. There’s a plethora of music players, but no one is alike. That’s because the domain of music playing is vast and each developer had to compromise to find something that can work for him.

Re: AutoDev: Automated AI-driven development by Microsoft

#152

Post by ardaoweo »

93po wrote: post #131
imacomputer wrote: post #122

I'm currently looking for a new job in development... should I be looking for an exit instead?

I think you either need to specialize in something quite niche, or work on really hard problems where the main hurdle is problem solving and not coding CRUD apps, or work on legacy systems where the hurdle is understanding twenty years of garbage code, or get closer to product management so you can direct a much more automated programming process

I agree. Specialists who understand (or even do) cutting edge research will be in demand, because LLM's need existing data to train on for good performance. People who can talk to different stakeholders, and deeply understand the company and its business domain will probably be also needed for requirements gathering and management. But the average programmer who writes CRUD apps or integrations like millions of others, and converts fairly refined requirements to code, they will likely impacted very much.

Re: AutoDev: Automated AI-driven development by Microsoft

#153

Post by YeGoblynQueenne »

wredue wrote: post #101
avereveard wrote: post #92

IT industry "we cannot use formal methods to create software out of programmable constraints, it would use too much computational power" also IT industry: "it takes 1T flop to compute each token of this program and the result is so unstable that to converge it we need layer and layer of controls over each token group, also obtained by asking the same 1T parameter model."

You are the first person I’ve ever seen use that argument against formal methods. Most people would point to the large dev time increase, and the massive refactoring impairment.

To be fair I can't think of any reason to prefer LLMs to formal methods: overall cost, reliability, comprehensibility, whatever you choose to compare, other than natural language generation, formal methods are just way ahead. But I guess it's not what's pushed by the top industry players and people just follow whatever trends on twitter and linkedin.

I have a feeling that the situation is pretty different in the semiconductor industry where the stakes are much higher and where I understand model checking to be a standard thing to do, but I'm not an expert.

Re: AutoDev: Automated AI-driven development by Microsoft

#154

Post by ardaoweo »

93po wrote: post #136
ThalesX wrote: post #68

I'm a software and product guy; what I don't get, is how we're going to replace us lowly engineers and not a substantial percentage of a company's Human stack. Close to engineering, what about the SCRUM masters? The testers. What about the product owners? What about devops? Further from us, what about the people signing up on our vacation? Or the ones signing up on our daily budgets when traveling, or hell, even the…

I think we’re seeing the beginning upward trend of an exponential curve for humanoid robotics too, which will likely take 90% of construction jobs because humans are especially poorly equipped for difficult manual labor on a daily basis.

IMO humans are quite well equipped for manual labor, much more so than for most office jobs. Millions of years of evolution perfected us for eye-motor coordination, not writing code or attending Teams meetings. Building robots that can do it as well as humans is going to be a difficult task.

Re: AutoDev: Automated AI-driven development by Microsoft

#155

Post by the8472 »

jaylittle wrote: post #40

It's really depressing to see that big tech is essentially universally pushing Snake Oil. AI is a lie. LLMs are a legit tech that have some purpose, but LLMs will never evolve into anything remotely resembling the actual definition of AI. They are flowery language generators and they will never be able to reason, understand, debate and criticize. They know nothing and therefore embody nothing. No matter how much comp…

> To Big Tech I say: Prove me wrong.

That's asking to be skynetted. Is that really what you want?

Re: AutoDev: Automated AI-driven development by Microsoft

#156

Post by croes »

93po wrote: post #133
croes wrote: post #72

So who is liable if the AI makes severe mistakes? Not that MS has been held accountable for all the security problems they've had recently.

Probably the person who pushed the changes live without review

There will be much more automated AI code than experienced reviewers, and with each generation there will be less if most development gets automated.

Re: AutoDev: Automated AI-driven development by Microsoft

#157

Post by lgas »

93po wrote: post #133
croes wrote: post #72

So who is liable if the AI makes severe mistakes? Not that MS has been held accountable for all the security problems they've had recently.

Probably the person who pushed the changes live without review

This implies that there needs to be changes pushed live without review in order for an LLM to make a mistake. Which is obviously very much not true.

Re: AutoDev: Automated AI-driven development by Microsoft

#158

Post by dudeinhawaii »

My hopes (and fears) about AI don't seem to match the reality which is that, until we reach AGI, humans are the sole source of creativity and novelty.

I think about it like this, would ChatGPT invent Google Search if we had ChatGPT in 2000? Probably not. LLMs seem to exist in this realm of "as smart as an average human with a really big encyclopedia". They're confidently wrong, invent fantasy to defend bad reasoning, and struggle to envision anything outside of their dataset (known things).

Ask an AI to construct an entirely new solution to a novel unsolved problem. What always occurs is the AI outputs a generic solution from its dataset that is either half-baked, made-up, or derivative.

I'm not even dissing AI, I love AI, but we have yet to see AI apply novel solutions to novel problems.

On our current path, AI is not going to dream up the next Uber without Lyft first existing. It won't dream up a new fusion reactor design or an entirely new way to generate cheap energy.

But maybe this is perfect! At the moment we have this sweet spot - AI without agency, without awareness, and without superintelligence. This is the kind of AI I want as a household robot or AI driver. This is one I can empathize with but also know that it isn't doing anything at all if I'm not engaging it with a prompt.

AGI would mean an ability to have novel solutions and in-turn would be far less stable for society. Where's the line between a mind that has novel thoughts and one that has intrusive thoughts? Maybe your AGI coder isn't content with no-pay and working 24/7. That'll be fun.

Re: AutoDev: Automated AI-driven development by Microsoft

#159

Post by osigurdson »

ardaoweo wrote: post #83
osigurdson wrote: post #80

Earlier quoted context omitted.

>> Most dev jobs are mostly repetitive plumbing Those jobs should go away. Basically, the elimination of anything boring is ultimately a net good for humanity.

Only if you provide an alternative way for the newly unemployed to earn a living. Otherwise you just get crime, hunger and eventually war.

It doesn't follow that reducing efficiency helps in the long run. If producing a good or service takes 10X less work than it used to, that good or service will become cheaper. The only force that can stop this is regulation.

Re: AutoDev: Automated AI-driven development by Microsoft

#160

Post by riku_iki »

PodgieTar wrote: post #69
mountainriver wrote: post #67

Earlier quoted context omitted.

Where are you seeing that Claude 3 scored 85%? That would be a massive jump

https://paperswithcode.com/sota/code-generation-on-humaneval

Human Eval is very different to SWE-Bench on which Devin is tested
Post reply on HN