Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

After outages, Amazon to make senior engineers sign off on AI-assisted changes

arstechnica.com

Post reply on HN
Print view View original

381–390 of 510 posts

Re: After outages, Amazon to make senior engineers sign off on AI-assisted changes

#381

Post by 8note »

daxfohl wrote: post #287

You could create an agent template for each incident you've ever had, with context pre-cached with the postmortem report, full code change, and any other information about the incident. Then for every new PR you could clone agents from all those templates and ask whether the PR could cause something similar to the pre-loaded incident. If any of them say yes, reject the PR unless there's a manual override. You'd never…

code review is too late to give some of that feedback, and design/requirements documents dont have nearly the standardization of presentation and feedback tools for that to be useable.

Amazon does have those things, and has fine tuning on models based on those postmortems.

Noisy reviews are also a problem causer. the PR doesnt know what scale a chunk of code is running at, without having access to 20 more packages and other details.

Re: After outages, Amazon to make senior engineers sign off on AI-assisted changes

#382

Post by booleandilemma »

testbjjl wrote: post #355

Then what’s the point of AI? Pay for the code gen, pay a human to review the code gen, when the senior can train a junior and coordinate output with their incentives and performance reviews, problems largely solved. Seems to me too low level in everyone’s stack to not have humans doing the work, especially at this stage. But what do I know, I certainly am not at the helm of a multibillion dollar operation.

To make the AI companies money and prop up the American economy, obviously.

Re: After outages, Amazon to make senior engineers sign off on AI-assisted changes

#383

Post by booleandilemma »

agoodusername63 wrote: post #299
zcw100 wrote: post #180

I just met a guy from Amazon this past weekend who was bragging, "We've got unlimited access to LLMs and our developers have 10 agents going at a time.". I tried telling him it wasn't all unicorns and rainbows but I didn't get the impression he cared and just kept crapping out skittles.

Because he doesn’t. The impression I get from SWEs I’ve met throughout my life is that most of them don’t actually care about their job. They got in because it paid well and demand was plentiful.

It's rare when I meet one who actually likes writing code nowadays. At my last company everyone was trying to be an architect.

Re: After outages, Amazon to make senior engineers sign off on AI-assisted changes

#384

Post by cobolcomesback »

cmiles74 wrote: post #263
belval wrote: post #23

Earlier quoted context omitted.

I am not in that specific meeting but it made me chuckle that a weekly ops meeting will somehow get media attention. It's been an Amazon thing forever. Wait until the public learns about CoEs!

A weekly ops meeting where they talk about ensuring PRs with AI contributions get extra scrutiny? I think that's significant news.

There was nothing mentioned in the meeting or messaging about PRs with AI contributions. There are no extra requirements for review or scrutiny of AI-generated-code. The media reports about this have been excessively misleading about this.

Re: After outages, Amazon to make senior engineers sign off on AI-assisted changes

#386

Post by 8note »

Insanity wrote: post #75

It's only going to get worse with the brain drain as a result of the layoffs. Which will increase the use of AI assisted coding and increase the number of outages related to this. Imagine having to debug code that caused an outage when 80% is written by an LLM and you now have to start actually figuring out the codebase at 2am.. :)

but thats what it was like when i started at amazon in 2016?

i think the team i was on was a bit of an outlier in terms of owning 40 dumptser fires at once, and the first time reading any one of them was at 2AM because it was down.

having an LLM give early passes on reading the godawful c++ code that you can tell at a glance that its not gonna work as expected, but you cant tell why, or what expected actually is would have been phenomenal, and gotten me back to sleep at 3 on those codebases rather than 5.

Re: After outages, Amazon to make senior engineers sign off on AI-assisted changes

#387

Post by cobolcomesback »

emp17344 wrote: post #276
mattgreenrocks wrote: post #222

Earlier quoted context omitted.

The defensiveness is almost as interesting as the meeting itself.

Way too many people have tied their egos to the success of AI.

And too many people have their egos tied to its failure, too.

Im a massive AI skeptic. If anyone were to be jumping up and down on the corpse of AI and this incessant drive to use it everywhere, it’d be me. But I also work at Amazon. I got the email. I attended the meeting. I can personally attest that there are no new requirements for AI-generated code. The articles about this in the meeting at extremely misleading, if not outright wrong. But instead of believing the person that was actually there in the room, this thread is full of people dismissing my first-hand account of the situation because it doesn’t align with the “haha AI failed” viewpoint.

Re: After outages, Amazon to make senior engineers sign off on AI-assisted changes

#388

Post by 8note »

fredgrott wrote: post #114

Curious question, how many Amazon Engineers flunk basic CS? If you know CS you know two things: 1. AI can not judge code either noise or signal, AI cannot tell. 2. CS-wise we use statistic analysis to judge good code from bad. How much time does it take to take AI output and run the basic statistic tools for most computer languages? Some juniors need firing outright

that isnt computer science at all though? if it is, that would be phd research topics, rather than than basics.

maybe as software engineering topics, but thats a different discipline

Re: After outages, Amazon to make senior engineers sign off on AI-assisted changes

#389

Post by 8note »

andsoitis wrote: post #2

> Amazon’s website and shopping app went down for nearly six hours this month in an incident the company said involved an erroneous “software code deployment.” The outage left customers unable to complete transactions or access functions such as checking account details and product prices. The environment breathed a little.

hardly.

as an alternative, a bunch of people got into their one-person trucks and drove to the store to buy whatever thing would have been efficiently delivered

Re: After outages, Amazon to make senior engineers sign off on AI-assisted changes

#390

Post by happytoexplain »

remarkEon wrote: post #149
happytoexplain wrote: post #19

>Junior and mid-level engineers can no longer push AI-assisted code without a senior signing off Review by a senior is one of the biggest "silver bullet" illusions managers suffer from. For a person (senior or otherwise) to examine code or configuration with the granularity required to verify that it even approximates the result of their own level of experience, even only in terms of security/stability/correctness, r…

Other than “don’t hire idiots”, what is the solution to this problem? I agree with you, and this particular systems management issue is not constrained to software.

I don't know.

We need smart people at every layer. If leadership isn't in that category, it spreads to all layers.

I don't know how we defeat capitalism to incentivize smart leadership. It's fundamentally opposed to market forces.

Post reply on HN