Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google fixed more Chrome bugs in June than over the past two years, thanks to AI

blog.google

Post reply on HN
Print view View original

361–370 of 668 posts

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#361

Post by ayewo »

unprovable wrote: post #273

The real datapoint was Firefox not paying any money in Berlin's Pwn2Own competition round this May just gone. Unheard of to have nothing confirmed... they've paid out every event since 2007 (I checked). Does this mean we must move past the low-hanging fruit now? Probably... Certainly indicates some usefulness of these models.

Perhaps this was due to their red-teaming partnership [1][2] with Anthropic which they wrote about a few months earlier in March?

1: https://www.anthropic.com/news/mozilla-firefox-security

2: https://blog.mozilla.org/en/firefox/hardening-firefox-anthro...

Previous discussion: https://news.ycombinator.com/item?id=47273854

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#362

Post by ssl-3 »

Tistron wrote: post #128

I have a software degree and some work experience but have been doing other things for a decade or so. I'm curious to try out some "vibe engineering" but it seems a bit daunting to get started, there are so many new tools available on top of the AI itself. Are there good resources for somebody like me to get started, that'll guide me though how to think about prompting, and using CI and how go have the agent successf…

Just jump in.

Build a sandbox, download Codex CLI or Claude Code or whatever and spend some time doing some creative stuff with it just for the sake of learning how it all fits together.

Pay attention to the inputs you provide and the outputs that they result in. Keep your bullshit detector engaged: Bots often lie.

If you get stuck, or it gets stuck, or you want prompting advice or whatever: Ask any frontier-level bot for help. Sometimes, it's very instructive to get help from Claude for an issue with Codex, or vice-versa.

Want better tools? Ask the bot to suggest some that exist. (None of the existing tools fit? Have the bot write new ones.)

All of this stuff is always in a state of flux, so even with the lies they'll do better at teaching than any fixed reference will. They're LLMs and processing language is what they tend to be best at... so go ahead and use that.

And remember: They're designed to behave kinda-sorta like humans, but they are not humans. They're just computer programs. If you don't like their output style, or they don't like your input style: Ask them how to implement rules that make them knock that shit off. :)

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#363

Post by Leynos »

altmanaltman wrote: post #101
madeofpalk wrote: post #72

Earlier quoted context omitted.

Tests! Unit tests, integration tests, random adhoc scripts. You know - TDD! I’ve been working on UI component improvements and it was doing a lousy job until i specifically told it to test in a headless browser to validate it works. I think somewhere in an AGENTS.md i have an instruction to “don’t state your guesses as fact - validate findings and results”.

If the agent can write the tests its supposed to pass and we are worried that agent produces quite a lot of slop (which is why we are doing the tests), then what is the defense against test slop? Test the tests?

1st line of defence, use something like ponytail to enforce brevity. Use property testing and behavioural testing on top of unit testing. Enforce readability standards so you will be able to understand the tests.

2nd line, code review.

Do the first pass with an agent, ask it to bounce back vacuous or tautological tests. Ask it to verify that the tests verify what they claim to. Then read them yourself.

3rd line, mutation testing. If the tests don't actually catch broken code, kill the mutants.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#364

Post by lbrito »

Okay, now how many bugs would have been fixed if Google had spent $130B (Google's AI capex) in human resources instead of AI?

Just to have a sense of proportion, considering an average software engineer salary of $200k, that same money would buy you 650,000 engineer-year salaries. Of course there are other expenses, but that gives you an idea of the order of magnitude.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#365

Post by dylan604 »

FranzFerdiNaN wrote: post #314
stack_framer wrote: post #301

Earlier quoted context omitted.

The useless suggestions are exhausting. One of my co-workers uses Claude for all asynchronous communication, including Slack messages, Jira comments, code reviews, and emails. Every single message from him, literally every time he communicates, it's a massive wall of text, overflowing with scope-creep suggestions like nothing I've ever seen before. It's impossible to ask him a simple question and get a simple answer.…

My product owner does the same. Every ticket she makes is multiple paragraphs now. It drives me up the wall, especially as some tickets are so simple I need two clear lines to know what I need to do. But no, I get 15 lines written by Claude with sometimes conflicting instructions.

it's one thing if it's malicious compliance. it's another if the person has no clue and is leaning into LLMs to get by. the loudest evangelists I experience are from people that are not coders but feel like they can dictate now they can generate code

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#366

Post by branko_d »

On a related note: Visual Studio still has dialogs that cannot be resized, even after years of people begging Microsoft to make them resizable. Shouldn't this be a trivial fix for AI?

I'll believe in AI when I get to resize my Configuration Manager!

https://developercommunity.microsoft.com/t/Resize-configurat...

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#367

Post by AussieWog93 »

Thanemate wrote: post #349
mgfist wrote: post #182

Earlier quoted context omitted.

The truth is that using AI well is a skill much like anything else. There's no one single way to use it much like there's no one single way to write a program that serves hackernews.

>There's no one single way to use it The fact that people jump on the "you're simply using it wrong" wagon in every HN comment section whenever someone shares their negative/neutral experience with it implies otherwise.

There can be many correct ways to use a tool, while still having incorrect ways to use it.

Sometimes you need to set your oven to grill, other times fan force. But if you're complaining about getting burned then, yes, you're simply using it wrong.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#368

Post by Tade0 »

swat535 wrote: post #358
Tade0 wrote: post #86

Earlier quoted context omitted.

It bothers me that you have to explicitly state this to the agent. Makes me think what else is missing from that file which also needs to be explicitly stated, but I don't know what don't know. "Do a good job"?

Additionally, if you are asking agents to write test, be very careful. They will write tautology tests. They will mock things to no end. They will flat out REMOVE assertions (saying it's not needed). They can also write test to assert the wrong result. You have to always review it, it's exhausting.

That is my experience as well. I write tests by hand because they really still can't do it properly.

And I think I know why - neither do most people.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#369

Post by lbrito »

Cthulhu_ wrote: post #58
goldenarm wrote: post #51

Elephant in the room : how many of these bugs were written by LLMs in the first place ? Because creating 100x more bugs and fixing 100x more isn't something to be proud of.

Can't have been much, as the Chrome project is 20+ years old and LLMs have only recently become a thing; they did not slacken their code review and testing practices with the advent of LLM code generators. And I doubt they do much development on the affected areas at the moment, they mention a 13 year old issue for example. But it's an open source project, you can go and figure out whether your assertion is correct.

>Can't have been much, as the Chrome project is 20+ years old and LLMs have only recently become a thing

That's not a great argument. GPT3.5 was launched over 4 years ago, so that's already significant compared to Chrome's age. LLMs notoriously also produce more verbose code than a human, so its not out of question that it might produce 5x more bugs on average given the same time frame.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#370

Post by derefr »

Tade0 wrote: post #86
madeofpalk wrote: post #72

Earlier quoted context omitted.

Tests! Unit tests, integration tests, random adhoc scripts. You know - TDD! I’ve been working on UI component improvements and it was doing a lousy job until i specifically told it to test in a headless browser to validate it works. I think somewhere in an AGENTS.md i have an instruction to “don’t state your guesses as fact - validate findings and results”.

It bothers me that you have to explicitly state this to the agent. Makes me think what else is missing from that file which also needs to be explicitly stated, but I don't know what don't know. "Do a good job"?

An agent doesn’t know what “job” it’s being asked to do until you tell it.

Imagine you’re blind and deaf and have temporary retrograde amnesia. You “wake up” one moment with a memory of some words in your head like “what is the bug?” …but you don’t recall the context of that question, and nor can you look/listen around to observe the context.

So you don’t know whether you’re e.g. at the office, in front of your computer, in the middle of doing some pair-programming (where, yes, you’d in investigate the bug thoroughly with tools), vs. having a conversation with a colleague over lunch (where the expectation is for you to tap into your knowledge + intuitions to either guess or say you don’t know — not to pull out your laptop.

That’s what it’s like to be one of these LLMs being prompted by some agent harness. Unless the harness injects the proper context into its “recent memory”, it just doesn’t know.

Post reply on HN