Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google fixed more Chrome bugs in June than over the past two years, thanks to AI

blog.google

Post reply on HN
Print view View original

101–110 of 668 posts

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#101

Post by altmanaltman »

madeofpalk wrote: post #72
stogot wrote: post #56

Earlier quoted context omitted.

What tooling makes this go?

Tests! Unit tests, integration tests, random adhoc scripts. You know - TDD! I’ve been working on UI component improvements and it was doing a lousy job until i specifically told it to test in a headless browser to validate it works. I think somewhere in an AGENTS.md i have an instruction to “don’t state your guesses as fact - validate findings and results”.

If the agent can write the tests its supposed to pass and we are worried that agent produces quite a lot of slop (which is why we are doing the tests), then what is the defense against test slop? Test the tests?

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#103

Post by oblio »

UqWBcuFx6NV4r wrote: post #95
VBprogrammer wrote: post #17

I've recently been using AI a lot for performance optimisation during a particularly busy period at work. I would say it was almost completely useless at the high-level direction - it would point out suspicious parts of SQL queries for example but on back to back testing these almost never resulted in any performance change. In fact, if it wasn't for the fact that it made making the actual changes I identified much e…

You have to learn to use your tools, not try whatever intuitively made sense to you at first (expecting the tool to do all the work) and then whining on Hacker News when it doesn’t work out for you.

This part is obviously their fault and they need to use their tools better:

> others dumping their raw AI output at me

Sheesh.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#104

Post by KptMarchewa »

herrkanin wrote: post #34
VBprogrammer wrote: post #17

I've recently been using AI a lot for performance optimisation during a particularly busy period at work. I would say it was almost completely useless at the high-level direction - it would point out suspicious parts of SQL queries for example but on back to back testing these almost never resulted in any performance change. In fact, if it wasn't for the fact that it made making the actual changes I identified much e…

The thing that makes it work really well is to make sure it has all the tooling to verify its hypotheses. If you allow it to run the full lifecycle in loops you will be surprised how well it works.

Exactly. In the above case, AI should be able to verify it's own hypothesis by having access to the database's table statistics and planner.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#105

Post by KptMarchewa »

madeofpalk wrote: post #72
stogot wrote: post #56

Earlier quoted context omitted.

What tooling makes this go?

Tests! Unit tests, integration tests, random adhoc scripts. You know - TDD! I’ve been working on UI component improvements and it was doing a lousy job until i specifically told it to test in a headless browser to validate it works. I think somewhere in an AGENTS.md i have an instruction to “don’t state your guesses as fact - validate findings and results”.

In this particular case tests won't help you, without the actual data. But overall you're right.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#106

Post by KptMarchewa »

altmanaltman wrote: post #101
madeofpalk wrote: post #72

Earlier quoted context omitted.

Tests! Unit tests, integration tests, random adhoc scripts. You know - TDD! I’ve been working on UI component improvements and it was doing a lousy job until i specifically told it to test in a headless browser to validate it works. I think somewhere in an AGENTS.md i have an instruction to “don’t state your guesses as fact - validate findings and results”.

If the agent can write the tests its supposed to pass and we are worried that agent produces quite a lot of slop (which is why we are doing the tests), then what is the defense against test slop? Test the tests?

Existing tests.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#107

Post by IanCal »

VBprogrammer wrote: post #17

I've recently been using AI a lot for performance optimisation during a particularly busy period at work. I would say it was almost completely useless at the high-level direction - it would point out suspicious parts of SQL queries for example but on back to back testing these almost never resulted in any performance change. In fact, if it wasn't for the fact that it made making the actual changes I identified much e…

This is the research taste part - often they can be good but human experts are really good at this (also you know your codebase, without the prompt these models have no other background).

Then being able to suggest several things to try and have them go off and build, measure and tweak is hugely useful in my experience.

Also things like making custom visualisations for comparing changes.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#108

Post by Filligree »

Tade0 wrote: post #86
madeofpalk wrote: post #72

Earlier quoted context omitted.

Tests! Unit tests, integration tests, random adhoc scripts. You know - TDD! I’ve been working on UI component improvements and it was doing a lousy job until i specifically told it to test in a headless browser to validate it works. I think somewhere in an AGENTS.md i have an instruction to “don’t state your guesses as fact - validate findings and results”.

It bothers me that you have to explicitly state this to the agent. Makes me think what else is missing from that file which also needs to be explicitly stated, but I don't know what don't know. "Do a good job"?

Will, it depends on the AI. Anthropic used to have a lecture-length system prompt for their models to explain this stuff—part of the secret sauce for Claude Code—and famously found that the 5 series models no longer need it.

As usual, if you use anything but the best model available I’m going to state that the better ones do better. If you do use the best model available, then I’ll just mention that Fable still has limits and still needs some guidance.

One thing it does not do is deliberately build tests which test nothing at all, or which restate the code under test. I mention this because certain other models absolutely would.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#109

Post by Filligree »

altmanaltman wrote: post #101
madeofpalk wrote: post #72

Earlier quoted context omitted.

Tests! Unit tests, integration tests, random adhoc scripts. You know - TDD! I’ve been working on UI component improvements and it was doing a lousy job until i specifically told it to test in a headless browser to validate it works. I think somewhere in an AGENTS.md i have an instruction to “don’t state your guesses as fact - validate findings and results”.

If the agent can write the tests its supposed to pass and we are worried that agent produces quite a lot of slop (which is why we are doing the tests), then what is the defense against test slop? Test the tests?

Read the tests.

Doing a pass where you just ask the AI to sanity-check the existing tests (against rules like “test against the spec, not the implementation” can also help.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#110

Post by ahartmetz »

Maybe they could ask the AI to fix their atrocious build times. Only in Chrome do you have 300 line source files that blow up to 20 megabytes after preprocessing as a matter of course. 3 GB per compile job is getting more common - and RAM is expensive right now!
Post reply on HN