Live data from Hacker News

Google fixed more Chrome bugs in June than over the past two years, thanks to AI

blog.google

81–90 of 668 posts

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#83

I've recently been using AI a lot for performance optimisation during a particularly busy period at work. I would say it was almost completely useless at the high-level direction - it would point out suspicious parts of SQL queries for example but on back to back testing these almost never resulted in any performance change. In fact, if it wasn't for the fact that it made making the actual changes I identified much e…

The thing that makes it work really well is to make sure it has all the tooling to verify its hypotheses. If you allow it to run the full lifecycle in loops you will be surprised how well it works.

Built exactly this for work a few months ago. Claude has a read-only account to the dev DB and a bunch of python postgres tools, I point it at queries/views and ask it optimize, and it has to:

1) verify identical query results

2) run repeatedly to get average, worst, best, etc duration of runs

Sped up so many legacy things that none of us were ever going to bother with.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#84
post #44

Earlier quoted context omitted.

Then you're not grounding it to reality properly. LLMs get bad pretty quickly when you just tell them to pull the answer out of thin air. Pin them to reality with actual performance tests to run to test theories and the results will be much better. Performance optimization involves simulating real world loads and measurable results. Give them the opportunity to actually have a closed loop if you want more than trivia…

Have you any example materials that show a harness that an agent can be pointed at an app with some sort of telemetry tool, the results it gained and the cost of doing so? Because my experience is the same as the parents - the LLM goes on massive tangents, and the more tangents it goes on the worse the results get.

Ultimately the harness is me and the experience is like managing an unruly toddler. You have to pay attention to what it does and issue corrections. Skills and prompts and AGENTS.md do some, nested sets of agents do some, but over it all is me keeping track of what it's doing and needs to do.

You have to size the unit of work to its useful attention span, you have to have code architecture that is conducive to units of work, and you have to have tools like ticket managers, etc. that keep the big picture and smaller pictures in mind. Reverting from agile practices to more pre-planning whole project documentation and architecture decisions helps.

But in the end it's you. LLMs have their limits and need humans to direct them.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#86
post #56

Earlier quoted context omitted.

What tooling makes this go?

Tests! Unit tests, integration tests, random adhoc scripts. You know - TDD! I’ve been working on UI component improvements and it was doing a lousy job until i specifically told it to test in a headless browser to validate it works. I think somewhere in an AGENTS.md i have an instruction to “don’t state your guesses as fact - validate findings and results”.

It bothers me that you have to explicitly state this to the agent. Makes me think what else is missing from that file which also needs to be explicitly stated, but I don't know what don't know. "Do a good job"?

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#87

Elephant in the room : how many of these bugs were written by LLMs in the first place ? Because creating 100x more bugs and fixing 100x more isn't something to be proud of.

That is a stat you invented out of thin air, based upon nothing but pejorative hope, against all logic or rational consideration of the timeline or the scale of the project, and you declare it the "elephant in the room"?

HN is so weird on the topic of AI, and so many desperately are trying to contrive a reality. Odd stuff.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#88

Earlier quoted context omitted.

[flagged]

Or people don't ask questions for which the answer is known to be "None, really." I get that many don't like what LLMs are doing to the industry, but this is just incorrect reaction to a very specific benefit that's proven beyond doubt (Security hardening). Accept it imo - LLMs are solving very large problems that have plagued software security.

Sorry it is your reaction that is really weird. These are legitimate questions, and I have definitely seen Claude finding the wrong cause and then implementing completely incorrect/irrelevant fixes, only to find that it didn't work and need to start over. Not saying humans don't do the same thing, but LLMs are far from perfect, and it would be delusional to only talk about successes.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#89
post #49
post #41

A lot of people here seem to be living in a different universe than me or simply don't know how to work with AI. I think detractors believe you should just let AI do the job blindly instead of leveraging it as a tool to accelerate you. They get mad at Excel for the poor investment returns. At this point, this is such a strawman, it isn't worth counter arguing. I think I'll abandon this discussion and keep using AI qu…

> A lot of people here seem to be living in a different universe than me I feel this way on this topic too. > I think detractors believe you should just let AI do the job blindly instead of leveraging it as a tool to accelerate you. The problem is; how _should_ I use AI? On a previous thread, I had two replies to the same comment, one saying "provide the LLM all the context it needs and let it go ham", and the other…

I don't generally use AI to write code I feel I can write efficiently. In these cases, I'll just ask AI for a review. But when I need to write code in areas I'm less confident about, I'll let AI take the reins and the do a thorough review myself. These instances are not too common, though.

I also ask direct questions to AI for brainstorming and advanced programming language usage. In these cases, AI isn't touching the code.

Re: Google fixed more Chrome bugs in June than over the past two years, thanks to AI

#90
post #74
post #16

How many of those automated fixes were reverted? How many introduced a new bug? What's the false positive rate on the finding agents? The post has counts for everything that went right and nothing for what could go wrong.

Exactly, also it lacks a lot of context as to why they did that. Possible (probable?) scenario: - Marketing: "we found and fixed lots of bugs thanks to AI" - Reality: the KPI is now to fix as many bugs as possible with the help of AI, so they used AI to search old and easy bugs in the backlog, and then fixed it manually

Huh? Maybe a third scenario is that AI helped fix the bugs? Like described in the actual post you are replying to? This level of conspiracy theory is getting a bit ridiculous.
Post reply on HN