I've recently been using AI a lot for performance optimisation during a particularly busy period at work. I would say it was almost completely useless at the high-level direction - it would point out suspicious parts of SQL queries for example but on back to back testing these almost never resulted in any performance change.
In fact, if it wasn't for the fact that it made making the actual changes I identified much easier (move these joins into a CTE etc) it would have been a detriment. Not only did I get sidetracked by a bunch of useless suggestions but I also had to put up with others dumping their raw AI output at me as if it was somehow a meaningful contribution.
The thing that makes it work really well is to make sure it has all the tooling to verify its hypotheses. If you allow it to run the full lifecycle in loops you will be surprised how well it works.
Built exactly this for work a few months ago. Claude has a read-only account to the dev DB and a bunch of python postgres tools, I point it at queries/views and ask it optimize, and it has to:
1) verify identical query results
2) run repeatedly to get average, worst, best, etc duration of runs
Sped up so many legacy things that none of us were ever going to bother with.
The same tooling that you'd use. AI is not magic, it cannot know exactly how a program will run under your exact circumstances and data. Therefore, if it can use a profiler, test suite or application, or be able to read logs, just like you would, it then can make changes, evaluate them, and iterate. Personally, I've created test MCP servers that will perform actions like submit orders, have the AI capture and analyze PerfViews and Concurrency Visualizer, write out tracing data to logs... all the same stuff I was doing by hand for years.
Tests! Unit tests, integration tests, random adhoc scripts. You know - TDD!
I’ve been working on UI component improvements and it was doing a lousy job until i specifically told it to test in a headless browser to validate it works. I think somewhere in an AGENTS.md i have an instruction to “don’t state your guesses as fact - validate findings and results”.
It bothers me that you have to explicitly state this to the agent. Makes me think what else is missing from that file which also needs to be explicitly stated, but I don't know what don't know. "Do a good job"?
Will, it depends on the AI. Anthropic used to have a lecture-length system prompt for their models to explain this stuff—part of the secret sauce for Claude Code—and famously found that the 5 series models no longer need it.
As usual, if you use anything but the best model available I’m going to state that the better ones do better. If you do use the best model available, then I’ll just mention that Fable still has limits and still needs some guidance.
One thing it does not do is deliberately build tests which test nothing at all, or which restate the code under test. I mention this because certain other models absolutely would.
If the agent can write the tests its supposed to pass and we are worried that agent produces quite a lot of slop (which is why we are doing the tests), then what is the defense against test slop? Test the tests?
Doing a pass where you just ask the AI to sanity-check the existing tests (against rules like “test against the spec, not the implementation” can also help.
You have to learn to use your tools, not try whatever intuitively made sense to you at first (expecting the tool to do all the work) and then whining on Hacker News when it doesn’t work out for you.
This is the research taste part - often they can be good but human experts are really good at this (also you know your codebase, without the prompt these models have no other background).
Then being able to suggest several things to try and have them go off and build, measure and tweak is hugely useful in my experience.
Also things like making custom visualisations for comparing changes.
The mistake there is to point it at code to figure out performance optimizations.
The place to find them would be performance profiles, query plans, telemetry. The guidance for perf still applies, measure before and after change.
The issue is that the code often does not contain the information to do a perf optimization. Eg. you can't tell your cache size, the volumes of data in your DB or the latency of your network through just the text. Should you provide this context, you can get better results.
> The place to find them would be performance profiles, query plans, telemetry.
Tbh, once this information is available (which is the tricky part), in 99% of cases there's really no AI needed to analyze the data, since the 'low hanging fruits' will usually stand out anyway. And once you get into the area where optimization hotspots are no longer obvious, you're already deep in the diminishing returns area and optimizations for one use case or hardware configuration may degrade performance on others. That's my experience anyway.
> there's really no AI needed once this information is available
Most people are not arguing that problems are too tricky for a human to solve once presented with it, but AI can look at 1000 things at the same time across a whole code base and then just come back with the results a human can review.
True, but at least IME, once you get beyond fixing the 'obvious bugs' (how many there are depends on the initial quality of the code base - but I agree that AI is really useful to find those), you'll get into a murky grey area of "maybe false positives" which require a lot of time to validate. As a result you sink a lot of time creating and tweaking code-base-specific rules to try filtering out false positives, and IMHO this is exactly the tipping point where the whole thing becomes pointless because it becomes a bureaucratic monster.
E.g. a good code analysis tool needs to work predictably at button press on any code base.
Still better than nothing of course, e.g. I actually think bug scanning / code analsysis is indeed the one area where LLMs are actually useful, but it suffers from the same 'diminishing returns' problem as traditional approaches, if not worse (e.g. still no silver bullet, but a mostly useful additional tool in the toolbox).
Which model/agent/harness tool did you use? I've found what you describe was my exact experience some ~6-8 months ago, but since about a month or so the game has completely changed. Using 5.6 Sol with highest reasoning setting in Codex or Fable in Code, the models come up with a list of possible improvements from static analysis (ranked by complexity/benefit), write and run their own custom profilers and deliver significant performance improvements with barely any input needed from my side. So this is no longer a model issue, it's a user toolchain issue.
Very hard to say anything definitive on this because it's a moving target, but last time I tried models still had a distinct sense of "consistently good, sometimes great at micro, bad at macro". Similar to how, even for relatively pedestrian CRUD, they'll do code that's objectively fine at the function/file/class level but can still make a mess if you don't supervise them at least at a high-level.
I've had a completely different experience. I usually know the problematic part and then ask the AI to optimize that exact part with benchmarks, telling it to try various different approaches, and it very often gains massive performance increases on various tasks, with me of course steering it and giving it pointers every now and then.
I was able to optimize a physical water simulation that was almost unrunnable on browsers to a buttery smooth 60fps version.
I did a lot of SQL optimisation with claude code over the last year, what made it work for me was making sure I verify everything with explain analyze. The whole loop was not fully “agentic” because of this but it was still faster and better than when done entirely manually. Claude also suggested plenty of small optimisations I would otherwise not do.
Ask it to create a small sample to reproduce the issue in isolation. If it can't reproduce the performance changes then it drops it.
That might work better. I haven't tried it with SQL but it worked with a few complex UI issues I had. It identified the actual issue after a few false starts.
AI can uncover certain issues and resolve certain problems but it also has its blind spots.
I've been using Claude to create a software architecture diagram. It came up with a lot of useful functionality that we had neglected to show but upon further examination a lot of steps didn't make any sense. It added a box to do validation on some data then it put another box downstream to do validation again on the exact same piece of data (the second box had a different name but upon questioning, Claude conceded that it was the exact same validation step). When doing inference on user input, it put a box to extract specific insights from the prompt and then it put another box to do inference again on the inferred data to categorize it... And this would add latency and unnecessarily strip out useful context when doing the second inference... It really only needed a single inference step...
I re-uploaded the diagram to Claude and confronted it and it conceded to all of my points and it even noticed a theme between them and suggested that the diagram exhibited "a pattern of redundant steps." But it was incapable of suggesting viable solutions besides name changes. All of its solutions made the design more complicated. It was incapable of simplifying the design. I've noticed this with AI code; it can make things more complicated, add more features but struggles to simplify things.
People in the comments sound like they're literally asking claude: "hey claude, can you find areas for performance improvements." Nah, it doesn't work like that. You have to be more like: "can you profile [...] and identify hot spots." then looks at results that are suspicious. if you don't ground it to the real world reality of your project then don't be surprised when it makes up its own.
that's the first mistake i made as a new claude user. i only knew what prompts people were sharing on forums. but since they had been written by vibe coders it was all complete non-sense. also think about this: software engineering is filled with highly specialised tooling that can easily improve software quality. the thing is, most developers dont know how to use them, and if they do -- using them is so slow that its almost not worth it.
any tool you can name claude can use. obscure academic proof verifiers, memory corruption scanners, debuggers, profilers, coverage tools, backdoor supply chain pattern scanners... it can run these and have results in seconds. really is like living in the future.
The trend I've noticed is that AI struggles to think outside the box when making optimisations, which exactly what's needed when you've made all of the practical DB and logic optimisations to the existing code.
Often you need to take a step back and question how the system is working and if there would be better ways to design it so the bottlenecks you're hitting wouldn't exist in the first place. Caching things, adding indexes, tweaking logic – these can help, but you'll quickly hit diminishing returns once you've done all of the obvious stuff.
I've seen people here say how AI is great at optimising code though, but I'm not sure if that's because they're giving it optimisation problems with a lot of low hanging fruit or if they're successfully getting AI to rework their systems to remove bottlenecks. This one area I find AI to still be particularly bad at.
Then you're not grounding it to reality properly. LLMs get bad pretty quickly when you just tell them to pull the answer out of thin air. Pin them to reality with actual performance tests to run to test theories and the results will be much better.
Performance optimization involves simulating real world loads and measurable results. Give them the opportunity to actually have a closed loop if you want more than trivial improvements.
Have you any example materials that show a harness that an agent can be pointed at an app with some sort of telemetry tool, the results it gained and the cost of doing so? Because my experience is the same as the parents - the LLM goes on massive tangents, and the more tangents it goes on the worse the results get.
Ultimately the harness is me and the experience is like managing an unruly toddler. You have to pay attention to what it does and issue corrections. Skills and prompts and AGENTS.md do some, nested sets of agents do some, but over it all is me keeping track of what it's doing and needs to do.
You have to size the unit of work to its useful attention span, you have to have code architecture that is conducive to units of work, and you have to have tools like ticket managers, etc. that keep the big picture and smaller pictures in mind. Reverting from agile practices to more pre-planning whole project documentation and architecture decisions helps.
But in the end it's you. LLMs have their limits and need humans to direct them.
As others have said, these were uninformed guesses which LLMs seem still to be willing to hand out deliberately. If you provide more information (your schemas, the distribution of values in your database tables, the EXPLAIN ANALYZE outputs, DB configuration etc.) it will do a much better job.
Yep, as a performance specialist I've had the same outcomes - it'll find "smoking guns" galore which will sound plausible and be completely wrong.
But that's ok - I still get a lot of value of the tool elsewhere (e.g. writing data analysis scripts, exploring code, breaking down bug artifacts...), but it doesn't stop the tiring BS from other people going "why not just AI it?" (especially from managers).
I usually let them try to "just AI it" and most of them learn quickly that it's not that simple. (The others stay deranged and are making my job miserable.)
That's uninteresting as long as you don't specify the model you used. For example, Mythos was far better at finding security bugs than previous models.
A lot of people here seem to be living in a different universe than me or simply don't know how to work with AI. I think detractors believe you should just let AI do the job blindly instead of leveraging it as a tool to accelerate you. They get mad at Excel for the poor investment returns. At this point, this is such a strawman, it isn't worth counter arguing.
I think I'll abandon this discussion and keep using AI quietly while exchanging tips with like-minded people who are interested in using it properly and efficiently.
> A lot of people here seem to be living in a different universe than me
I feel this way on this topic too.
> I think detractors believe you should just let AI do the job blindly instead of leveraging it as a tool to accelerate you.
The problem is; how _should_ I use AI? On a previous thread, I had two replies to the same comment, one saying "provide the LLM all the context it needs and let it go ham", and the other saying "Carefully guide and craft it and review everything". Both subthreads had people agreeing.
My experience with LLMs is leaving them unattended gives very poor results within a handful of iterations, and that carefully guiding them can be a value add, but the effort ot do that is very often as much as just writing the damn code myself, particularly if there's a few iterations that go on.
What can be concluded is that some get immense value out of Ai and some are incredibly critical of that claim.
I do not know what the secret sauce is.
But I have a responsibility to the team I lead. I need to protect our velocity. Which means that my hiring practices now include specific checks that people are able to work with ai.
In the end, it might be a step thing. Some people are just inherently incapable of working with this technology...
I don't generally use AI to write code I feel I can write efficiently. In these cases, I'll just ask AI for a review. But when I need to write code in areas I'm less confident about, I'll let AI take the reins and the do a thorough review myself. These instances are not too common, though.
I also ask direct questions to AI for brainstorming and advanced programming language usage. In these cases, AI isn't touching the code.
The reality of the current situation is that we have dozens of families of different model; each of which is differentiated further into various revision numbers and power levels, and depending on which one you use and what the nature of the tasks is, you very much need a different usage style that can only be discerned after multiple (Sometimes very many) sessions of using that individual model for that category of task. It is not one size fits all, and trying to pretend that it is will lead to failure.
Did you think for a second that maybe other people work with other languages, domains, tools and work approaches than you?
It's so bizarre that noone realizes that there's a bit of a difference between banging together JavaScript into websites, writing firmware for widgets, herding servers or fixing medical devices?
Not that I don't believe its possible to fix a lot of bugs, I also wonder what the actual dynamic was. Were the people in team working much more than usual as well? Given its Google, I wouldn't be surprised if there was an "internal push" to fix more bugs over next X sprints so that they can publish this blog and some manager can show impact and AI adaption to his superior.
Google's entire modus operandi has been "automate everything" for decades. They've been doing this with fuzzers, with project zero and so on. Adding LLMs on top is a very obvious next step. And LLMs improving and finding more bugs also follows. Then improve the harness and the dev tools, to better use the LLMs. And then everything together end-to-end to find-triage-fix-confirm. Your LLMs are as good as the loop they run in, and the loop is as good as the verifier. Seems a reasonable enough dynamic without (or despite, depends how cynical you wanna be) the need for managers to show number goes up on some chart.
I would guess they have been confronted with an initial flood of newly discovered bugs whenever they added a new analysis tool or approach (e.g. automated static analysis, fuzzing, ...). Once working through that initial flurry of newly discovered bugs one would assume that the frequency goes down again.
1. Our backlog of bugs gets processed quicker because instead of staring at the code for 10 minutes fiuring out what's happening, there's a tool that can reason about it quicker.
2. Code reviews and security reviews happen quicker and produce more findings.
I would think that (m)any team(s) using AI might also be seeing a higher rate of finding and fixing issues.
Even the Linux Kernel (I'd say Windows and Apple too) are seeing the same phenomenon.
How many of those automated fixes were reverted? How many introduced a new bug? What's the false positive rate on the finding agents? The post has counts for everything that went right and nothing for what could go wrong.
Exactly, also it lacks a lot of context as to why they did that.
Possible (probable?) scenario:
- Marketing: "we found and fixed lots of bugs thanks to AI"
- Reality: the KPI is now to fix as many bugs as possible with the help of AI, so they used AI to search old and easy bugs in the backlog, and then fixed it manually
Huh? Maybe a third scenario is that AI helped fix the bugs? Like described in the actual post you are replying to? This level of conspiracy theory is getting a bit ridiculous.
I think you’re right, my base assumption is that the models can code and can fix bugs, and can code more in parallel and faster than humans at a lower cost.
If Google are tackling lower value bugs with AI the number is in a way inflated compared to some utility measure (fixing a smaller number of worse bugs could be preferable) but it’s still things fixed.
At my company there's a lot of discussion about AI and complaints that people run out of tokens within a day, but zero results are shown. No measurable (or measured) gains. Or nothing that people are willing to talk about, in any case.
Or people don't ask questions for which the answer is known to be "None, really."
I get that many don't like what LLMs are doing to the industry, but this is just incorrect reaction to a very specific benefit that's proven beyond doubt (Security hardening).
Accept it imo - LLMs are solving very large problems that have plagued software security.
Sorry it is your reaction that is really weird. These are legitimate questions, and I have definitely seen Claude finding the wrong cause and then implementing completely incorrect/irrelevant fixes, only to find that it didn't work and need to start over. Not saying humans don't do the same thing, but LLMs are far from perfect, and it would be delusional to only talk about successes.
I've done security work before, and I've done it now with frontier. I've seen the difference first hand, and as the OP and so many other articles show, so have the leading experts in the world.
So either they are lying, or you may not yet be seeing and experiencing what they are. If that's wierd, ok.
AI critique often funnels itself into a narrow bucket: creating code blindly with AI is bad. That's easy to grant.
Adversarial testing, checking developer assumptions, refactor suggestions, small dev tools and even some guided coding all sit on the other side of the spectrum of what you can do with coding and AI. For larger and larger codebases even simple things like tracing dependencies or behavior might be greatly aided.
And the critiques reserved for that narrow bucket on the other end, blindly generating code, are too easily conflated with the rest.
Yes. AI is a tool, it’s supposed to be used a certain way, anything else is a misunderstanding og what AI is. You have to aim it in the direction you want it to go, not expect it to solve all your problems magically
LLMs are increasingly not tools anymore but agents. The difference is still one of degree, but it's clear in which direction we are currently moving. It was even clear to some people 10 years ago: https://gwern.net/tool-ai
I am actually using AI to learn about Windows OS performance. I have a couple of prompts scheduled to research, test, and implement performance gains in the OS, then post them here [0] and here [1]. I always admired Brendan Greg and Bryan Cantrill at Sun/Illumos/ for their work on OS performance but never had the time to sit down and catch up to their level due to my time in the military.
I got Codex to find and apply a bunch of settings I didn't know about and in the future I hope to use AI to make real changes in performance that venders probably don't have time or resources for.
that being said, what I've learned so far is essentially
WhyIsItAlwaysHN's entire comment "The place to find them would be performance profiles, query plans, telemetry."
I just find it so frustrating that for a user interface that was basically solved in the 90s, Windows GUI still struggles to survive on hardware exceptionally more powerful than what we had 30 years ago
Can't have been much, as the Chrome project is 20+ years old and LLMs have only recently become a thing; they did not slacken their code review and testing practices with the advent of LLM code generators. And I doubt they do much development on the affected areas at the moment, they mention a 13 year old issue for example.
But it's an open source project, you can go and figure out whether your assertion is correct.
If people are handing off the coding to AI more and more then they'll be less skilled at spotting potential bugs in the first place. AI, course of, could be run to find potential bugs in the AI-coded features before commit. But then you're getting further and further towards a world where code, and code that runs infrastructure, is incomprehensible. We'll have to start sacrificing the first-born to the silicon in the hopes of a good build for code that controls your electrical sub-station.
That is a stat you invented out of thin air, based upon nothing but pejorative hope, against all logic or rational consideration of the timeline or the scale of the project, and you declare it the "elephant in the room"?
HN is so weird on the topic of AI, and so many desperately are trying to contrive a reality. Odd stuff.
Maybe they could ask the AI to fix their atrocious build times. Only in Chrome do you have 300 line source files that blow up to 20 megabytes after preprocessing as a matter of course. 3 GB per compile job is getting more common - and RAM is expensive right now!
I wish they'd spent more time on improving the UI now. It's largely been the same since Chrome appeared. Their recent vertical tabs are so much inferior to Dia, which is built by a much smaller team.
That's something those products need to figure out, not the users, one potential solution is to erect paywalls and find out how much those products are actually worth to their users ;)
Good. Now please fix youtube. Amount of bugs that I'm getting in youtube app on iphone should be embarassing for company like Google. I cannot believe how bad it become in last couple years.
give it another year and AI will be filing bugs faster than humans can merge the fixes, and the Chrome team will need a separate AI just to triage the first AI's bug reports
I've mostly only used Codex for reviews, for Godot/GDScript code, and it helped me catch bugs that would've taken me ages to even notice on my own; games be tricky like that.
Claude was mostly useless or annoying up until the last time I tried it (about 3 months ago)
You have to be vigilant though: Codex often picks out obscure edge cases and suggests adding multiple new functions and flags to avoid issues that would be better off left as fast-failure crashes. Maybe that's because of the 5.6 Sol Max I leave it on.
The biggest bug still has not been fixed here, which is:
Google.
We really need an alternative to this greedy and evil corporations
de-facto controlling a huge portion of the modern www stack. All
decision-making processes are here subjugated to what fits Google's
adEmpire. This is a perpetual system of control amplifying abusive
systems in place. We already see this with the recent age sniffing;
Google just yesterday announced that all android users must hand
over their age to everyone else. Before that google kicked out
or locked out open source alternatives to android (or, at the least,
made their life significantly harder than before). These are not
isolated ways of abuse - this is systematic abuse.
We really need alternatives to Google here. It can not be that one
tyrant company dictates so much of people's open lives (and no,
Firefox is no alternative; Mozilla is basically a nerfed and bribed
entity that has given up on firefox many years ago already; ladybird
may become an alternative at one point in time, but right now is not
and there are questions over how Kling handles the overall project,
but these complaints are significantly below what Google is doing
globally here).
Fixing more bugs in the adChromium code base does not change the underlying problem at hand.
Seems like they have enough AI they could find a way to put ManifestV2 back into the browser and fix things we actually wanted instead of putting more bullshit "AI improvements" we didn't?
A company valued at $4+ trillion based on the recent AI hype alone is touting the benefits of using said AI, and we're supposed to take this shit for granted.
I'm wondering what the Alphabet employees still commenting on this forum have to say about it? I guess for the right comps they can keep their mouths shut no matter the high level of idiocy involved.
I don't know about Chromium specifically, but in general Google requires 2 humans to have looked at every change (normally one is the author, but not necessarily for bot-authored changes).
I’ve heard from people inside Google that they’re prohibited from using anything but their own models. AlbertaTech on YT, ex-Google working on YT said in a video that right before she quit someone had threatened to quit if they couldn’t get access to Claude Code
Kind of, but you can also assert it yourself; the big tech companies have billions of lines of code not publicly available, they can train their in-house code assistants on those instead of or in addition to what's available out in the open.
I've recently been using AI a lot for performance optimisation during a particularly busy period at work. I would say it was almost completely useless at the high-level direction - it would point out suspicious parts of SQL queries for example but on back to back testing these almost never resulted in any performance change.
In fact, if it wasn't for the fact that it made making the actual changes I identified much easier (move these joins into a CTE etc) it would have been a detriment. Not only did I get sidetracked by a bunch of useless suggestions but I also had to put up with others dumping their raw AI output at me as if it was somehow a meaningful contribution.
The thing that makes it work really well is to make sure it has all the tooling to verify its hypotheses. If you allow it to run the full lifecycle in loops you will be surprised how well it works.
Built exactly this for work a few months ago. Claude has a read-only account to the dev DB and a bunch of python postgres tools, I point it at queries/views and ask it optimize, and it has to:
1) verify identical query results
2) run repeatedly to get average, worst, best, etc duration of runs
Sped up so many legacy things that none of us were ever going to bother with.
But them it's more like AI assisted performance fuzzing? Just throw stuff against some measuring harness and see what sticks?
Exactly. In the above case, AI should be able to verify it's own hypothesis by having access to the database's table statistics and planner.
What tooling makes this go?
The same tooling that you'd use. AI is not magic, it cannot know exactly how a program will run under your exact circumstances and data. Therefore, if it can use a profiler, test suite or application, or be able to read logs, just like you would, it then can make changes, evaluate them, and iterate. Personally, I've created test MCP servers that will perform actions like submit orders, have the AI capture and analyze PerfViews and Concurrency Visualizer, write out tracing data to logs... all the same stuff I was doing by hand for years.
Tests! Unit tests, integration tests, random adhoc scripts. You know - TDD!
I’ve been working on UI component improvements and it was doing a lousy job until i specifically told it to test in a headless browser to validate it works. I think somewhere in an AGENTS.md i have an instruction to “don’t state your guesses as fact - validate findings and results”.
It bothers me that you have to explicitly state this to the agent. Makes me think what else is missing from that file which also needs to be explicitly stated, but I don't know what don't know. "Do a good job"?
Will, it depends on the AI. Anthropic used to have a lecture-length system prompt for their models to explain this stuff—part of the secret sauce for Claude Code—and famously found that the 5 series models no longer need it.
As usual, if you use anything but the best model available I’m going to state that the better ones do better. If you do use the best model available, then I’ll just mention that Fable still has limits and still needs some guidance.
One thing it does not do is deliberately build tests which test nothing at all, or which restate the code under test. I mention this because certain other models absolutely would.
In this particular case tests won't help you, without the actual data. But overall you're right.
If the agent can write the tests its supposed to pass and we are worried that agent produces quite a lot of slop (which is why we are doing the tests), then what is the defense against test slop? Test the tests?
Manual testing.
I work on user facing applications, and since the models do not have good taste, testing the UX is essential.
If you spot a bug, usually the model will attempt to reproduce it in a new test case that does cover the actual issue.
Read the tests.
Doing a pass where you just ask the AI to sanity-check the existing tests (against rules like “test against the spec, not the implementation” can also help.
Existing tests.
The ability to run queries and get the metadata about the run, e.g. length of run, execution plan, engine, engine params, etc
You have to learn to use your tools, not try whatever intuitively made sense to you at first (expecting the tool to do all the work) and then whining on Hacker News when it doesn’t work out for you.
AI can not fail. It can only be failed. AI can not have limitations. Only the user can be limited.
This part is obviously their fault and they need to use their tools better:
> others dumping their raw AI output at me
Sheesh.
This is the research taste part - often they can be good but human experts are really good at this (also you know your codebase, without the prompt these models have no other background).
Then being able to suggest several things to try and have them go off and build, measure and tweak is hugely useful in my experience.
Also things like making custom visualisations for comparing changes.
The mistake there is to point it at code to figure out performance optimizations.
The place to find them would be performance profiles, query plans, telemetry. The guidance for perf still applies, measure before and after change.
The issue is that the code often does not contain the information to do a perf optimization. Eg. you can't tell your cache size, the volumes of data in your DB or the latency of your network through just the text. Should you provide this context, you can get better results.
> The place to find them would be performance profiles, query plans, telemetry.
Tbh, once this information is available (which is the tricky part), in 99% of cases there's really no AI needed to analyze the data, since the 'low hanging fruits' will usually stand out anyway. And once you get into the area where optimization hotspots are no longer obvious, you're already deep in the diminishing returns area and optimizations for one use case or hardware configuration may degrade performance on others. That's my experience anyway.
> there's really no AI needed once this information is available
Most people are not arguing that problems are too tricky for a human to solve once presented with it, but AI can look at 1000 things at the same time across a whole code base and then just come back with the results a human can review.
True, but at least IME, once you get beyond fixing the 'obvious bugs' (how many there are depends on the initial quality of the code base - but I agree that AI is really useful to find those), you'll get into a murky grey area of "maybe false positives" which require a lot of time to validate. As a result you sink a lot of time creating and tweaking code-base-specific rules to try filtering out false positives, and IMHO this is exactly the tipping point where the whole thing becomes pointless because it becomes a bureaucratic monster.
E.g. a good code analysis tool needs to work predictably at button press on any code base.
Still better than nothing of course, e.g. I actually think bug scanning / code analsysis is indeed the one area where LLMs are actually useful, but it suffers from the same 'diminishing returns' problem as traditional approaches, if not worse (e.g. still no silver bullet, but a mostly useful additional tool in the toolbox).
Which model/agent/harness tool did you use? I've found what you describe was my exact experience some ~6-8 months ago, but since about a month or so the game has completely changed. Using 5.6 Sol with highest reasoning setting in Codex or Fable in Code, the models come up with a list of possible improvements from static analysis (ranked by complexity/benefit), write and run their own custom profilers and deliver significant performance improvements with barely any input needed from my side. So this is no longer a model issue, it's a user toolchain issue.
Very hard to say anything definitive on this because it's a moving target, but last time I tried models still had a distinct sense of "consistently good, sometimes great at micro, bad at macro". Similar to how, even for relatively pedestrian CRUD, they'll do code that's objectively fine at the function/file/class level but can still make a mess if you don't supervise them at least at a high-level.
I've had a completely different experience. I usually know the problematic part and then ask the AI to optimize that exact part with benchmarks, telling it to try various different approaches, and it very often gains massive performance increases on various tasks, with me of course steering it and giving it pointers every now and then.
I was able to optimize a physical water simulation that was almost unrunnable on browsers to a buttery smooth 60fps version.
I did a lot of SQL optimisation with claude code over the last year, what made it work for me was making sure I verify everything with explain analyze. The whole loop was not fully “agentic” because of this but it was still faster and better than when done entirely manually. Claude also suggested plenty of small optimisations I would otherwise not do.
Ask it to create a small sample to reproduce the issue in isolation. If it can't reproduce the performance changes then it drops it.
That might work better. I haven't tried it with SQL but it worked with a few complex UI issues I had. It identified the actual issue after a few false starts.
AI can uncover certain issues and resolve certain problems but it also has its blind spots.
I've been using Claude to create a software architecture diagram. It came up with a lot of useful functionality that we had neglected to show but upon further examination a lot of steps didn't make any sense. It added a box to do validation on some data then it put another box downstream to do validation again on the exact same piece of data (the second box had a different name but upon questioning, Claude conceded that it was the exact same validation step). When doing inference on user input, it put a box to extract specific insights from the prompt and then it put another box to do inference again on the inferred data to categorize it... And this would add latency and unnecessarily strip out useful context when doing the second inference... It really only needed a single inference step...
I re-uploaded the diagram to Claude and confronted it and it conceded to all of my points and it even noticed a theme between them and suggested that the diagram exhibited "a pattern of redundant steps." But it was incapable of suggesting viable solutions besides name changes. All of its solutions made the design more complicated. It was incapable of simplifying the design. I've noticed this with AI code; it can make things more complicated, add more features but struggles to simplify things.
People in the comments sound like they're literally asking claude: "hey claude, can you find areas for performance improvements." Nah, it doesn't work like that. You have to be more like: "can you profile [...] and identify hot spots." then looks at results that are suspicious. if you don't ground it to the real world reality of your project then don't be surprised when it makes up its own.
that's the first mistake i made as a new claude user. i only knew what prompts people were sharing on forums. but since they had been written by vibe coders it was all complete non-sense. also think about this: software engineering is filled with highly specialised tooling that can easily improve software quality. the thing is, most developers dont know how to use them, and if they do -- using them is so slow that its almost not worth it.
any tool you can name claude can use. obscure academic proof verifiers, memory corruption scanners, debuggers, profilers, coverage tools, backdoor supply chain pattern scanners... it can run these and have results in seconds. really is like living in the future.
I've noticed the same thing.
The trend I've noticed is that AI struggles to think outside the box when making optimisations, which exactly what's needed when you've made all of the practical DB and logic optimisations to the existing code.
Often you need to take a step back and question how the system is working and if there would be better ways to design it so the bottlenecks you're hitting wouldn't exist in the first place. Caching things, adding indexes, tweaking logic – these can help, but you'll quickly hit diminishing returns once you've done all of the obvious stuff.
I've seen people here say how AI is great at optimising code though, but I'm not sure if that's because they're giving it optimisation problems with a lot of low hanging fruit or if they're successfully getting AI to rework their systems to remove bottlenecks. This one area I find AI to still be particularly bad at.
Then you're not grounding it to reality properly. LLMs get bad pretty quickly when you just tell them to pull the answer out of thin air. Pin them to reality with actual performance tests to run to test theories and the results will be much better.
Performance optimization involves simulating real world loads and measurable results. Give them the opportunity to actually have a closed loop if you want more than trivial improvements.
Have you any example materials that show a harness that an agent can be pointed at an app with some sort of telemetry tool, the results it gained and the cost of doing so? Because my experience is the same as the parents - the LLM goes on massive tangents, and the more tangents it goes on the worse the results get.
Ultimately the harness is me and the experience is like managing an unruly toddler. You have to pay attention to what it does and issue corrections. Skills and prompts and AGENTS.md do some, nested sets of agents do some, but over it all is me keeping track of what it's doing and needs to do.
You have to size the unit of work to its useful attention span, you have to have code architecture that is conducive to units of work, and you have to have tools like ticket managers, etc. that keep the big picture and smaller pictures in mind. Reverting from agile practices to more pre-planning whole project documentation and architecture decisions helps.
But in the end it's you. LLMs have their limits and need humans to direct them.
As others have said, these were uninformed guesses which LLMs seem still to be willing to hand out deliberately. If you provide more information (your schemas, the distribution of values in your database tables, the EXPLAIN ANALYZE outputs, DB configuration etc.) it will do a much better job.
Yep, as a performance specialist I've had the same outcomes - it'll find "smoking guns" galore which will sound plausible and be completely wrong.
But that's ok - I still get a lot of value of the tool elsewhere (e.g. writing data analysis scripts, exploring code, breaking down bug artifacts...), but it doesn't stop the tiring BS from other people going "why not just AI it?" (especially from managers).
I usually let them try to "just AI it" and most of them learn quickly that it's not that simple. (The others stay deranged and are making my job miserable.)
> I've recently been using AI
That's uninteresting as long as you don't specify the model you used. For example, Mythos was far better at finding security bugs than previous models.
Do you really think they mean they’ve been using mythos
A lot of people here seem to be living in a different universe than me or simply don't know how to work with AI. I think detractors believe you should just let AI do the job blindly instead of leveraging it as a tool to accelerate you. They get mad at Excel for the poor investment returns. At this point, this is such a strawman, it isn't worth counter arguing.
I think I'll abandon this discussion and keep using AI quietly while exchanging tips with like-minded people who are interested in using it properly and efficiently.
> A lot of people here seem to be living in a different universe than me
I feel this way on this topic too.
> I think detractors believe you should just let AI do the job blindly instead of leveraging it as a tool to accelerate you.
The problem is; how _should_ I use AI? On a previous thread, I had two replies to the same comment, one saying "provide the LLM all the context it needs and let it go ham", and the other saying "Carefully guide and craft it and review everything". Both subthreads had people agreeing.
My experience with LLMs is leaving them unattended gives very poor results within a handful of iterations, and that carefully guiding them can be a value add, but the effort ot do that is very often as much as just writing the damn code myself, particularly if there's a few iterations that go on.
What can be concluded is that some get immense value out of Ai and some are incredibly critical of that claim.
I do not know what the secret sauce is.
But I have a responsibility to the team I lead. I need to protect our velocity. Which means that my hiring practices now include specific checks that people are able to work with ai.
In the end, it might be a step thing. Some people are just inherently incapable of working with this technology...
I don't generally use AI to write code I feel I can write efficiently. In these cases, I'll just ask AI for a review. But when I need to write code in areas I'm less confident about, I'll let AI take the reins and the do a thorough review myself. These instances are not too common, though.
I also ask direct questions to AI for brainstorming and advanced programming language usage. In these cases, AI isn't touching the code.
The reality of the current situation is that we have dozens of families of different model; each of which is differentiated further into various revision numbers and power levels, and depending on which one you use and what the nature of the tasks is, you very much need a different usage style that can only be discerned after multiple (Sometimes very many) sessions of using that individual model for that category of task. It is not one size fits all, and trying to pretend that it is will lead to failure.
Did you think for a second that maybe other people work with other languages, domains, tools and work approaches than you?
It's so bizarre that noone realizes that there's a bit of a difference between banging together JavaScript into websites, writing firmware for widgets, herding servers or fixing medical devices?
Not that I don't believe its possible to fix a lot of bugs, I also wonder what the actual dynamic was. Were the people in team working much more than usual as well? Given its Google, I wouldn't be surprised if there was an "internal push" to fix more bugs over next X sprints so that they can publish this blog and some manager can show impact and AI adaption to his superior.
Google's entire modus operandi has been "automate everything" for decades. They've been doing this with fuzzers, with project zero and so on. Adding LLMs on top is a very obvious next step. And LLMs improving and finding more bugs also follows. Then improve the harness and the dev tools, to better use the LLMs. And then everything together end-to-end to find-triage-fix-confirm. Your LLMs are as good as the loop they run in, and the loop is as good as the verifier. Seems a reasonable enough dynamic without (or despite, depends how cynical you wanna be) the need for managers to show number goes up on some chart.
I would guess they have been confronted with an initial flood of newly discovered bugs whenever they added a new analysis tool or approach (e.g. automated static analysis, fuzzing, ...). Once working through that initial flurry of newly discovered bugs one would assume that the frequency goes down again.
More likely just getting ahead of the AI attacks before they hit. The threat risk increase caused by AI has gone off the chart.
1. Our backlog of bugs gets processed quicker because instead of staring at the code for 10 minutes fiuring out what's happening, there's a tool that can reason about it quicker.
2. Code reviews and security reviews happen quicker and produce more findings.
I would think that (m)any team(s) using AI might also be seeing a higher rate of finding and fixing issues.
Even the Linux Kernel (I'd say Windows and Apple too) are seeing the same phenomenon.
How many of those automated fixes were reverted? How many introduced a new bug? What's the false positive rate on the finding agents? The post has counts for everything that went right and nothing for what could go wrong.
Exactly, also it lacks a lot of context as to why they did that.
Possible (probable?) scenario:
- Marketing: "we found and fixed lots of bugs thanks to AI"
- Reality: the KPI is now to fix as many bugs as possible with the help of AI, so they used AI to search old and easy bugs in the backlog, and then fixed it manually
Huh? Maybe a third scenario is that AI helped fix the bugs? Like described in the actual post you are replying to? This level of conspiracy theory is getting a bit ridiculous.
I think you’re right, my base assumption is that the models can code and can fix bugs, and can code more in parallel and faster than humans at a lower cost.
If Google are tackling lower value bugs with AI the number is in a way inflated compared to some utility measure (fixing a smaller number of worse bugs could be preferable) but it’s still things fixed.
I assumed it was really just AI that found them, which allowed Google to know to fix them.
> The post has counts for everything that went right and nothing for what could go wrong.
That's AI for you.
At Amazon we have many forums to share our AI wins, but none to share AI failures or disappoinments.
No wonder execs make bad decisions regarding AI, they only hear completely one-sided stories.
At my company there's a lot of discussion about AI and complaints that people run out of tokens within a day, but zero results are shown. No measurable (or measured) gains. Or nothing that people are willing to talk about, in any case.
"It can't be that stupid—you must be prompting it wrong." - Ed Zitron
And how many were introduced by AI?
It's the opposite: humans. The most critical one (sandbox escape) has been sitting there for 13 years.
You're not supposed to ask sane questions here
Or people don't ask questions for which the answer is known to be "None, really."
I get that many don't like what LLMs are doing to the industry, but this is just incorrect reaction to a very specific benefit that's proven beyond doubt (Security hardening).
Accept it imo - LLMs are solving very large problems that have plagued software security.
Sorry it is your reaction that is really weird. These are legitimate questions, and I have definitely seen Claude finding the wrong cause and then implementing completely incorrect/irrelevant fixes, only to find that it didn't work and need to start over. Not saying humans don't do the same thing, but LLMs are far from perfect, and it would be delusional to only talk about successes.
I've done security work before, and I've done it now with frontier. I've seen the difference first hand, and as the OP and so many other articles show, so have the leading experts in the world.
So either they are lying, or you may not yet be seeing and experiencing what they are. If that's wierd, ok.
I’d rephrase this to something like: “People fixed more Chrome bugs (…), thanks to people who leveraged AI tools”
AI critique often funnels itself into a narrow bucket: creating code blindly with AI is bad. That's easy to grant.
Adversarial testing, checking developer assumptions, refactor suggestions, small dev tools and even some guided coding all sit on the other side of the spectrum of what you can do with coding and AI. For larger and larger codebases even simple things like tracing dependencies or behavior might be greatly aided.
And the critiques reserved for that narrow bucket on the other end, blindly generating code, are too easily conflated with the rest.
Yes. AI is a tool, it’s supposed to be used a certain way, anything else is a misunderstanding og what AI is. You have to aim it in the direction you want it to go, not expect it to solve all your problems magically
LLMs are increasingly not tools anymore but agents. The difference is still one of degree, but it's clear in which direction we are currently moving. It was even clear to some people 10 years ago: https://gwern.net/tool-ai
AI has skills no individual can have. That part is true.
But AI is not smarter than you, AI is as smart as the user who uses it. It often makes wrong decisions unless user corrects it.
I assume this was thanks to Project Glasswing + Mythos Preview?
This is only a flex if AI also didn’t cause an increase in bugs that needed to be fixed.
Convenient that the LLM vendor is the one saying their own product works, isn’t it?
Company advertises their products. In other news, water is wet
The next step is to build software without bugs
I am actually using AI to learn about Windows OS performance. I have a couple of prompts scheduled to research, test, and implement performance gains in the OS, then post them here [0] and here [1]. I always admired Brendan Greg and Bryan Cantrill at Sun/Illumos/ for their work on OS performance but never had the time to sit down and catch up to their level due to my time in the military.
I got Codex to find and apply a bunch of settings I didn't know about and in the future I hope to use AI to make real changes in performance that venders probably don't have time or resources for.
that being said, what I've learned so far is essentially WhyIsItAlwaysHN's entire comment "The place to find them would be performance profiles, query plans, telemetry."
I just find it so frustrating that for a user interface that was basically solved in the 90s, Windows GUI still struggles to survive on hardware exceptionally more powerful than what we had 30 years ago
0 - https://www.lacksan.com/updates/ 1 - https://github.com/Lacksan-Dev/HP-ZBook-Performance
Elephant in the room : how many of these bugs were written by LLMs in the first place ?
Because creating 100x more bugs and fixing 100x more isn't something to be proud of.
Can't have been much, as the Chrome project is 20+ years old and LLMs have only recently become a thing; they did not slacken their code review and testing practices with the advent of LLM code generators. And I doubt they do much development on the affected areas at the moment, they mention a 13 year old issue for example.
But it's an open source project, you can go and figure out whether your assertion is correct.
The single 13yo issue is anecdata.
They obviously have the full git blame statistics but chose not to include them in the blog post, which is a bit concerning.
If people are handing off the coding to AI more and more then they'll be less skilled at spotting potential bugs in the first place. AI, course of, could be run to find potential bugs in the AI-coded features before commit. But then you're getting further and further towards a world where code, and code that runs infrastructure, is incomprehensible. We'll have to start sacrificing the first-born to the silicon in the hopes of a good build for code that controls your electrical sub-station.
That is a stat you invented out of thin air, based upon nothing but pejorative hope, against all logic or rational consideration of the timeline or the scale of the project, and you declare it the "elephant in the room"?
HN is so weird on the topic of AI, and so many desperately are trying to contrive a reality. Odd stuff.
It's not an invention it's a question. If the number is <20% that's great news for the Chrome team.
Sad they didn't say `Thanks to Gemini`
Concerning even
Nice marketing Google, now tell us how many of the bugs that were fixed were actually introduced by AI usage to begin with?
Maybe they could ask the AI to fix their atrocious build times. Only in Chrome do you have 300 line source files that blow up to 20 megabytes after preprocessing as a matter of course. 3 GB per compile job is getting more common - and RAM is expensive right now!
I wish they'd spent more time on improving the UI now. It's largely been the same since Chrome appeared. Their recent vertical tabs are so much inferior to Dia, which is built by a much smaller team.
a lot of these bugs wouldn't have ever been found without AI
Still can't stand Chrome.
won't forgive them either for making it harder to filter out ads
. .
Why would anybody want reddit and YouTube of all pages to be funded?
That's my point.
That's something those products need to figure out, not the users, one potential solution is to erect paywalls and find out how much those products are actually worth to their users ;)
We don't.
You see, grandma is supposed to see the ads and buy the Mighty Putty. I'm supposed to get the adblock version. That's just how the world works.
So will the Chromium bug 5569 finally get fixed?
I'm not sure why you're asking here instead of opening the ticket and having a look at the current status.
But how many new ones did it introduce?
Good. Now please fix youtube. Amount of bugs that I'm getting in youtube app on iphone should be embarassing for company like Google. I cannot believe how bad it become in last couple years.
More important thing:
Do you import new more bugs?
Real question is: how much those "fixed" bugs actually bothered the users to begin with?
But since Google cares mostly about its investors, the numbers and a mention of Gemini in such a blog post are more important.
The whole point is that these are security bugs that didn't necessarily bother people so far but could be exploited
give it another year and AI will be filing bugs faster than humans can merge the fixes, and the Chrome team will need a separate AI just to triage the first AI's bug reports
Me too! It's actually tiring! In a good way
I've mostly only used Codex for reviews, for Godot/GDScript code, and it helped me catch bugs that would've taken me ages to even notice on my own; games be tricky like that.
Claude was mostly useless or annoying up until the last time I tried it (about 3 months ago)
You have to be vigilant though: Codex often picks out obscure edge cases and suggests adding multiple new functions and flags to avoid issues that would be better off left as fast-failure crashes. Maybe that's because of the 5.6 Sol Max I leave it on.
The biggest bug still has not been fixed here, which is:
Google.
We really need an alternative to this greedy and evil corporations de-facto controlling a huge portion of the modern www stack. All decision-making processes are here subjugated to what fits Google's adEmpire. This is a perpetual system of control amplifying abusive systems in place. We already see this with the recent age sniffing; Google just yesterday announced that all android users must hand over their age to everyone else. Before that google kicked out or locked out open source alternatives to android (or, at the least, made their life significantly harder than before). These are not isolated ways of abuse - this is systematic abuse.
We really need alternatives to Google here. It can not be that one tyrant company dictates so much of people's open lives (and no, Firefox is no alternative; Mozilla is basically a nerfed and bribed entity that has given up on firefox many years ago already; ladybird may become an alternative at one point in time, but right now is not and there are questions over how Kling handles the overall project, but these complaints are significantly below what Google is doing globally here).
Fixing more bugs in the adChromium code base does not change the underlying problem at hand.
> We really need an alternative to this greedy and evil corporations de-facto controlling a huge portion of the modern www stack.
We do / have? Multiple even. Of course, the vast majority of consumers doesn't care so they don't get as much traction.
Maybe fix all the decade-long old bugs first.
Big AI player promotes AI through blog to make AI look better woo
Seems like they have enough AI they could find a way to put ManifestV2 back into the browser and fix things we actually wanted instead of putting more bullshit "AI improvements" we didn't?
If only there was alternative browsers...
A company valued at $4+ trillion based on the recent AI hype alone is touting the benefits of using said AI, and we're supposed to take this shit for granted.
I'm wondering what the Alphabet employees still commenting on this forum have to say about it? I guess for the right comps they can keep their mouths shut no matter the high level of idiocy involved.
Did anyone check the code changes manually? What if it is just slop code?
I don't know about Chromium specifically, but in general Google requires 2 humans to have looked at every change (normally one is the author, but not necessarily for bot-authored changes).
Did you go and look at their open source repositories and open code review tools instead of fearmongering on HN? The answer can be found.
AI meaning Claude or Codex, because Gemini is a laughingstock for coding. Also Gemini was mentioned only once in the article.
I’ve heard from people inside Google that they’re prohibited from using anything but their own models. AlbertaTech on YT, ex-Google working on YT said in a video that right before she quit someone had threatened to quit if they couldn’t get access to Claude Code
The Gemini (and Claude / Codex) that we have, is much different to their internal models / harnesses.....
The latest and greatest internally at Google is more or less just Antigravity 2.0 with the latest Gemini models.
Trust me bro?
Kind of, but you can also assert it yourself; the big tech companies have billions of lines of code not publicly available, they can train their in-house code assistants on those instead of or in addition to what's available out in the open.