Hey everyone, I just want to clear some things up here based on the comments:
I (Dave) am a real person, and yes I used AI to help me with every part of this project, including my initial comment introducing it here. That was a mistake, I should have just written it myself. I am a self-taught developer who is passionate about AI development and curious about seeing what I can do with it. I come up with my ideas and test them with the help of AI. Then I have AI help me understand it. This all started because I was building multi-agent systems, just experimenting, and I noticed that they looked to me like electrical circuit diagrams. So it made me curious if there was any cross over between the laws of electrical engineering and how it might apply to agentic systems. With AI today you can pursue any idea you have, so I have fun by seeing what happens when you take concepts from one established field and apply them to something else. I had AI do deep research on electrical engineering concepts, and asked it to research how those might be applied to agentic systems. It came up with interesting ideas, so I had it plan scientific experiments on ones that looked promising, and the one that looked like the most obvious application is what turned into LoopGain. With the help of AI I developed it into a working system, created a website and company around it, made the project open source, wrote blog articles about the experiments, then asked AI how to best share it with the world. I love developing useful tools and researching the edge of AI capabilities, so that is what I am doing here. Thanks for reading.
I'm not sure you can so easily calculate 'error' for most real world cases.
In your example you are calculating error as a function of missed tests, for example. That's a pretty good example, you can't just leave that value at 2.5% of starting. It needs to be zero. All existing loops have logic to not proceed until those tests all pass.
If you can propose a means of generating an error signal in actual code you've just invented a way of automatically grading the goodness of code, which should make you quite a lot of money!
> All existing loops have logic to not proceed until those tests all pass.
My understanding is that this tool is mostly about “giving up” sooner, when it’s obvious that additional loops are not getting any closer to passing those tests.
It’s a cost saving measure when the problem is intractable, and needs more human context, etc.
Hey HN — Dave here. While building a multi-agent system a couple of years ago, I kept noticing that the diagrams of agents passing feedback to each other looked just like the electrical circuit diagrams from control theory. I got curious whether the established math transferred, and to my surprise it did. LoopGain is the first product of that research: an open-source library that replaces the max_iterations=N cap on agent loops with an actual measurement of whether the loop is still improving.
The cap is how nearly every verify-revise loop gets stopped today, and it's wrong in both directions. Stop too early and you clip a loop that was still improving. Stop too late and you pay for iterations after the loop already found its best answer — and ship the final attempt, which is sometimes worse than one it already had.
On a 2,000-trial benchmark (paired real-API runs, five loop patterns across six framework adapters, three model providers, pre-registered protocol with kill criteria), LoopGain cut total API spend by 92.8% vs max_iterations=20 ($27.05 → $1.94) and median wall-clock ~15x (30.9s → 2.1s). A cross-vendor judge preferred LoopGain's outputs on the weighted average (0.678 across 1,800 pairwise comparisons) — mostly because LoopGain returns the iteration with the lowest error it saw, while a fixed cap ships whatever the final iteration produced. The raw data and methodology are public, the full run is browsable at dashboard.loopgain.ai/benchmark, and there's a replay of a real benchmark run (all measured, no simulation) at dashboard.loopgain.ai/demo.
How it works: each iteration, LoopGain takes the ratio of the current error to the previous error — the loop's empirical loop gain (Aβ, borrowed from control theory). Aβ<1 means the error shrank — the loop is improving. Aβ≥1 means it held or grew — the loop is stuck or making things worse. A trajectory classifier reads the recent Aβ values, labels the loop (FAST_CONVERGE / CONVERGING / STALLING / OSCILLATING / DIVERGING), and decides whether to keep going, stop here, or stop and roll back to the lowest-error output so far.
Integration is a few lines around any loop that produces an error signal:
from loopgain import LoopGain
lg = LoopGain(target_error=0.0)
output = generate(task) # first attempt
while lg.should_continue():
errors = verify(output) # e.g. count of failing tests
lg.observe(errors, output=output) # the only LoopGain call in the loop
output = revise(output, errors)
result = lg.result # .best_output, .outcome, .convergence_profile, .savings_vs_fixed_cap
Honest limits, because they matter more than the headline: LoopGain detects convergence, not correctness — it inherits your verifier's blind spots. I re-graded my own benchmark and 4.5% of "converged" code-gen runs passed every check the loop ran but failed a fuller held-out test suite. And savings depend on workload: failure-heavy loops save ~78–84%, not 92.8%. There's a writeup on designing verifiers strong enough to trust on the blog.
Apache-2.0, pip install loopgain. Adapters for LangGraph, CrewAI, AutoGen, LangChain, OpenAI Agents SDK, and the Claude Agent SDK; the raw API works for anything with a measurable error.
What I'd really love from HN: if you run production agent loops, I'm interested in whether the stop decisions match what you see empirically — and what your loops' error signals actually look like, since the verifier is what makes or breaks the stop. Happy to answer anything.
To the author (Dave): have some faith in your voice. If you can’t describe the project as well as the AI (because you vibe coded it), then take a dang hour to ask the AI how it works and do a proper write up without copy pasting. Nobody is going to bother otherwise; it’s too obviously AI generated.
Hey, thanks for your comment. This is an idea that I thought up, researched, developed, and in the end built a useful, working, open source tool. Yes I used AI to assist me in the process, it's a complicated subject. But I take your point that it would be better to write in my own words. You're right, lesson learned.
Anyhow, whether I used a tool like AI to assist me or not, I am here to share something I built that I hope others will find useful. I am building what I am passionate about.
Your fully generated project description makes people lose faith in the actual project. If you went through the trouble of writing the whole code yourself, why generate the comment presenting it to the public wholesale.
I've got an application for this. Will try and get some benchmarks in the next few weeks. We're definitely attempting to converge on a quality inference without wasting tokens/time.
For now, we're passing the first pass to the humans, and that's roughly 80 for 20, but to get the last 20 without blowing up our token spend? This might be the approach.
What prompt are you using to generate this exact form of thing?
Problem history story...
Everyone's doing it wrong...
Proof of claims ...
Abstract ...
"Honest" bustedness ...
Keyword stuffing ...
Call for feedback ...
"What I'd really love from Dave:" is what's causing this particular structure. What model/harness/agent are you using, whether automated or pasting it yourself?
Hey everyone, I just want to clear some things up here based on the comments:
I (Dave) am a real person, and yes I used AI to help me with every part of this project, including my initial comment introducing it here. That was a mistake, I should have just written it myself. I am a self-taught developer who is passionate about AI development and curious about seeing what I can do with it. I come up with my ideas and test them with the help of AI. Then I have AI help me understand it. This all started because I was building multi-agent systems, just experimenting, and I noticed that they looked to me like electrical circuit diagrams. So it made me curious if there was any cross over between the laws of electrical engineering and how it might apply to agentic systems. With AI today you can pursue any idea you have, so I have fun by seeing what happens when you take concepts from one established field and apply them to something else. I had AI do deep research on electrical engineering concepts, and asked it to research how those might be applied to agentic systems. It came up with interesting ideas, so I had it plan scientific experiments on ones that looked promising, and the one that looked like the most obvious application is what turned into LoopGain. With the help of AI I developed it into a working system, created a website and company around it, made the project open source, wrote blog articles about the experiments, then asked AI how to best share it with the world. I love developing useful tools and researching the edge of AI capabilities, so that is what I am doing here. Thanks for reading.
I'm not sure you can so easily calculate 'error' for most real world cases.
In your example you are calculating error as a function of missed tests, for example. That's a pretty good example, you can't just leave that value at 2.5% of starting. It needs to be zero. All existing loops have logic to not proceed until those tests all pass.
If you can propose a means of generating an error signal in actual code you've just invented a way of automatically grading the goodness of code, which should make you quite a lot of money!
> All existing loops have logic to not proceed until those tests all pass.
My understanding is that this tool is mostly about “giving up” sooner, when it’s obvious that additional loops are not getting any closer to passing those tests.
It’s a cost saving measure when the problem is intractable, and needs more human context, etc.
Hey HN — Dave here. While building a multi-agent system a couple of years ago, I kept noticing that the diagrams of agents passing feedback to each other looked just like the electrical circuit diagrams from control theory. I got curious whether the established math transferred, and to my surprise it did. LoopGain is the first product of that research: an open-source library that replaces the max_iterations=N cap on agent loops with an actual measurement of whether the loop is still improving.
The cap is how nearly every verify-revise loop gets stopped today, and it's wrong in both directions. Stop too early and you clip a loop that was still improving. Stop too late and you pay for iterations after the loop already found its best answer — and ship the final attempt, which is sometimes worse than one it already had.
On a 2,000-trial benchmark (paired real-API runs, five loop patterns across six framework adapters, three model providers, pre-registered protocol with kill criteria), LoopGain cut total API spend by 92.8% vs max_iterations=20 ($27.05 → $1.94) and median wall-clock ~15x (30.9s → 2.1s). A cross-vendor judge preferred LoopGain's outputs on the weighted average (0.678 across 1,800 pairwise comparisons) — mostly because LoopGain returns the iteration with the lowest error it saw, while a fixed cap ships whatever the final iteration produced. The raw data and methodology are public, the full run is browsable at dashboard.loopgain.ai/benchmark, and there's a replay of a real benchmark run (all measured, no simulation) at dashboard.loopgain.ai/demo.
How it works: each iteration, LoopGain takes the ratio of the current error to the previous error — the loop's empirical loop gain (Aβ, borrowed from control theory). Aβ<1 means the error shrank — the loop is improving. Aβ≥1 means it held or grew — the loop is stuck or making things worse. A trajectory classifier reads the recent Aβ values, labels the loop (FAST_CONVERGE / CONVERGING / STALLING / OSCILLATING / DIVERGING), and decides whether to keep going, stop here, or stop and roll back to the lowest-error output so far.
Integration is a few lines around any loop that produces an error signal:
Honest limits, because they matter more than the headline: LoopGain detects convergence, not correctness — it inherits your verifier's blind spots. I re-graded my own benchmark and 4.5% of "converged" code-gen runs passed every check the loop ran but failed a fuller held-out test suite. And savings depend on workload: failure-heavy loops save ~78–84%, not 92.8%. There's a writeup on designing verifiers strong enough to trust on the blog.Apache-2.0, pip install loopgain. Adapters for LangGraph, CrewAI, AutoGen, LangChain, OpenAI Agents SDK, and the Claude Agent SDK; the raw API works for anything with a measurable error.
What I'd really love from HN: if you run production agent loops, I'm interested in whether the stop decisions match what you see empirically — and what your loops' error signals actually look like, since the verifier is what makes or breaks the stop. Happy to answer anything.
To the author (Dave): have some faith in your voice. If you can’t describe the project as well as the AI (because you vibe coded it), then take a dang hour to ask the AI how it works and do a proper write up without copy pasting. Nobody is going to bother otherwise; it’s too obviously AI generated.
Hey, thanks for your comment. This is an idea that I thought up, researched, developed, and in the end built a useful, working, open source tool. Yes I used AI to assist me in the process, it's a complicated subject. But I take your point that it would be better to write in my own words. You're right, lesson learned.
Anyhow, whether I used a tool like AI to assist me or not, I am here to share something I built that I hope others will find useful. I am building what I am passionate about.
I don’t think this is the right take-away though.
Your fully generated project description makes people lose faith in the actual project. If you went through the trouble of writing the whole code yourself, why generate the comment presenting it to the public wholesale.
He’s passionate about using AI!
I've got an application for this. Will try and get some benchmarks in the next few weeks. We're definitely attempting to converge on a quality inference without wasting tokens/time.
For now, we're passing the first pass to the humans, and that's roughly 80 for 20, but to get the last 20 without blowing up our token spend? This might be the approach.
Thanks for the write up comment, btw.
Hey gavmor, thanks for your comment! Let me know if you need any assistance, I would be happy to help or get any feedback at dave@loopgain.ai
Fitz, Dave, whoever is "here" ...
What prompt are you using to generate this exact form of thing?
Problem history story...
Everyone's doing it wrong...
Proof of claims ...
Abstract ...
"Honest" bustedness ...
Keyword stuffing ...
Call for feedback ...
"What I'd really love from Dave:" is what's causing this particular structure. What model/harness/agent are you using, whether automated or pasting it yourself?
Hey, please see my response to lochnessduck, thanks