> Mo is a custom agent harness for catching bugs that builds on the past 3 years of Momentic and millions of test traces from complex, real-world applications. Mo is built for recall: it finds the most real bugs, though Codex is a bit more precise at 86% vs 84%. The tradeoff is cost: Mo costs $2.67 per bug vs $0.80 for Codex, less than Claude Code and Devin.
is this good? so Mo costs more money per bug? I'm confused.
Hey! I'm Henry, one of the engineers working on Mo. You're looking at the right numbers, but the takeaway is slightly different from ours.
The main thing we're trying to show is that Mo finds over 3x as many correct bugs in the same amount of time as the next-best QA agent. This is the main thing we optimize for since the worst thing you can do in QA is miss a real bug and ship broken code to prod. Currently, the tradeoff you make for that better recall is higher cost. I would say the precision is more or less comparable, but marginally lower.
Obviously we'd love to be beating every other agent on all of these numbers, and we're actively working on it, but for now this is an honest and transparent look at how the tool is doing.
Feel free to take a look at the full writeup for more detail!
Hey I'm Gabe (one of the engineers @Momentic). In general Mo's advantage is in the total amount of bugs found. In comparison to claude code we find roughly 5x as many bugs at 35% less per bug (for the 60 min session).
Some questions are top of mind after having done POCs and trials with other companies that have products in this space (excuse me if they are answered somewhere obvious - I skimmed the site and your post before drafting this):
1. What is vendor lock-in like with Momentic? In my mind there's space for a mature QA platform that agentically generates and maintains playwright/similar tests, but perhaps I'm wrong.
1.b. will Momentic code or testing standard or some aspect be open enough that it is feasible to move away from it if needed, or continue using it if Momentic goes under/deploys major breaking changes?
2. How deterministic are Momentic tests?
3. Who is your target user? Product owner / developer / SDET / ?
----
Also - is this saying Codex is better and cheaper than Mo?
"... though Codex is a bit more precise at 86% vs 84%. The tradeoff is cost: Mo costs $2.67 per bug vs $0.80 for Codex, less than Claude Code and Devin."
Are you open to listing other agents on the benchmark? I’m building my own and would like to run it on the full suite, but want to check before putting in the effort.
The moat is about a week before the frontier models integrate the same capabilities. They aren’t snapping up projects like this for talent acquisitions anymore, they will just build it themselves.
Hey there, I'm Jeff (one of the founders). To be honest I don't know if there is or will be any kind of moat in SaaS. But today:
1/ General purpose models are still not good at this task. I would personally love it if I could trust my agent when it says its done after a 24h build. But whether it's because of sycophancy, context degradation over time, or something else coding agents still struggle to verify their own work. Maybe with 1B context windows this will change.
2/ Infra is still annoying to setup. We wanted to provide a unified interface across web, mobile and soon desktop as well. So users don't have to worry about provisioning, concurrency, security, etc.
3/ As with any SaaS, even beyond the initial build, maintenance is a headache. I think most companies would rather not consider agentic QA as one of their core competencies and investments. It takes a ton of eng effort to keep the lights on, continually update the models used, train new ones, groom ground truth, onboard infra providers, etc.
So is this yet another SaaS product that's going to attempt to charge me for token usage? Sorry to sound negative, but nowadays it's an instant turn off because it's like the current subscription crisis on steroids. BYOA should be a default.
To be more constructive: let's say I have 2 frontier subscriptions and I end up with 20% left on each at the end of the week. It feels awful knowing that I would _love_ to use that with Momentic but I can't because I have to pay for yet another set of proprietary non-transferable tokens. It's like dealing with airline rewards.
Hey I'm Jeff (one of the co-founders). That's fair feedback. We wrestled a lot with whether to support BYOA or open source the harness. Ultimately there was no configuration with off the shelf models that we were happy with performance-wise. A lot of the eng work that went into this came tuning custom agents from large datasets to do the "grunt work" like element grounding and visual understanding. Plugging in vanilla GPT6.1 Sol for every task into our harness just doesn't yield a pareto optimal result that makes sense as a standalone product (you are better off asking codex to test).
That doesn't mean we won't eventually support byok, especially as OSS models catch up! We have our finger on the trigger there.
the harness is the lock-in with these subscriptions. if you could just take your tokens to any app, there would be no point to stay on that companies ecosystem. I definitely agree that tokens and subscriptions should be portable and fungible, but any service offering that would be much more expensive than just getting a claude or codex sub.
I literally just made it up: Bring your own Agent. It was a poor attempt at making fun of the current SaaS landscape. Although I would be surprised to be the first one to have ever used that acronym.
Hey! I'm Henry, one of the engineers working on Mo. The approach to testing itself is very similar to UAT, except that it's fully automated. This means you can speed up the feedback loop significantly. Imagine instead of having to wait for the next release and run a time-consuming manual bugbash in order to find out that your changes broke something, you can get feedback immediately on any PR you want, at a fraction of the cost.
The last 3 years saw tons of "custom agent harnesses" come up, only for them to be superseded by the next generation of frontier models, which could do the same (and more) in their generic harness. So, what will you sell when Opus 6.0 swallows Mo's edge in 5 months from now? And probably cheaper, when using a Claude subscription instead of paying API prices.
Hiya, I'm one of the co-founders at Momentic. I would actually push back against that take, we work with frontier labs a ton and this year we've actually seen a lot of realization on their part that custom harnesses matter. E.g. you could argue Devin is just a frontier model wrapper and yet it does feel better and is super popular. Same with code review tooling -- afaik everyone on https://codereview.withmartian.com/ mostly uses off-the-shelf models and yet it's the architecture, durable context (knowledge base), and data they have that gives them an edge. IMO opus 6 would probably just unlock more capabilities for Mo rather than eroding the edge.
> Mo is a custom agent harness for catching bugs that builds on the past 3 years of Momentic and millions of test traces from complex, real-world applications. Mo is built for recall: it finds the most real bugs, though Codex is a bit more precise at 86% vs 84%. The tradeoff is cost: Mo costs $2.67 per bug vs $0.80 for Codex, less than Claude Code and Devin.
is this good? so Mo costs more money per bug? I'm confused.
Hey! I'm Henry, one of the engineers working on Mo. You're looking at the right numbers, but the takeaway is slightly different from ours.
The main thing we're trying to show is that Mo finds over 3x as many correct bugs in the same amount of time as the next-best QA agent. This is the main thing we optimize for since the worst thing you can do in QA is miss a real bug and ship broken code to prod. Currently, the tradeoff you make for that better recall is higher cost. I would say the precision is more or less comparable, but marginally lower.
Obviously we'd love to be beating every other agent on all of these numbers, and we're actively working on it, but for now this is an honest and transparent look at how the tool is doing.
Feel free to take a look at the full writeup for more detail!
Right? Is the takeaway from that not just that Codex is better for cheaper?
it's right there in the writeup, it costs more but finds more bugs.
Why is this better than running Opus 5.5 directly against my app with a custom Playwright-based harness for in my own subscription?
Hey I'm Gabe (one of the engineers @Momentic). In general Mo's advantage is in the total amount of bugs found. In comparison to claude code we find roughly 5x as many bugs at 35% less per bug (for the 60 min session).
Congrats on the launch!
Some questions are top of mind after having done POCs and trials with other companies that have products in this space (excuse me if they are answered somewhere obvious - I skimmed the site and your post before drafting this):
1. What is vendor lock-in like with Momentic? In my mind there's space for a mature QA platform that agentically generates and maintains playwright/similar tests, but perhaps I'm wrong.
1.b. will Momentic code or testing standard or some aspect be open enough that it is feasible to move away from it if needed, or continue using it if Momentic goes under/deploys major breaking changes?
2. How deterministic are Momentic tests?
3. Who is your target user? Product owner / developer / SDET / ?
----
Also - is this saying Codex is better and cheaper than Mo?
"... though Codex is a bit more precise at 86% vs 84%. The tradeoff is cost: Mo costs $2.67 per bug vs $0.80 for Codex, less than Claude Code and Devin."
Are you open to listing other agents on the benchmark? I’m building my own and would like to run it on the full suite, but want to check before putting in the effort.
Yes we are (it's MIT licensed)! Feel free to email me at jeff@momentic.ai or file a pull request directly on https://github.com/momentic-ai/qa-bench
Everything has to be reproducible and runnable by the general public
So, what's the moat?
The moat is about a week before the frontier models integrate the same capabilities. They aren’t snapping up projects like this for talent acquisitions anymore, they will just build it themselves.
Hey there, I'm Jeff (one of the founders). To be honest I don't know if there is or will be any kind of moat in SaaS. But today:
1/ General purpose models are still not good at this task. I would personally love it if I could trust my agent when it says its done after a 24h build. But whether it's because of sycophancy, context degradation over time, or something else coding agents still struggle to verify their own work. Maybe with 1B context windows this will change.
2/ Infra is still annoying to setup. We wanted to provide a unified interface across web, mobile and soon desktop as well. So users don't have to worry about provisioning, concurrency, security, etc.
3/ As with any SaaS, even beyond the initial build, maintenance is a headache. I think most companies would rather not consider agentic QA as one of their core competencies and investments. It takes a ton of eng effort to keep the lights on, continually update the models used, train new ones, groom ground truth, onboard infra providers, etc.
we also build similar product in-house for our apps.
I have something similar I made for my side project. You don't really need a platform for this.
So is this yet another SaaS product that's going to attempt to charge me for token usage? Sorry to sound negative, but nowadays it's an instant turn off because it's like the current subscription crisis on steroids. BYOA should be a default.
To be more constructive: let's say I have 2 frontier subscriptions and I end up with 20% left on each at the end of the week. It feels awful knowing that I would _love_ to use that with Momentic but I can't because I have to pay for yet another set of proprietary non-transferable tokens. It's like dealing with airline rewards.
Hey I'm Jeff (one of the co-founders). That's fair feedback. We wrestled a lot with whether to support BYOA or open source the harness. Ultimately there was no configuration with off the shelf models that we were happy with performance-wise. A lot of the eng work that went into this came tuning custom agents from large datasets to do the "grunt work" like element grounding and visual understanding. Plugging in vanilla GPT6.1 Sol for every task into our harness just doesn't yield a pareto optimal result that makes sense as a standalone product (you are better off asking codex to test).
That doesn't mean we won't eventually support byok, especially as OSS models catch up! We have our finger on the trigger there.
the harness is the lock-in with these subscriptions. if you could just take your tokens to any app, there would be no point to stay on that companies ecosystem. I definitely agree that tokens and subscriptions should be portable and fungible, but any service offering that would be much more expensive than just getting a claude or codex sub.
What does BYOA mean?
I literally just made it up: Bring your own Agent. It was a poor attempt at making fun of the current SaaS landscape. Although I would be surprised to be the first one to have ever used that acronym.
Interesting, I assumed Bring Your Own API-key
Assuming agent/access tokens, in other words you're not going through a middleman for LLM access.
I assume bring your own agent
What does this do that UAT doesn't do? Sounds like you're combining UAT with gloves-on red teaming and I guess property testing APIs.
Hey! I'm Henry, one of the engineers working on Mo. The approach to testing itself is very similar to UAT, except that it's fully automated. This means you can speed up the feedback loop significantly. Imagine instead of having to wait for the next release and run a time-consuming manual bugbash in order to find out that your changes broke something, you can get feedback immediately on any PR you want, at a fraction of the cost.
The last 3 years saw tons of "custom agent harnesses" come up, only for them to be superseded by the next generation of frontier models, which could do the same (and more) in their generic harness. So, what will you sell when Opus 6.0 swallows Mo's edge in 5 months from now? And probably cheaper, when using a Claude subscription instead of paying API prices.
Hiya, I'm one of the co-founders at Momentic. I would actually push back against that take, we work with frontier labs a ton and this year we've actually seen a lot of realization on their part that custom harnesses matter. E.g. you could argue Devin is just a frontier model wrapper and yet it does feel better and is super popular. Same with code review tooling -- afaik everyone on https://codereview.withmartian.com/ mostly uses off-the-shelf models and yet it's the architecture, durable context (knowledge base), and data they have that gives them an edge. IMO opus 6 would probably just unlock more capabilities for Mo rather than eroding the edge.