The answer to that is no. A lot of these are released on vibes and showing that it does what it says it does. For most organizations that care about processes and procedures thats... enough? I've worked on agentic development in enterprise and they just want their way to stamp their old methods on to the new agentic world so it can pass through their release hoops.
I've tried to benchmark them the most common public skills including those for ADLC (ie SuperPowers) couple of times here https://orcabot.com/benchmarks/agent-skills. Annoyingly I haven't been able to release one in a while because its been next to impossible to stop the new frontier models from cheating one way or another.
I'm not sure that it needs to be this complicated. But yes, I am interested in the problem of hard guardrails between multiple agents, and it is not an easy problem to solve.
Would someone tell me how this happened? We were the fucking vanguard of agent DLC in this country. The AWS Bedrock AgentCore was agent DLC to own. Then the other guy came out with an eight-agent team. Were we scared? Hell, no. Because we hit back with a little thing called the Bedrock Managed Agents Cloudtrail Datalake. That’s eight agents and a data lake. For moisture. But you know what happened next? Shut up, I’m telling you what happened—the bastards went to nine domain experts, a reviewer and a composer. Now we’re standing around with our cocks in our hands, selling eight agents and a datalake. Moisture or no, suddenly we’re the chumps. Well, fuck it. We’re going to fourteen agents with eleven domain experts.
Sure, we could go to nine domain expert agents next, like the competition. That seems like the logical thing to do. After all, eight worked out pretty well, and nine is the next number after eight. So let’s play it safe. Let’s make a thicker sandbox, add some IOPS and call it the Bedrock9SuperTurbo. Why innovate when we can follow? Oh, I know why: Because we’re a business, that’s why!
You think it’s crazy? It is crazy. But I don’t give a shit. From now on, we’re the ones who have the edge in the multi-agent game. Is Devin the best a man can get? Fuck, no. Bedrock AI-DLC is the best a man can get.
Here’s the report from Engineering. Someone put it in the bathroom: I want to wipe my ass with it. They don’t tell me what to invent—I tell them. And I’m telling them to stick two more agents in there. I don’t care how. Make the agents so token efficient they’re invisible. Put some on an Alexa. I don’t care if they have to cram the fourteenth agent in perpendicular to the other ones, just do it!
This is a riff on this [1] classic for those who forgot like I did.
Earlier this year I spent considerable time making a six-agent multi-LLM configuration in OpenCode with a Bifrost multi-model router before just switching to OpenSpec and GLM-5.3 and calling it a day.
I prefer https://github.com/Fission-AI/OpenSpec because it works also on brown-field. If you want an unbias-demo, take a look also at
https://github.com/daitangio/take-ai-control/tree/demo/nello...
branch demo/nello-openspec
Sounds good in theory, but are there evaluations of whether or not it actually works?
The answer to that is no. A lot of these are released on vibes and showing that it does what it says it does. For most organizations that care about processes and procedures thats... enough? I've worked on agentic development in enterprise and they just want their way to stamp their old methods on to the new agentic world so it can pass through their release hoops.
I've tried to benchmark them the most common public skills including those for ADLC (ie SuperPowers) couple of times here https://orcabot.com/benchmarks/agent-skills. Annoyingly I haven't been able to release one in a while because its been next to impossible to stop the new frontier models from cheating one way or another.
> A lot of these are released on vibes
Yeah, I kind of figured. I was hoping was AWS-affiliated lab would be a little more rigorous
Probably the greatest problem in AI right now is proof / benchmarking
I check every benchmark site weekly, compare harnesses and models, and its still pretty much just vibes
My org has something similar.
Evals showed that speed and token use improved when it was removed.....
The models have improved and internalized some of these principles.
They still have value, IMO, because they help improve consistency (at the expense of speed and cost).
I am one of the organisational victims who have been trialling this.
My assessment has been just use the native harness.
I'm not sure that it needs to be this complicated. But yes, I am interested in the problem of hard guardrails between multiple agents, and it is not an easy problem to solve.
Would the submitter like to explain what he likes about it?
Well, if it was any good we know for sure that they didn't use it to generate their README.MD with it.
Would someone tell me how this happened? We were the fucking vanguard of agent DLC in this country. The AWS Bedrock AgentCore was agent DLC to own. Then the other guy came out with an eight-agent team. Were we scared? Hell, no. Because we hit back with a little thing called the Bedrock Managed Agents Cloudtrail Datalake. That’s eight agents and a data lake. For moisture. But you know what happened next? Shut up, I’m telling you what happened—the bastards went to nine domain experts, a reviewer and a composer. Now we’re standing around with our cocks in our hands, selling eight agents and a datalake. Moisture or no, suddenly we’re the chumps. Well, fuck it. We’re going to fourteen agents with eleven domain experts.
Sure, we could go to nine domain expert agents next, like the competition. That seems like the logical thing to do. After all, eight worked out pretty well, and nine is the next number after eight. So let’s play it safe. Let’s make a thicker sandbox, add some IOPS and call it the Bedrock9SuperTurbo. Why innovate when we can follow? Oh, I know why: Because we’re a business, that’s why!
You think it’s crazy? It is crazy. But I don’t give a shit. From now on, we’re the ones who have the edge in the multi-agent game. Is Devin the best a man can get? Fuck, no. Bedrock AI-DLC is the best a man can get.
Here’s the report from Engineering. Someone put it in the bathroom: I want to wipe my ass with it. They don’t tell me what to invent—I tell them. And I’m telling them to stick two more agents in there. I don’t care how. Make the agents so token efficient they’re invisible. Put some on an Alexa. I don’t care if they have to cram the fourteenth agent in perpendicular to the other ones, just do it!
This is a riff on this [1] classic for those who forgot like I did.
Earlier this year I spent considerable time making a six-agent multi-LLM configuration in OpenCode with a Bifrost multi-model router before just switching to OpenSpec and GLM-5.3 and calling it a day.
[1] https://theonion.com/fuck-everything-were-doing-five-blades-...
> 5 phases and 33 stages from initialization through operation
> 14 agents: 11 domain experts, 2 reviewers, and an adaptive composer
wow, that's a lot of domain experts, I'm convinced
> The current recommended model is Claude Opus 4.8.
Staying well behind the frontier, just in case.
the technical term is “pacing”
It was like that time I nearly dropped out of undergrad, but it was my mistake, I had just forgotten that I was actually a doctorate.
Not unless you look at my framework (Which I've just hacked together after looking at this) - it has 12! domain experts!
Wow does it really have 479.001.600 experts?
It's a great for tokenmaxxing /s