> codebase with hundreds of thousands of lines of code and go from 0% to 80%+ coverage in the next few weeks
I had a coworker do this with windsurf + manual driving awhile back and it was an absolute mess. Awful tests that were unmaintainable and next to useless (too much mocking, testing that the code “works the way it was written”, etc.). Writing a useful test suite is one of the most important parts of a codebase and requires careful deliberate thought. Without deep understanding of business logic (which takes time and is often lost after the initial devs move on) you’re not gonna get great tests.
To be fair to AI, we hired a “consultant” that also got us this same level of testing so it’s not like there is a high bar out there. It’s just not the kind of problem you can solve in 2 weeks.
I find coding agents can produce very high quality tests if and only if you give them detailed guidance and good starting examples.
Ask a coding agent to build tests for a project that has none and you're likely to get all sorts of messy mocks and tests that exercise internals when really you want them to exercise the top level public API of the project.
Give them just a few starting examples that demonstrate how to create a good testable environment without mocking and test the higher level APIs and they are much less likely to make a catastrophic mess.
You're still going to have to keep an eye on what they're doing and carefully review their work though!
> I find coding agents can produce very high quality tests if and only if you give them detailed guidance and good starting examples.
I find this to be true for all AI coding, period. When I have the problem fully solved in my head, and I write the instructions to explicitly and fully describe my solution, the code that is generated works remarkably well. If I am not sure how it should work and give more vague instructions, things don't work so well.
Yeah, same. Usually I'll ask the agent for a few alternatives, to make sure I'm not missing something, but the solution I wanted tends to be the best one. I also get into a lot of me saying "hm, why are you doing it that way?" "Oh yeah, that isn't actually going to work, sorry".
Yes, but the act of writing code is an important part of figuring out what you need. So I’m left wondering how much of a prefect the AI can actually help with.
To be clear I do use AI for some code gen. But I try to use it less than I see others use it.
Eh, I think my decades of experience writing my own code was necessary for me to develop the skills to be able to precisely tell the AI what to build, but I don't think I need to (always) write new code to know how to know what I need.
Now, if the thing I am building requires a technology I am not familiar with, I will spend some time reading and writing some simple test code to learn how it works, but once I understand it I can then let the AI build from scratch.
Of course, this does rely on the fact that I have years of coding experience that came prior to AI, and I do wonder how new coders can do it without putting in the work to learn how to build working software without AI before using AI.
I feel like that leaves me with the hard part of writing tests, and only saves me the bit I can usually power through quickly because it's easy to get into a flow state for it.
I've think they're also much better at creating useful end to end UI tests than unit or integration tests, but unfortunately those are hard to create self contained environments for without bringing a lot of baggage and docker containers, which not all agent VMs might support yet. Getting headless QT running was a pain too, but now ChatGPT Codex can see screenshots and show them in chat (Claude Code can't show them in the chat for some reason) and it's been generating much better end to end tests than I've seen for unit/integration.
Has anyone had success with specific prompts to avoid the agent over-indexing on implementation details? For instance, something like: "Before each test case, add a comment justifying the business case for every assumption made here, without regards to implementation details. If this cannot be made succinct, or if there is ambiguity in the business case, the test case should not be generated."
I've had reasonable success from doing something like this, though it is my current opinion that it's better to write the first few tests yourself to establish a clear pattern and approach. However, if you don't care that much (which is common with side projects):
Starting point: small-ish codebase, no tests at all:
> I'd like to add a test suite to this project. It should follow language best practices. It should use standard tooling as much as possible. It should focus on testing real code, not on mocking/stubbing, though mocking/stubbing is ok for things like third party services and parts of the code base that can't reasonably run in a test environment. What are some design options we could do? Don't write any code yet, present me the best of the options and let me guide you.
> Ok, I like option number two. Put the basic framework in place and write a couple of dummy tests.
> Great, let's go ahead and write some real tests for module X.
and etc. For a project with an existing and mature test suite, it's much easier:
> I'd like to add a test (or improve a test) for module X. Use the existing helpers and if you find yourself needing new helpers, ask me about the approach before implementing
I've also found it helpful to put things in AGENTS.md or CLAUDE.md about tests and my preferences, such as:
- Tests should not rely on sleep to avoid timing issues. If there is a timing issue, present me with options and let me guide you
- Tests should not follow an extreme DRY pattern, favor human readability over absolute DRYness
- Tests should focus on testing real code, not on mocking/stubbing, though mocking/stubbing is ok for things like third party services and parts of the code base that can't reasonably run in a test environment.
- Tests should not make assumptions about the current running state of the environment, nor should they do anything that isn't cleaned up before completing the test to avoid polluting future tests
I do want to stress that every project and framework is different and has different needs. As you discover the AI doing something you don't like, add it to the prompts or the AGENTS.md/CLAUDE.md. Eventually it will get pretty decent, though never blindly trust it because a butterfly flapping it's wings in Canada sometimes causes it to do unexpected things.
Does it depend on the model? I would have expected the bigger ones to be better with common sense and not fixating on irrelevant details. But I have only used them with quite small codebases so far. (Which have basically no internals to exercise!)
There is no free lunch. The amount of prompt writing to give the LLM enough context about your codebase etc is comparable to writing the tests yourself.
Code assistance tools might speed up your workflow by maybe 50% or even 100%, but it's not the geometric scaling that is commonly touted as the benefits of autonomous agentic AI.
And this is not a model capability issue that goes away with newer generations. But it's a human input problem.
For example, you can spend a few hours writing a really good set of initial tests that cover 10% of your codebase, and another few hours with an AGENTS.md that gives the LLM enough context about the rest of the codebase. But after that, there's a free* lunch because the agent can write all the other tests for you using that initial set and the context.
This also works with "here's how I created the Slack API integration, please create the Teams integration now" because it has enough to learn from, so that's free* too. This kind of pattern recognition means that prompting is O(1) but the model can do O(n) from that (I know, terrible analogy).
*Also literally becomes free as the cost of tokens approaches zero
A neat part of this is it mimics how people get onboarded onto codebases. People usually aren't figuring out how to write tests from scratch; they look at the current best practices for similar functionality in the codebase and start there. And then as they continue to work there they try to influence new best practices.
I recently had a bunch of Claude credits so got it to write a language implementation for me. It probably took 4 hours of my time, but judging by other implementations online I'd say the average implementation time is hundreds of hours.
The fact that the model knew the language and there are existing tests I could use is a radical difference.
I agree. It is very easy to fall in the trap: "I let AI write all the tests" and then find yourself in a situation where you have an unmaintainable mess with the only way to fix broken test within a reasonable time is to blindly accept AI to do that. Which exposes you to the similar level of risk as running any unchecked AI code - you just can't trust that it works correctly
I had a funny experience with Claude (Web) the other day.
Uploaded a Prolog interpreter in Python and asked for a JS version. It surprised my by not just giving me a code block, but actually running a bunch of commands in its little VM, setting up a npm project, it even wrote a test suite and ran it to make sure all the tests pass!
I was very impressed, then I opened the tests script and saw like 15 lines of code, which ran some random functions, did nothing to test their correctness, and just printed "Test passed!" regardless of the result.
With recent experience I'm thinking the correct solution is a separate agent with prompting to exclusively be a test critic given a growing list of bad testing patterns to avoid, agent 2 gives feedback to agent 1. Separating agents into having unique jobs.
An agent does a good job fixing it's own bad ideas when it can run tests, but the biggest blocker I've been having is the agent writing bad tests and getting stuck or claiming success by lobotomizing a test. I got pretty far with myself being the test critic and that being mostly the only input the agent got after the initial prompt. I'm just betting it could be done with a second agent.
Cleanroom design of "this is a function's interface, it does this and that, write tests for that function to pass" generally can get you pretty decent results.
But "throw vague prompt at AI direction" does about as well as doing same thing with an intern.
I don't think its that surprising. Bash is old as dirt and scripts by definition are meant to be simple. Where AI struggles is when you add complexity like object-oriented design. That's when the effect of it trying to solve every problem in a way unique to it just takes shit off the rails. LLMs known design patterns exist but they don't know how to use them because that's not how deep learning approaches problem solving.
FYI, you're missing two patterns that allow the `--key=value` admirers and the `-alltheshortopsinasinglestring` spacebar savers among us to be happy (for the otherwise excellent options parsing code).
shopt -s extglob
case "$1"
# Flag support - allow -xyz z-takes-params
-@(a|b|c)*) _flag=${1:1:1}; _rest=${1:2}; shift; set -- "-$_flag" "-$_rest" "$@";;
# Param=Value support
-?(-)*=*) _key=${1%%=*}; _value=${1#*=}; shift; set -- "${_key}" "$_value" "$@";;
esac
I got a couple bug reports years ago on https://github.com/jdavisclark/jsformat for not dealing unicode emojis correctly when used in variable names in javascript. people are weird.
I've dubbed my loop of this as 'sicko mode' at work as I've become a bit obsessed with automating everything little thing in my flow, so I can focus on just features and bugs. It feels like a game to me and I enjoy it a lot.
can it read code review comments? I've been finding that having claude write code but letting codex review PRs is a productive workflow, claude code is capable of reading the feedback left in comments and is pretty good at following the advice.
I’m letting Claude Code review the code as part of a gitlab CI job.
It adds inline comments (using curl and the http API, nightmare to get right as glab does not support this)
CC can also read the inline comments and creates fixes. Now thinking of adding an extra CI job that will address the review comments in a separate MR.
I find that ChatGPT’s Codex reviews - which can also be set up to happen automatically on all PRs - seem smarter than Copilot’s, and make fewer mistakes. But these things change fast, maybe Copilot caught up and I didn’t notice
No codex catches genuine bugs here that multiple reviewers would have overlooked, whilst copilot only comes with nitpicks. And codex does none of those, which is also great.
You can do this with a single while loop in bash. No need for this project. Search for “ralph wiggum ai coding” and find a couple of guys that share plenty of examples and nerd out about it
It sends a flag that dangerously allows Claude to just do whatever it wants and only give us the final answer. It doesn't do the back-and-forth or ask questions.
> codebase with hundreds of thousands of lines of code and go from 0% to 80%+ coverage in the next few weeks
I had a coworker do this with windsurf + manual driving awhile back and it was an absolute mess. Awful tests that were unmaintainable and next to useless (too much mocking, testing that the code “works the way it was written”, etc.). Writing a useful test suite is one of the most important parts of a codebase and requires careful deliberate thought. Without deep understanding of business logic (which takes time and is often lost after the initial devs move on) you’re not gonna get great tests.
To be fair to AI, we hired a “consultant” that also got us this same level of testing so it’s not like there is a high bar out there. It’s just not the kind of problem you can solve in 2 weeks.
I find coding agents can produce very high quality tests if and only if you give them detailed guidance and good starting examples.
Ask a coding agent to build tests for a project that has none and you're likely to get all sorts of messy mocks and tests that exercise internals when really you want them to exercise the top level public API of the project.
Give them just a few starting examples that demonstrate how to create a good testable environment without mocking and test the higher level APIs and they are much less likely to make a catastrophic mess.
You're still going to have to keep an eye on what they're doing and carefully review their work though!
> I find coding agents can produce very high quality tests if and only if you give them detailed guidance and good starting examples.
I find this to be true for all AI coding, period. When I have the problem fully solved in my head, and I write the instructions to explicitly and fully describe my solution, the code that is generated works remarkably well. If I am not sure how it should work and give more vague instructions, things don't work so well.
Yeah, same. Usually I'll ask the agent for a few alternatives, to make sure I'm not missing something, but the solution I wanted tends to be the best one. I also get into a lot of me saying "hm, why are you doing it that way?" "Oh yeah, that isn't actually going to work, sorry".
Yes, but the act of writing code is an important part of figuring out what you need. So I’m left wondering how much of a prefect the AI can actually help with. To be clear I do use AI for some code gen. But I try to use it less than I see others use it.
Eh, I think my decades of experience writing my own code was necessary for me to develop the skills to be able to precisely tell the AI what to build, but I don't think I need to (always) write new code to know how to know what I need.
Now, if the thing I am building requires a technology I am not familiar with, I will spend some time reading and writing some simple test code to learn how it works, but once I understand it I can then let the AI build from scratch.
Of course, this does rely on the fact that I have years of coding experience that came prior to AI, and I do wonder how new coders can do it without putting in the work to learn how to build working software without AI before using AI.
It’s not just about new tech. It’s about new businesses and projects.
I feel like that leaves me with the hard part of writing tests, and only saves me the bit I can usually power through quickly because it's easy to get into a flow state for it.
Left to his own devices, I found Claude liked to copy the code under test into the test files to 'remove dependencies' :/
Or would return early from playwright tests when the desired targets couldn't be found instead of failing.
But I agree that with some guidance and a better CLAUDE.md, can work well!
I've think they're also much better at creating useful end to end UI tests than unit or integration tests, but unfortunately those are hard to create self contained environments for without bringing a lot of baggage and docker containers, which not all agent VMs might support yet. Getting headless QT running was a pain too, but now ChatGPT Codex can see screenshots and show them in chat (Claude Code can't show them in the chat for some reason) and it's been generating much better end to end tests than I've seen for unit/integration.
Has anyone had success with specific prompts to avoid the agent over-indexing on implementation details? For instance, something like: "Before each test case, add a comment justifying the business case for every assumption made here, without regards to implementation details. If this cannot be made succinct, or if there is ambiguity in the business case, the test case should not be generated."
I've had reasonable success from doing something like this, though it is my current opinion that it's better to write the first few tests yourself to establish a clear pattern and approach. However, if you don't care that much (which is common with side projects):
Starting point: small-ish codebase, no tests at all:
and etc. For a project with an existing and mature test suite, it's much easier: I've also found it helpful to put things in AGENTS.md or CLAUDE.md about tests and my preferences, such as: I do want to stress that every project and framework is different and has different needs. As you discover the AI doing something you don't like, add it to the prompts or the AGENTS.md/CLAUDE.md. Eventually it will get pretty decent, though never blindly trust it because a butterfly flapping it's wings in Canada sometimes causes it to do unexpected things.Does it depend on the model? I would have expected the bigger ones to be better with common sense and not fixating on irrelevant details. But I have only used them with quite small codebases so far. (Which have basically no internals to exercise!)
Indeed the case - luckily my codebase had some tests already and a pretty decent CLAUDE.md file so I got results I’m happy with.
I was able to do this with vitest and a ton of lint rules.
There is no free lunch. The amount of prompt writing to give the LLM enough context about your codebase etc is comparable to writing the tests yourself.
Code assistance tools might speed up your workflow by maybe 50% or even 100%, but it's not the geometric scaling that is commonly touted as the benefits of autonomous agentic AI.
And this is not a model capability issue that goes away with newer generations. But it's a human input problem.
I don't know if this is true.
For example, you can spend a few hours writing a really good set of initial tests that cover 10% of your codebase, and another few hours with an AGENTS.md that gives the LLM enough context about the rest of the codebase. But after that, there's a free* lunch because the agent can write all the other tests for you using that initial set and the context.
This also works with "here's how I created the Slack API integration, please create the Teams integration now" because it has enough to learn from, so that's free* too. This kind of pattern recognition means that prompting is O(1) but the model can do O(n) from that (I know, terrible analogy).
*Also literally becomes free as the cost of tokens approaches zero
A neat part of this is it mimics how people get onboarded onto codebases. People usually aren't figuring out how to write tests from scratch; they look at the current best practices for similar functionality in the codebase and start there. And then as they continue to work there they try to influence new best practices.
It depends on the problem domain.
I recently had a bunch of Claude credits so got it to write a language implementation for me. It probably took 4 hours of my time, but judging by other implementations online I'd say the average implementation time is hundreds of hours.
The fact that the model knew the language and there are existing tests I could use is a radical difference.
I agree. It is very easy to fall in the trap: "I let AI write all the tests" and then find yourself in a situation where you have an unmaintainable mess with the only way to fix broken test within a reasonable time is to blindly accept AI to do that. Which exposes you to the similar level of risk as running any unchecked AI code - you just can't trust that it works correctly
"My code isn't working. I know, I'll have an AI write my unit tests." Now you have two problems.
I had a funny experience with Claude (Web) the other day.
Uploaded a Prolog interpreter in Python and asked for a JS version. It surprised my by not just giving me a code block, but actually running a bunch of commands in its little VM, setting up a npm project, it even wrote a test suite and ran it to make sure all the tests pass!
I was very impressed, then I opened the tests script and saw like 15 lines of code, which ran some random functions, did nothing to test their correctness, and just printed "Test passed!" regardless of the result.
With recent experience I'm thinking the correct solution is a separate agent with prompting to exclusively be a test critic given a growing list of bad testing patterns to avoid, agent 2 gives feedback to agent 1. Separating agents into having unique jobs.
An agent does a good job fixing it's own bad ideas when it can run tests, but the biggest blocker I've been having is the agent writing bad tests and getting stuck or claiming success by lobotomizing a test. I got pretty far with myself being the test critic and that being mostly the only input the agent got after the initial prompt. I'm just betting it could be done with a second agent.
Cleanroom design of "this is a function's interface, it does this and that, write tests for that function to pass" generally can get you pretty decent results.
But "throw vague prompt at AI direction" does about as well as doing same thing with an intern.
Ah yes, classic "increase test coverage for the sake of increasing test coverage".
Aligns with vibe-coding values well: number go up – exec happy.
Which language? I've found Claude very good at Elixir test coverage (surprisingly) but a dumpster fire with any sort JS/TS testing.
Kudos on making Bash readable.
(https://github.com/AnandChowdhary/continuous-claude/blob/mai...)
im not saying OP did this, but I've actually had AI spit out some pretty stellar bash scripts, surprisingly
I don't think its that surprising. Bash is old as dirt and scripts by definition are meant to be simple. Where AI struggles is when you add complexity like object-oriented design. That's when the effect of it trying to solve every problem in a way unique to it just takes shit off the rails. LLMs known design patterns exist but they don't know how to use them because that's not how deep learning approaches problem solving.
No, you're right. It was a pretty collaborative effort with me and Claude!
FYI, you're missing two patterns that allow the `--key=value` admirers and the `-alltheshortopsinasinglestring` spacebar savers among us to be happy (for the otherwise excellent options parsing code).
For letting me know! Would you like to create a PR? Otherwise I'll add you as a Co-Authored-By!
Co-authored is fine, thanks for asking!
Gesundheit
The emojis give it away
Nah - as you'll see from my work from a pre-coding agents age, I like emoji :)
Here are receipts some from 2020: - https://github.com/AnandChowdhary/bsc-thesis - https://github.com/AnandChowdhary/slack-netlify-trigger - https://github.com/AnandChowdhary/analytics-icons
LLMs had to learn where to use emoji from somewhere, now we know who to blame ;)
I got a couple bug reports years ago on https://github.com/jdavisclark/jsformat for not dealing unicode emojis correctly when used in variable names in javascript. people are weird.
I've dubbed my loop of this as 'sicko mode' at work as I've become a bit obsessed with automating everything little thing in my flow, so I can focus on just features and bugs. It feels like a game to me and I enjoy it a lot.
It's oddly satisfying to watch your tooling improve itself.
I was expecting to see links to a bunch of opensource successful examples, projects self-managed and continuously adding code and getting better.
99.9999% of AI software is vaporware.
>run Claude code in a loop
And watch your bank account go brrr!
If you have a max plan you just watch your usage get throttled
can it read code review comments? I've been finding that having claude write code but letting codex review PRs is a productive workflow, claude code is capable of reading the feedback left in comments and is pretty good at following the advice.
I’m letting Claude Code review the code as part of a gitlab CI job. It adds inline comments (using curl and the http API, nightmare to get right as glab does not support this)
CC can also read the inline comments and creates fixes. Now thinking of adding an extra CI job that will address the review comments in a separate MR.
Have you tried GitHub Copilot? I've been trying it out directly in my PRs like you suggest. Works pretty well sometimes.
I find that ChatGPT’s Codex reviews - which can also be set up to happen automatically on all PRs - seem smarter than Copilot’s, and make fewer mistakes. But these things change fast, maybe Copilot caught up and I didn’t notice
No codex catches genuine bugs here that multiple reviewers would have overlooked, whilst copilot only comes with nitpicks. And codex does none of those, which is also great.
Exactly what I needed! I might use it for test coverage on an ancient project I need to improve...
Does this exist for Codex?
You can do this with a single while loop in bash. No need for this project. Search for “ralph wiggum ai coding” and find a couple of guys that share plenty of examples and nerd out about it
Missed opportunity to call it Claude Incontinent (CLI).
How does it handle questions asked by Claude?
It sends a flag that dangerously allows Claude to just do whatever it wants and only give us the final answer. It doesn't do the back-and-forth or ask questions.
The `--dangerously-skip-permissions` flag (a.k.a. "YOLO mode") does do the back-and-forth and asks questions, so this is a bit more than that.
Yes. I did not look but most probably the non interactive mode flag is used (-p)
It does `claude -p "This is the prompt" --dangerously-skip-permissions --output-format json`
Oh! TIL, thank you.
Iteratively working is a MUST for more than trivial fixes. This continuous loop could work for trivial refactorings / maintenance tasks.
Fyi