I’m eager to test this out. I have agent instructions to try to limit the worst of this already, but patterns still sneak through. I have a review agent run after every single edit looking for all of the following if you need more ideas for checks:
- DRY principle violations, multiple definitions of the same helpers or utilities.
- Changes that deviate from existing patterns and architecture already in the code, especially in nearby and related code
- Comments that add no context or simply restate the field name.
- Naming violations (enterprise factoryfactoryabstraction stuff, excessively long names, overly technical names, banned words like “seam”, “durable”, and no-value-qualifiers like “SaveGame” -> “Save”).
- Tests that check implementations instead of correct business behavior.
- Overly backwards-compatible unless asked for (this one is incredibly hard to keep under control, as AI loves to guard everything even if the previous code was never deployed and thus there is no contract break)
- Un-necessary guard code (this is hard to control, most common case is the AI not relying on the serializer error handler and instead adding guards that the library already handles)
- Changing public API contracts without express permission to do so (depends on the code, eg a library JAR or versioned REST service)
- Meta references to previous code versions, to tasks or todos, or to instructions and other non-code context (e.g you tell the AI the adder should ignore negative numbers and that meta fact enters the comments or code)
I usually hand review all changes myself but it’s incredibly tedious so I try to first pass with the review agent until it comes back clean. I hate wasting tokens on it though.
Oh my gosh, the guard code drives me crazy. In try so hard to get Kimi to put in asserts instead of silently swallowing corrupt values, but it keeps handling bad values poorly instead of crashing. I've even explicitly put in CLAUDE.md that correctness is more important then continuing to run, but it still keeps defensively programming when it should loudly crash.
A linter with rules for AI-specific weirdness is absolutely a great idea, thank you! Are there any plans to support other languages besides javascript?
I was about to write what advantage it has over linters but then saw the built on section. Good work. We use megalinter with our flavour of go and vite rules, plus extensive e2e testing after each agent run. Quality of the spec driven agentic PRs are significantly better than the baseline. Megalinter is quite resource heavy and slow, so will definitely check this out
This is a great idea. Even if you're one of those developers squarely focused on getting the final result working, code quality still matters (to people and LLMs).
Everyone should be doing regular code reviews and this helps a lot.
I don’t see if this is one of the covered cases, but one of the more common and nefarious patterns I run into is what you might call "sweeping exceptions under the rug." I think the agent’s motivation to get things running encourages these antipatterns of designing routines that are fault tolerant in a sort of maladaptive way: e.g. catching an error, logging a warning that something didn’t work, and continuing, but with now potentially missing/broken state.
This has bitten me a couple of times, and it’s surprisingly annoying to nudge agents into good/resilient patterns or identify situations that should fail loudly, at least in my experience. The retry mechanisms they come up with on their own are often pretty terrible as well.
I’ll note, though, that I have seen this from human engineers plenty of times, and at least the AI usually adds some logs rather than just totally silently absorbing an exception!
Gave it a try but there were a lot of false positives. SQLModel's exec method for example gets flagged every time thinking it's python's exec() function.
I think a lot of the telltale signs of AI can be found in the comments. Besides the slop writing style, I've found AI comments to 1. be overly verbose, 2. unnecessarily describe before/after code state (# This function used to do foo, but now it does bar), and 3. reference its own internal "plan" (# This function is part of Stage 3 of the implementation of Use Case X from the requirements doc) WTF is Stage 3? - says code reader 2 years from now. Although I bet you can probably prompt these behaviors away.
I tried it on my codebase. There's a lot of overlap with tools like Oxlint / ESLint, I'm not sure that's too valuable vs. a more focused tool that actually focuses solely on 'slop' signals. These lint rules tend to get very opinionated which is why those tools expose so many configuration options.
One real bug tho:
> [ERROR] Imports "mdast" but it's not declared in package.json
A type-only import like `import type { Blockquote, RootContent } from 'mdast';` is actually acceptable if `"@types/mdast": "^4.0.4"` is included in the package.json.
Apparently https://github.com/scanaislop/aislop with AGENTS.md is not AI slop, but projects of mine last modified before 2016 all contain critical AI slop. So very accurate.
One funny thing I see it doing is deleting seemingly random comments lines, for example if a file has a comment that spans multiple lines but doesn’t use a multi line comment syntax. It just chooses one at random transforming the once useful comment into slop.
This is a super cool paper! The only thing I'm a bit skeptical of is the detection of 'slop' to begin with - it would be very exciting to see some sort of way to steer a model more towards human-like output more generally. Here I imagine a bit of a challenge is that you're 1) creating the measurement for 'slop' and then 2) reducing it, so yes, you can have amazing results in the detection of the slop you've found (the methodology may be novel - a slight improvement over DPO), but this is a chicken and egg problem, as each new model would suffer differently based on training, RLHF, etc. It's not a very useful method to improve an LLM's writing quality, but rather just reducing the symptoms of said bad writing.
I'm additionally concerned about removing the LLM's voice to begin with, as though an LLM may far overuse specific words and phrases, so too do individual authors. Reducing an LLM towards the average author would result in an 'average' voice, paradoxically unlike any specific author.
I am currently doing research on something very similar over the summer but more towards the detection as opposed to the generation side - I'd love to discuss this with someone in the field if you had a few minutes sometime!
The intent is good, but frankly, credibility is lost by using "slop" words imo, OP you might seriously want to re-evaluate who is the target market, probably users that leverage high automation 24/7, startups and so-on, they don't want to incorporate products that talks about the modern way (and somehow only way to compete) of development as "slop" imo because soon enough, it's not true anymore (it's already not true with the right tooling).
Yeah, to be honest, it really is a bit weird to incorporate a tool called slop into a corporate CICD flow. Anubis fixed a similar problem by creating a corp-safe version without the anime mascot.
Thanks, I actually concentrated on improving code quality, the patterns I flagged are poor design choices that humans wouldn’t write. Examples are duplicated functions doing same thing, dead or redundant codes etc. These builds up and degrade the codebase over time.
I'm interpreting this not as a "catch ai submissions gotcha" tool, but as a "last pass in review catch mistakes AI made that i may have missed" tool. Having more linters is a good thing IMO (I say this as someone who doesn't use AI to generate code, but works with people who do and has to review a lot of AI generated code)
I’m eager to test this out. I have agent instructions to try to limit the worst of this already, but patterns still sneak through. I have a review agent run after every single edit looking for all of the following if you need more ideas for checks:
- DRY principle violations, multiple definitions of the same helpers or utilities.
- Changes that deviate from existing patterns and architecture already in the code, especially in nearby and related code
- Comments that add no context or simply restate the field name.
- Naming violations (enterprise factoryfactoryabstraction stuff, excessively long names, overly technical names, banned words like “seam”, “durable”, and no-value-qualifiers like “SaveGame” -> “Save”).
- Tests that check implementations instead of correct business behavior.
- Overly backwards-compatible unless asked for (this one is incredibly hard to keep under control, as AI loves to guard everything even if the previous code was never deployed and thus there is no contract break)
- Un-necessary guard code (this is hard to control, most common case is the AI not relying on the serializer error handler and instead adding guards that the library already handles)
- Changing public API contracts without express permission to do so (depends on the code, eg a library JAR or versioned REST service)
- Meta references to previous code versions, to tasks or todos, or to instructions and other non-code context (e.g you tell the AI the adder should ignore negative numbers and that meta fact enters the comments or code)
I usually hand review all changes myself but it’s incredibly tedious so I try to first pass with the review agent until it comes back clean. I hate wasting tokens on it though.
Oh my gosh, the guard code drives me crazy. In try so hard to get Kimi to put in asserts instead of silently swallowing corrupt values, but it keeps handling bad values poorly instead of crashing. I've even explicitly put in CLAUDE.md that correctness is more important then continuing to run, but it still keeps defensively programming when it should loudly crash.
Petition to rename this “SlopCop”
Sold!
nice one
Apparently I need to check in with a Doctor because code written by myself is seen as AI, and the lazy AI bits aren't. More Human than Human?
Unfortunately, AI detection can never be assumed to be accurate for the same reason the AI itself can never be assumed to be accurate.
(Not that I think you didn't probably already know that.)
A linter with rules for AI-specific weirdness is absolutely a great idea, thank you! Are there any plans to support other languages besides javascript?
Thank you. I currently support up to 8 languages: php, go, rust, python, js and ts
I was about to write what advantage it has over linters but then saw the built on section. Good work. We use megalinter with our flavour of go and vite rules, plus extensive e2e testing after each agent run. Quality of the spec driven agentic PRs are significantly better than the baseline. Megalinter is quite resource heavy and slow, so will definitely check this out
Thank you, really appreciate. Feel free to create issue if you have any
This is a great idea. Even if you're one of those developers squarely focused on getting the final result working, code quality still matters (to people and LLMs).
Everyone should be doing regular code reviews and this helps a lot.
Thanks for the feedback. Really appreciate it
I don’t see if this is one of the covered cases, but one of the more common and nefarious patterns I run into is what you might call "sweeping exceptions under the rug." I think the agent’s motivation to get things running encourages these antipatterns of designing routines that are fault tolerant in a sort of maladaptive way: e.g. catching an error, logging a warning that something didn’t work, and continuing, but with now potentially missing/broken state.
This has bitten me a couple of times, and it’s surprisingly annoying to nudge agents into good/resilient patterns or identify situations that should fail loudly, at least in my experience. The retry mechanisms they come up with on their own are often pretty terrible as well.
I’ll note, though, that I have seen this from human engineers plenty of times, and at least the AI usually adds some logs rather than just totally silently absorbing an exception!
For anyone who wants something like this for Elixir, there is an open source hex package: https://hex.pm/packages/ex_slop
Gave it a try but there were a lot of false positives. SQLModel's exec method for example gets flagged every time thinking it's python's exec() function.
I think a lot of the telltale signs of AI can be found in the comments. Besides the slop writing style, I've found AI comments to 1. be overly verbose, 2. unnecessarily describe before/after code state (# This function used to do foo, but now it does bar), and 3. reference its own internal "plan" (# This function is part of Stage 3 of the implementation of Use Case X from the requirements doc) WTF is Stage 3? - says code reader 2 years from now. Although I bet you can probably prompt these behaviors away.
> I’ve been building aislop.
This made me chuckle.
Cool tool, the dead code checks can be very helpful. I’ve seen Claude leave unused functions when iterating which always frustrates me.
haha. I love the name - aislop
Can you explain the benefits of running this over rector / eslint? (and to certain degree phpstan / deptrac)
Ran it in my codebase, detected some good stuff, was able to pass the issues to my agent so they can be validated and fixed. Good job!
I'd love for it to have flags for Pi and Crush too :)
Thank you, really appreciate.
I tried it on my codebase. There's a lot of overlap with tools like Oxlint / ESLint, I'm not sure that's too valuable vs. a more focused tool that actually focuses solely on 'slop' signals. These lint rules tend to get very opinionated which is why those tools expose so many configuration options.
One real bug tho:
> [ERROR] Imports "mdast" but it's not declared in package.json
A type-only import like `import type { Blockquote, RootContent } from 'mdast';` is actually acceptable if `"@types/mdast": "^4.0.4"` is included in the package.json.
thanks for this catch, really appreciate. Can you create an issue on GitHub so you can easily track when I resolve this?
Apparently https://github.com/scanaislop/aislop with AGENTS.md is not AI slop, but projects of mine last modified before 2016 all contain critical AI slop. So very accurate.
I tried it but see a lot of false positives.
One funny thing I see it doing is deleting seemingly random comments lines, for example if a file has a comment that spans multiple lines but doesn’t use a multi line comment syntax. It just chooses one at random transforming the once useful comment into slop.
Thanks for the feedback, we’ll check that out
awesome work, someone is caring about using less token :clap:
Related, we made "Antislop", a framework for removing repetitive phraseology from LLMs: https://arxiv.org/abs/2510.15061
This is a super cool paper! The only thing I'm a bit skeptical of is the detection of 'slop' to begin with - it would be very exciting to see some sort of way to steer a model more towards human-like output more generally. Here I imagine a bit of a challenge is that you're 1) creating the measurement for 'slop' and then 2) reducing it, so yes, you can have amazing results in the detection of the slop you've found (the methodology may be novel - a slight improvement over DPO), but this is a chicken and egg problem, as each new model would suffer differently based on training, RLHF, etc. It's not a very useful method to improve an LLM's writing quality, but rather just reducing the symptoms of said bad writing.
I'm additionally concerned about removing the LLM's voice to begin with, as though an LLM may far overuse specific words and phrases, so too do individual authors. Reducing an LLM towards the average author would result in an 'average' voice, paradoxically unlike any specific author.
I am currently doing research on something very similar over the summer but more towards the detection as opposed to the generation side - I'd love to discuss this with someone in the field if you had a few minutes sometime!
The intent is good, but frankly, credibility is lost by using "slop" words imo, OP you might seriously want to re-evaluate who is the target market, probably users that leverage high automation 24/7, startups and so-on, they don't want to incorporate products that talks about the modern way (and somehow only way to compete) of development as "slop" imo because soon enough, it's not true anymore (it's already not true with the right tooling).
Imagine an operating system company making a product called “quick and dirty”.
Yeah, to be honest, it really is a bit weird to incorporate a tool called slop into a corporate CICD flow. Anubis fixed a similar problem by creating a corp-safe version without the anime mascot.
I don't think this approach is wise.
Concentrate on code quality, and whether it does what it needs to do. Not whether it was written by AI or not.
Thanks, I actually concentrated on improving code quality, the patterns I flagged are poor design choices that humans wouldn’t write. Examples are duplicated functions doing same thing, dead or redundant codes etc. These builds up and degrade the codebase over time.
I'm interpreting this not as a "catch ai submissions gotcha" tool, but as a "last pass in review catch mistakes AI made that i may have missed" tool. Having more linters is a good thing IMO (I say this as someone who doesn't use AI to generate code, but works with people who do and has to review a lot of AI generated code)