Earlier this year we wrote about what AI-generated code means for QA teams. The short version was that more code, written faster, by a tool that does not understand your domain, means more testing, not less. Most of the teams we speak to have accepted that argument. Many of them have acted on it. Their coverage numbers are higher than they have ever been.
We want to talk about why that number is now worth less than it was.
In a growing number of FinTech teams, the tests are being written by the same assistant that wrote the code, in the same session, from the same prompt, with the same gaps in understanding. The developer asks for a feature. The tool writes the feature. The developer asks for tests. The tool writes tests that confirm what it just built. Everything passes. Coverage goes up. A reviewer sees green and approves.
Nothing in that sequence verified anything.
The independence problem
Independent testing has one foundational idea behind it. The person checking the work should not be the person who did the work. Not because developers are careless, but because you cannot find the flaw in your own understanding by consulting your own understanding. If you believed the settlement date logic was correct when you wrote it, you will believe it is correct when you test it.
For decades the QA profession has been built on that separation. A tester reads the requirement, forms their own view of what correct looks like, and then checks whether the software matches it. The value is in the second, independent reading.
AI test generation quietly removes it. When a model writes tests from the implementation, it is not reading the requirement. It is reading the code. It produces tests that describe what the code does, not what the code should do. If the code has misunderstood the requirement, the tests inherit the misunderstanding perfectly. They will pass. They will keep passing. They will pass right up to the moment a customer's payment is settled on the wrong day.
This is not a hypothetical. The pattern is showing up in the industry data that has accumulated through 2026.
What the numbers say
The headline research this year has been about AI-generated code rather than AI-generated tests, but it tells you where the tests are coming from.
CodeRabbit's analysis of 470 open-source pull requests found that AI co-authored code carried around 1.7 times more major issues than human-written code, with logic errors up sharply. Veracode's finding that roughly 45 percent of AI-generated code samples contained a vulnerability from the OWASP Top 10 has been repeated so often it has become received wisdom. METR's controlled trial found that experienced developers working on real, complex codebases were actually 19 percent slower with AI assistance, largely because of the time spent checking and correcting output.
Capgemini's World Quality Report for 2025 to 2026 found that 89 percent of organisations are piloting or deploying generative AI in quality engineering. Only 15 percent have it working at enterprise scale. Sembi's 2026 quality pulse found that only 17 percent of teams say AI-driven testing has delivered significant gains.
So the picture is this. AI is writing a large share of new code. That code has more defects than human code. The tooling that is supposed to catch those defects is being adopted everywhere and working well almost nowhere. And the tests themselves, in many teams, are being generated by the same models that are producing the defects.
The most telling data point is the one that is hardest to verify. Several practitioners this year have reported that AI-generated test suites score between 40 and 55 percent when subjected to mutation testing. We treat the exact figure with caution because it comes from small samples. But the direction matches what we see in client codebases. Roughly half of the generated tests would still pass if you broke the code they claim to test.
What a lying test looks like
It helps to be concrete. These are the patterns we find most often when we review AI-generated test suites in FinTech codebases.
The mirror test. The test asserts that the function returns what the function returns. It was generated by reading the implementation, so the expected value in the assertion is whatever the code currently produces. If the code calculates a fee incorrectly, the test enshrines the incorrect fee as correct.
The mock that tests itself. The test mocks the dependency, calls the function, and asserts that the mock was called. Nothing about the real behaviour is exercised. These tests are especially common around payment gateways, ledger services and anything with an external API, which is exactly where the risk lives.
The tautology. The test checks that a value is not null, or that an array has a length, or that no exception was thrown. It will pass for almost any implementation, including a wrong one.
The happy path only. Generated tests overwhelmingly test the main flow with clean inputs. The boundary conditions that matter in financial software, the zero amount, the negative balance, the currency with no minor units, the leap year, the timezone crossing at midnight, are absent because the model was never asked about them and the code never hinted at them.
The hallucinated fixture. The test constructs data that cannot exist in the real system, a transaction with no account, a customer with no jurisdiction, and tests the function against it. The test passes. It has told you nothing about production.
Every one of these increases the coverage percentage. None of them increases confidence, and the dangerous part is that the reviewer looking at a pull request cannot easily tell the difference. A test called "should calculate settlement date correctly" looks like a real test. You have to read the assertion to know it is a mirror.
Coverage was always a weak proxy
We wrote about this earlier in the year from a different direction. Coverage measures which lines were executed during the test run. It has never measured whether anything was checked. A test that executes a line and asserts nothing counts the same as a test that executes the line and verifies every edge case.
That weakness was tolerable when tests were written by people, because a human writing a test usually had an intent behind it. Generated tests have no intent. They have a prompt. So the proxy that was already weak has now been decoupled from the thing it was standing in for, and the number on your dashboard is measuring how thoroughly the AI described your code, not how thoroughly anyone tested it.
If your board or your regulator is being shown a coverage figure as evidence of control, that is worth a conversation.
What to do instead
None of this is an argument against using AI to write tests. It is an argument against trusting tests you have not verified, which was always the argument. The practical response has five parts.
1. Separate who writes the code from who writes the tests. This is the old rule, applied to a new situation. If a developer uses an assistant to build a feature, the tests for that feature should be generated from the requirement, in a separate context, ideally by a different person, and certainly not by pasting the implementation in and asking for coverage. In our engagements we increasingly write the test specification before the code exists and hold the generated tests against it. That is shift-left with the independence put back in.
2. Run mutation testing on anything that moves money. Mutation testing changes the code deliberately, flips a comparison, removes a condition, alters a constant, and checks whether the tests notice. It is the only automated way to find out whether a test suite is actually checking anything. The tools are mature, the run time is manageable if you scope it to the risk-critical modules, and the results are usually sobering. Feed the surviving mutants back as failing cases and the generated tests improve quickly. This should be a release gate for payment, ledger and pricing code.
3. Review assertions, not test names. Train reviewers to read what a test asserts, not what it is called. A fifteen second check for mirror tests, mock-only tests and tautologies in every pull request catches most of the damage. Make it a checklist item. It is cheaper than any tool.
4. Report verified behaviour, not coverage. Replace the coverage figure on your quality dashboard with something that measures checking. Mutation score on critical modules. Count of boundary conditions with explicit tests. Defect escape rate by area. These are the metrics a regulator should be shown, and the metrics a CTO should be asking for.
5. Keep a human gate on the flows that cannot fail. For anything that touches customer money, customer data or a regulatory report, an independent tester with domain knowledge should still be forming their own view of what correct looks like and checking against it. Not because the tools are bad, but because that independent view is the entire point, and no tool that was given the implementation can provide it.
The honest conclusion
The teams in trouble are not the ones using AI to write tests. They are the ones who stopped asking what a test is for. A test exists to catch the gap between what you built and what you meant. When the same model writes both, there is no gap to catch, and the green build is reporting on a conversation the model had with itself.
If your coverage has gone up this year and your production defect rate has not gone down, that is the pattern. It is fixable, and the fix is mostly discipline rather than tooling. But it starts with accepting that the number you are looking at no longer means what it used to.
If you want an independent view of whether your test suite is actually checking anything, that is the kind of work our QA Advisory engagements exist for. And if the honest answer is that nobody independent is testing your critical flows, Testing as a Service puts that back without you having to build a team to do it.