AI made engineering velocity a lot easier to fake. A team can generate more code, open more pull requests, accept more AI suggestions, and ship more often while quality quietly degrades somewhere else.
In review. In rework. In production incidents. In security exposure.
Over the last several months, I’ve spoken with leaders across virtual and in-person workshops, including engineering leaders inside large enterprise organizations.
The mandate coming from their bosses usually sounds familiar: improve developer velocity, integrate AI into engineering workflows, and help the organization direct software outcomes more effectively through its engineering teams.
Many teams are reaching for some version of software factory status. Repeatable delivery. Better leverage. More predictable execution. A clearer path from idea to shipped system. And aspiration makes sense, but there is a clear issue with visibility into agentic process that would even allow them to make a claim about what’s working and what isn’t. At least for the average engineering leader where their org isn’t extremely mature with AI adoption.
Leaders can often see that AI is being used. They see that developers are moving faster. But they can’t always see whether or not their engineering processes and outcomes are trustworthy when AI is embedded.
This came up in a conversation I had with Nicky Pike, Field CTO at Coder, about how AI will change the engineering development lifecycle and the metrics we use to understand software quality.
AI changes how software is produced and leadership dashboards have to change with it.
Velocity Became Easier To Manufacture
Traditional velocity signals are still useful, but they are easier to misread now.
AI can raise:
pull request volume
code generated per developer
AI suggestion acceptance
deployment frequency
commit frequency
feature throughput
Those signals may represent real improvement, but they could also represent local acceleration inside a system that is absorbing the cost later.
That is the mode leaders need to watch: when visible output improves while invisible quality debt compounds.
So I reviewed a 103-response forum for engineering leaders that asked which KPI development teams should watch more closely as AI coding tools speed up delivery.
Here were the patterns I extracted:
change failure rate
defect escape rate
rework rate
code comprehension debt
MTTD and MTTR
security vulnerability density
review quality
technical debt growth
token and cost efficiency
lead time for changes
That matches what I am seeing in the field.
Hard engineering responsibilities still remain significant: reviewing the code, securing it, understanding it, operating it, maintaining it, and defending the decisions behind it.
Agentic Engineering Velocity Metrics
If I were advising an engineering leader rolling out AI-assisted development, I would start with velocity metrics listed below.
Use it to ask whether AI is creating durable leverage or moving cost downstream.
Metric | What It Reveals | Failure Signal |
|---|---|---|
Change failure rate | Whether faster delivery is destabilizing production | Deploys increase alongside incidents, rollbacks, or hotfixes |
Defect escape rate | Whether bugs are reaching customers or production systems | More defects are found after release instead of during review, testing, or validation |
Rework rate / 30-day churn | Whether AI-assisted work was durable | Code is accepted quickly, then rewritten, reverted, patched, or discarded soon after |
Code comprehension debt | Whether engineers can still explain and safely modify what ships | The team merges code faster than it can understand it |
Review quality / thoroughness | Whether human judgment is keeping pace with AI output | Review time drops while escaped defects, security issues, or architectural drift rise |
MTTD / MTTR | Whether the organization can detect and recover as fast as it ships | Delivery speeds up, but detection and remediation stay flat or worsen |
Security vulnerability density | Whether AI-assisted changes are expanding risk | Generated code introduces more vulnerable patterns, dependencies, credentials, or unsafe defaults |
Technical debt growth | Whether the codebase is becoming easier or harder to change | Complexity, duplication, coupling, or architectural inconsistency increases over time |
Cost per production-ready feature | Whether AI spend becomes reliable business value | Token usage looks efficient, but rework, review, remediation, and failed deployments erase the gain |
Lead time for changes | Whether AI improves the whole delivery system | Coding speeds up while work still gets stuck in review, testing, security, release, or adoption |
Treat these metrics as control points. Each metric answers a different leadership question:
Is faster delivery reaching production safely?
Are defects moving earlier in the lifecycle or closer to customers?
Is AI-assisted code durable after the first merge?
Can engineers still explain what the system is doing?
Is review preserving judgment under higher volume?
Can the organization detect and recover when speed creates failure?
Is AI increasing the attack surface?
Is the codebase becoming easier to evolve?
Is AI spend tied to production-ready value?
Is the whole system faster, or only the coding step?
Bad Metric Correlations to Look Out For
Watch what happens when output metrics improve and software quality metrics decline.
For example:
PR volume rises, but defect escape rate rises too.
Deployment frequency improves, but change failure rate climbs.
AI suggestion acceptance is high, but 30-day churn increases.
Review time gets shorter, but architectural drift becomes more common.
Token cost per task drops, but cost per production-ready feature rises.
Lead time for coding improves, but lead time to safe production does not move.
That pattern means the system is not getting healthier.
A 30-Minute Diagnostic
Pick one AI-assisted workflow your team already uses.
Then answer:
Which velocity signal improved?
Which quality signal proves the improvement is safe?
Which review or comprehension signal shows that human judgment still scales?
Which recovery signal tells you failure will be detected quickly?
Which rework signal tells you the output was durable after merge?
If the team can answer those questions with evidence, AI adoption is becoming more governable. If the team can only point to output, the system is still in the experimentation stage.
That is fine. Experimentation has value! And it leaves more to learn about what’s truly working and what isn’t.
