We’re a team of eight, and over the past couple of months we’ve reached plan 660 in our own software factory.
Every change starts with a plan reviewed by another person. An agent then builds it, while automated reviewers check the code for bugs and compare it with what the team approved. Some changes merge without an engineer reading the full diff.
In Software Factories, Light and Dark, Addy Osmani calls those implementation loops dark: machines build the code, other machines verify it, and no person reads the output before it ships.
Across a sample of 234 reviewed plans, we sent 17% back for changes. That is roughly 2.8 times the rate at which we previously requested changes on pull requests. Yet we’ve seen 18–26% of the resulting pull requests receive no recorded human review on GitHub.
Human judgment still sits at the entrance to the factory. We keep the lights on while people decide what should be built, then let individual implementation loops operate with different levels of autonomy.
The question is what a loop must prove before we let it run with the lights off.
Our review gate was in the wrong place
We published AI Is Not Your Peer because code review had started to feel like a different job.
When a person writes code, the pull request contains choices they made. You can ask why they chose a particular boundary or what they expect to happen on failure. The author can explain themselves, disagree with you and occasionally change your mind.
An agent-generated diff is a messier artifact. It contains the decisions a person made, mixed with hundreds of mechanical choices the agent made on their behalf.
I experienced this from the slightly unusual position of a product manager opening code changes.
The comments engineers left on my pull requests were generally reasonable. They just concerned decisions I had never made and often did not care about: which internal pattern the agent chose, how it organized a package, or whether another library would have been more idiomatic.
The engineers were doing what code review asked of them. They read the diff and formed an opinion about the implementation. Meanwhile, the product behavior I cared about was buried somewhere inside it.
The timing made those conversations worse. A pull request opened in the morning might receive feedback the following day, when I was already deep in customer calls or another piece of work. A question that took five minutes to answer while the problem was fresh could cost half an hour once I had to reconstruct the whole thing.
We were spending our most expensive conversations on an artifact that fused human intent with machine implementation.
So we moved the conversation.
We moved peer review before the code
We started writing plans that contained the business context, the proposed behavior, the systems affected and the evidence that would show the work was complete.
The important implementation choices went in there too. If the team cared which library the agent used, how a migration worked or what happened during failure, that decision had to be visible in the plan. Anything left unspecified was something we were explicitly allowing the implementing agent to decide.
Another person then reviewed that plan.
This is where peer review now happens for us. A human who knows the system pushes on the approach and its consequences. They catch the cases the author missed. The author can defend the decision because it is still their decision.
Only after that review does an agent start implementing.
The change sounds almost administrative. We swapped one review artifact for another. What surprised us was how much the timing altered the cost of disagreement.
Feedback became roughly 22 times cheaper
We measured how much a round of feedback added to delivery.
During our pull-request-review era, work that received a change request took a median 15.9 hours longer to merge than work that did not. That comparison covered 13 pull requests with feedback and 194 without it.
During the plan-review era, a round of feedback added a median 0.7 hours. That sample contained 11 plans sent back and nine accepted without a feedback round, each linked to its resulting pull request.
This does not mean plans are 22 times faster to read. It means changing your mind is much cheaper before an agent has built the decision.
The samples are small, especially on the plan side. This is one team working in one repository. It is not a controlled experiment or a promise that another team will see the same result. The size of the difference still changed how I think about review.
When feedback was expensive, reviewers used it carefully. A significant objection on a finished pull request meant asking someone to unwind work that already existed.
Once the same objection cost less than an hour, people made it more often.
Across a sample of 202 reviewed pull requests, approximately 6% received a change request. Across 234 reviewed plans, 17% were sent back at least once.
We challenged work roughly 2.8 times more often after moving review upstream.
Thirteen of the first 165 reviewed plans were rejected and never revived. About 8% of proposed work produced no code at all.
Those thirteen plans are invisible if you measure engineering output through GitHub. I think they are among the most valuable things the process produced. Nobody had to maintain, understand or eventually remove code for an idea that did not survive its first serious conversation.
After the plan, we audit the machine
An approved plan gives the implementing agent a definition of what done means. It also gives the rest of the factory something concrete to verify.
The coding agent builds the change and works through CI. An automated code reviewer checks for likely defects. A separate plan-adherence check compares the implementation with the approved plan on every push.
That comparison looks for four kinds of disagreement:
- Something required by the plan is missing.
- The implementation has grown beyond the approved scope.
- A planned decision changed during implementation.
- The agent added a smaller adjacent change that the plan never mentioned.
Ordinary tests do not reliably answer these questions. A missing requirement leaves no failing line in the diff. An unrequested capability can pass every test written for it. You need to hold the intent next to the implementation to see either problem.
The check treats the implementation as hostile. A vague explanation such as “covered by integration and unit tests” is not enough to resolve a finding. The agent has to point to the specific evidence or fix the difference.
When the implementation matches the approved plan and all the other checks pass, it can merge without a person reading the complete diff. Under Addy’s definition, that implementation loop is dark.
When the implementation differs from the plan, the change stays blocked. The agent can fix the difference or explain why it departed from what the team approved. Accepting that departure requires the person who reviewed the plan to return and judge the affected code.
That person still does not read the entire output, so this does not turn the whole loop light. It puts human judgment back at the point where the machine has exceeded or changed the decision people made.
During one measured week, 12 agent-built pull requests merged. The median time from an approved plan to a pull request worth opening was around 14 minutes. Most agents needed four or five attempts to work through CI, with the longest taking nine.
The automated code reviewer commented on six of the 12 pull requests and produced 16 findings. Approximately half of the changes contained deviations that required human sign-off. Those reviewers generally considered two or three specific differences rather than reading the whole diff.
Every implementation starts from a decision another person reviewed. Human attention returns downstream when the implementation leaves the boundaries of that decision, without pretending that targeted review is the same as reading the full output.
A plan does not earn the dark by itself
A reviewed plan tells us what people agreed to build. It does not prove that the decision was good or that its implementation is safe.
The plan could be wrong, the reviewer could miss something. The code could follow the plan faithfully while introducing a subtle production bug. A machine reviewer could misunderstand both artifacts in the same convenient direction.
Long-term architecture is particularly hard to reduce to an immediate machine-verifiable answer. Addy calls the accumulated consequence comprehension debt: the widening gap between how much code exists and how much any human still understands.
Plan review helps with one part of this. Important decisions remain written down, reviewed and connected to the implementation. Rejected plans prevent unnecessary code from joining the system at all.
It does not give engineers detailed knowledge of every line an agent generated. Pretending otherwise would recreate the problem the phrase is trying to describe.
That is why Addy’s idea that a loop must “earn the dark” matters.
He argues that automated status is appropriate when the check is cheap, frequent, immediate, hard to fake and unlikely to drift. Short loops are easier to verify than long ones. A bounded lint fix has a much clearer oracle than a redesign of authentication or billing.
Our experience adds a few questions:
- Can people agree on a precise plan before implementation?
- Are the important constraints checked outside the coding model?
- Can the system detect when the implementation diverges from the plan?
- Does uncertainty block the change or quietly wave it through?
- What is the cost if every automated check passes and the result is still wrong?
A high-risk change may still need a person to inspect the code even when the adherence check reports a match. That policy should be based on the blast radius and the strength of the available evidence.
We have more work to do here. Risk-based review needs to become an explicit part of the factory instead of something teams apply from memory.
Output rose alongside scrutiny
Across three separate before-and-after windows, our team merged 1.8 times more pull requests after moving review to the plan.
Median open-to-merge time fell from 7.5 hours to 1.6 hours, a 4.7-fold improvement. The median pull request grew from 109 to 215 added lines, so the increase did not come from slicing the same work into smaller pieces.
CI failure rate fell from 22.8% to 8.6%. Revert rate stayed flat.
These results do not prove that plan review caused every improvement. We changed models, strengthened the factory and gained experience operating the workflow. The work going through it may also have changed.
Throughput rose while reviewers sent substantially more work back. Faster delivery did not require us to make fewer objections.
The factory has now handled hundreds of plans from a team of eight. It has moved beyond the original handful of experiments, but it is still one company’s experience. The numbers are evidence that the workflow can operate at a meaningful cadence, not a universal benchmark.
“But your engineers still aren’t reading all the code”
Correct.
That trade has consequences, and I do not think we have resolved all of them.
Code review used to catch mistakes, but it was also where engineers learned. Juniors improved by defending their work and reading choices made by more experienced people. Seniors had to explain habits that had quietly hardened into rules.
Reviewing an agent’s diff is a poor substitute. The agent does not learn from the conversation, and no person authored many of the choices being discussed.
Plan review gives some of that learning a new home. We have watched less experienced people catch gaps in plans written by senior engineers because the decision was still expressed in words they could question. Those conversations feel much closer to the whiteboard sessions where I learned the most.
It does not teach language fluency or the instincts that come from maintaining a system. It cannot show a junior what a race condition feels like before it turns into a postmortem. I do not know how the next generation absorbs all of that if agents increasingly perform the implementation and audit.
Keeping human judgment upstream recovers the argument about what should be built. It does not recover every kind of knowledge people once gained from writing and reviewing the code.
People still own the consequences
The machines in our workflow can implement a plan, work through CI, look for bugs and compare the result with what people approved.
People still decide whether the work should exist. We choose which details must appear in the plan, what evidence counts as verification and which systems are too expensive to trust to an unattended loop. When production proves us wrong, the consequences belong to us too.
Addy is right that the difficult job is deciding where each light switch belongs. Turning every light off creates code nobody understands. Leaving every light on turns finite human attention into the factory’s permanent bottleneck.
Our experience suggests that the first light should come on before the factory starts.
Review the plan while disagreement is cheap. Make the consequential decisions visible. Give the machines enough evidence to audit the implementation. Then decide which loops have actually earned the dark and where a person still needs to read what comes out before it ships.
People remain responsible for the factory. Our aim is to spend their attention while it can still change what gets built.