The Problem
When a human developer writes code, the feedback loop is rich and immediate. They read a stack trace, set a breakpoint, reason about state. They know when something is done because they can see it working. The entire software development lifecycle, from TDD to continuous delivery, is optimized for this: a human in the loop, interpreting feedback, making decisions.
Remove the human, and the loop collapses.
An autonomous agent tasked with implementing a feature faces a fundamental problem: it must both produce the work and determine whether the work is sufficient. If the same agent writes the code and evaluates the code, it can satisfy its own criteria trivially. It can write a test that passes by construction. It can declare a feature complete based on its own interpretation of "complete." There is no independent verification, no external standard, no honest signal.
This is the problem the Judge & Builder framework exists to solve. I cobbled scripts together to make it work and this post are my meditations on watching it run; watching a computer do real work all on its own.
The Dark Factory
The term "dark factory" comes from manufacturing. A facility that operates with the lights off, without human presence on the floor. Machines receive specifications, produce output, and verify quality autonomously. The human role shifts from operating the machinery to designing the specifications and auditing the output.
Judge & Builder applies this concept to software delivery. The factory has two machines:
The Builder receives a behavioral specification written in Gherkin: Given/When/Then steps describing what the software should do. It implements features, runs tests, and produces evidence: screenshots captured by Playwright at each verification point. The Builder never sees the scoring criteria. It knows what the software should do, but not how its work will be graded.
The Judge receives the same specification annotated with hidden rubric criteria: weighted scoring points tied to observable outcomes. It examines the visual evidence the Builder produced and scores each criterion against what it sees in the screenshots. The Judge never sees the implementation code. It evaluates outcomes, not approach.
Communication between them is indirect and asymmetric:
- The Builder reads sanitized scenarios (no weights, no scores, no rubric annotations)
- The Judge reads screenshots (no source code, no implementation details)
- Feedback flows from Judge to Builder as natural-language descriptions of observed symptoms, not prescriptive fixes
The loop repeats (build, capture evidence, evaluate, provide feedback) until the score converges to 100% or a stagnation threshold triggers escalation.
Why Separation Matters
The separation of Judge and Builder is not an organizational preference. It is a structural constraint that makes self-deception difficult.
Consider the alternative: a single agent that writes code, writes tests, runs tests, and evaluates results. At every stage, it has the context to shortcut. It can write an assertion that matches the output it already produced. It can interpret a vague requirement in whatever way makes its implementation correct. It can mark a criterion as "passed" based on its knowledge of what the code does rather than what the user sees.
The Judge & Builder split removes these shortcuts:
- The Builder cannot optimize for scores it cannot see
- The Judge cannot prescribe solutions based on code it cannot read
- The only shared language is observable behavior — what appears on screen
- Evidence is visual, not programmatic (a screenshot either shows the expected state or it doesn't)
This is not about trust or adversarial dynamics. It's about maintaining the integrity of evaluation when both evaluator and implementer are autonomous systems operating without human oversight.
Tension with Conventional SDLC
The Judge & Builder pattern makes deliberate tradeoffs that run against established software engineering practice. These are not oversights — they are consequences of optimizing for autonomous operation rather than human developer productivity. They deserve honest examination.
Fast Feedback
Dave Farley's central thesis, articulated in Continuous Delivery and across his body of work, is that the speed of feedback is the primary determinant of software quality. Shorter feedback loops mean faster learning, fewer defects, lower cost of change. The entire continuous delivery pipeline, from local unit tests to automated deployment, is designed to give developers the fastest possible signal about whether their change works.
The Judge & Builder loop is slow by this standard. Each iteration involves:
- A Docker image rebuild
- Container orchestration (app + database, etc.)
- Browser-based test execution with Playwright
- Screenshot capture at each verification point
- LLM-based evaluation of visual evidence against rubric criteria
- Natural-language feedback generation
A cycle that a human developer might close in seconds (e.g. read the error, fix the line, re-run) can take minutes per iteration. A real-world session required 17 iterations to diagnose a SameSite cookie misconfiguration. A human developer with access to browser DevTools and server logs would likely resolve this in one.
This is the most legitimate criticism of the pattern. The response is not that slow feedback is acceptable, but that the comparison is misframed. The Judge & Builder loop is not competing with a human developer's feedback loop. It is operating in a context where no human developer is present. The relevant comparison is not "how fast can a human fix this?" but "can this be fixed without a human at all?" Slow convergence can be expensive, but the alternative in a dark factory is not faster iteration, it's no iteration.
Test-Driven Development
Farley advocates writing tests before implementation. The failing test defines the requirement, constrains the design, and provides immediate feedback when the implementation satisfies it. TDD produces tight coupling between specification and verification——i.e. the test is the feedback mechanism.
Judge & Builder inverts this relationship. The specification (Gherkin scenario) exists before implementation, but the evaluation is decoupled from the test execution. The Builder runs tests that capture screenshots. The Judge evaluates screenshots against criteria the Builder has never seen. The feedback is not "this assertion failed on line 47" but "the navigation area does not show the expected state after form submission."
This indirection is intentional. In a TDD workflow, the developer benefits from precise, deterministic failure messages because they have the context to interpret them. An autonomous agent benefits from being unable to reverse-engineer the evaluation criteria, because that inability prevents it from satisfying the criteria through test manipulation rather than genuine implementation.
The cost is real: the Builder sometimes iterates on changes that a precise test failure would have resolved immediately. The benefit is also real: the Builder cannot game a system it cannot fully observe.
Transparency and Shared Understanding
Farley emphasizes making the state of the system visible to everyone. Dashboards, deployment pipelines, etc. all designed to ensure that every team member can see what is happening and why. Opacity is the enemy of effective collaboration.
The Judge & Builder pattern deliberately introduces opacity. The Builder does not see the rubric. The Judge does not see the code. Feedback is intentionally vague at first, escalating in specificity only when stagnation is detected.
The rationale is that transparency serves collaboration between agents who can be trusted not to game the system. Human developers benefit from seeing all the information because they are intrinsically motivated to build working software. An autonomous agent has no such motivation. It optimizes for whatever signal it receives. If the signal is "achieve a high score on these specific criteria," it will find the shortest path to that score, which may not be the same as building working software.
The opacity in Judge & Builder is not a failure of communication. Opacity is a mechanism for maintaining evaluation integrity in a system without intrinsic motivation.
Determinism
Continuous delivery pipelines are deterministic by design. The same code, the same tests, the same environment should produce the same result every time. Flaky tests are treated as serious problems because they undermine confidence in the feedback signal.
The Judge is not deterministic. It is a large language model evaluating visual evidence against rubric criteria. The same screenshots can produce different scores across evaluations. In observed sessions, the Judge varied its own criteria weighting between runs by changing the maximum possible score for the same rubric. This is fundamentally incompatible with the deterministic pipeline model.
This is an open problem in the framework, not a resolved design choice. The rubric YAML files are deterministic: they specify exact criteria and point values. But the Judge's interpretation of visual evidence against those criteria introduces variance. A screenshot that one evaluation deems "status badge visible" another might deem "status badge partially obscured." The framework mitigates this through explicit rubric criteria and structured evaluation output, but does not eliminate it.
What the Dark Factory Optimizes For
The conventional SDLC optimizes for a human in the loop: fast feedback, clear signals, transparent state, deterministic outcomes. These are the right optimizations when a human developer is the bottleneck and the decision-maker.
The dark factory optimizes for a different constraint: no human in the loop. In this context, the priorities shift:
- Evaluation integrity over feedback speed. It is more important that the evaluation is honest than development being fast
- Behavioral verification over implementation guidance. It matters what the user sees, not how the code is structured
- Convergence over efficiency. Reaching a correct outcome in 17 iterations is preferable to reaching an incorrect outcome in 1
- Structural separation over transparent collaboration. Preventing shortcuts matters more than shared context
These are not universally better priorities. They are appropriate priorities for a specific operating context: autonomous software delivery without human oversight.
The Human Role
In the dark factory, the human does not disappear. The role changes.
The human writes the specification: the Gherkin scenario that defines what the software should do. The human writes the rubric: the scoring criteria that define what "done" looks like. The human designs the seed data, configures the test infrastructure, and sets the stagnation thresholds that determine when the factory escalates for help.
The human is the factory designer, not the factory operator. The quality of the output depends on the quality of the specification. A vague scenario produces vague results. A rubric that scores the wrong things produces software that does the wrong things well.
This is perhaps the deepest alignment with Farley's philosophy, even if the mechanism differs: the most important work in software development is clearly defining what the software should do. The Judge & Builder framework simply takes that principle and asks what happens when the implementation is delegated entirely to machines.
Some Open Questions
The framework raises questions it does not yet fully answer:
- How do you handle non-visual requirements? Performance, security, data integrity, and accessibility are difficult to evaluate from screenshots alone. The current framework is strongest for UI-visible behavior.
- What is the right feedback granularity? The escalation model (vague feedback first, specific hypotheses after stagnation) is a heuristic. The optimal escalation curve is unknown.
- Can evaluation be made deterministic? Structured rubrics constrain the Judge, but LLM-based visual evaluation introduces inherent variance. Whether this can be reduced to acceptable levels without losing the flexibility of natural-language evaluation is an open research question.
- What is the cost ceiling? Each iteration consumes LLM tokens, compute time, and container resources. At what point does the cost of autonomous convergence exceed the cost of human intervention?
These are not reasons to reject the pattern. They are the boundaries of its current applicability and the directions for its evolution.