From plausible PCB artifacts to verified engineering work
Five end-to-end PCB tasks combine agent work traces, automated checks, two stages of expert review, and matched reference boards to reveal where complete-looking work fails engineering validation.
An agent can submit every expected file and report successful checks while serious engineering defects remain.
Running the same check on agent and expert boards helps separate real model failures from evaluator noise.
Logs and revisions show where the agent got stuck, introduced a new problem, or stopped before the board was ready.
One agent submission passed the initial gate. Four exposed a clear failure mode.
Open a task to see the decision, the supporting evidence, and how the agent submission compares with the matched expert reference.
Failed copper-geometry check
The model checked syntax, outline, placement, endpoint coverage, and file presence, but it did not independently check copper geometry. The evaluator found 27 cross-net copper crossings; the expert reference had zero.
Cross-net copper crossing: two routed copper segments from different electrical nets meet on the same board layer, creating a likely short. This is a focused geometry signal, not a full KiCad DRC score.
This data shows where an engineering workflow breaks—not just whether it passed.
Each record connects the task intent, the agent’s full work product, its repair trajectory, independent checks, and an expert reference solution. Researchers can trace a failure to the point where the workflow went off course.
Every decision changes the constraints at the next step.
Moving one component can change trace paths, current loops, heat, noise, and the final verification result.
Inputs, outputs, ripple, limits
Topology and key parts
Schematic and BOM
Placement, routing, current and thermal paths
Rules, performance, and manufacturing files
One task record connects intent, execution, evidence, and reference.
Task Explorer contains the full specifications and source files. This card summarizes the research signals available in every record.
The same records support measurement, comparison, improvement, and training.
Verify task completion
Measure whether the full artifact is discoverable, internally consistent, and supported by independent evidence—not merely present.
Test agent setups
Compare self-checks, native-tool access, whole-board acceptance tests, and evidence requirements to identify what improves performance.
Strengthen repair behavior
Study checkpoints, regressions, rollback, and which intermediate handoffs predict a successful final artifact.
Create verified repair examples
At scale, pair expert diagnosis with independently verified corrections to create contrastive trajectories for post-training.
Pilot: identifies failure mechanisms and tests whether the data carries useful signal. Expanded evaluation: estimates frequency and compares interventions. Training program: produces verified repair and process-supervision data across broader task families.
A completed board is not the same as a trustworthy engineering result.
The model receives a realistic engineering brief and reference material. The evaluator then asks whether the model solved the right problem, whether the work holds together, and whether the result can be trusted.
A design brief, the relevant datasheets, and a clean workspace.
Each layer answers a different question about whether the work is usable.
- Required project files and expected output paths
- Parseable KiCad structure and schematic-to-board agreement
- Same-layer copper crossings between different electrical nets
- Repair behavior and comparison with expert reference boards
Automated checks plus two stages of expert review.
Automated evidence made repeatable issues visible. Human review checked the engineering judgment that file-level tools cannot fully assess.
- 01Automated checks
Required files, parseability, schematic-to-board agreement, and copper geometry.
- 02Stage 1 expert review
An electrical-engineering expert reviewed the task requirements, artifacts, calculations, and observed defects.
- 03Stage 2 expert review
A second expert checked the technical judgment, expert reference solution, and final task disposition.
Three questions organize the full rubric.
- Did it solve the right problem?Trace the written requirements into design choices and required outputs.
- Did the work hold together?Check the handoffs across parts, schematic, board, routing, and manufacturing files.
- Can we trust it?Use independent, calibrated evidence and state clearly what remains unmeasured.
Required project files, output paths, parseable KiCad files, and a complete submission package.
Reference designators, net names, footprints, and schematic-to-board parity.
Copper crossings between different nets, plus a fixed-clearance diagnostic that is not used as a final grade.
Self-check claims, independent check results, repair regressions, stopping behavior, and final evidence.
Topology, component sizing, protection, layout quality, thermal choices, and use of datasheet constraints.
Native DRC/ERC, simulation, fabrication review, and hardware measurements.
Each layer supports a stronger claim than the one before it.
- 01Files presentMeasured
- 02Schematic ↔ boardMeasured
- 03Copper geometryMeasured
- 04Failures diagnosedMeasured
- 05Native KiCadPending
- 06Physical performanceNot measured
Six failure modes across the agent and evaluator stack.
Each row separates the observed evidence, the mechanism behind it, why the current workflow missed it, and the research intervention it motivates. The first four are agent failures; the last two are evaluator failures.
| Layer | Stage | Failure and impact | Evidence from the pilot | Mechanism and blind spot | Research opportunity |
|---|---|---|---|---|---|
| Agent | Submission | Required project file missingBlocks evaluation | Tasks 3 and 5 omitted .kicad_pro even though other KiCad and manufacturing files were present. | The agent treated downstream files as proof of completion, but the submission package was not discoverable as a complete project. Why it was missedThere was no evaluator-aware package manifest or immutable preflight at the start and end of the run. | Train package-contract checking and block submission until every required file, path, and project reference passes preflight. |
| Agent | Board geometry | Copper paths crossedLikely electrical short | Tasks 3, 4, and 5 had 32 cross-net copper crossings; the five expert references had zero. | Local routing actions reached endpoints without preserving net separation across the board, so geometric completion masked a likely short. Why it was missedThe agent checked file structure and endpoint coverage, not net-aware same-layer copper geometry or native DRC closure. | Put native DRC and an independent geometry gate inside the loop; collect hard-negative boards for verifier training. |
| Agent | Repair loop | Repairs did not settleNon-convergent workflow | Task 2 oscillated across route, pad, and via conflicts; the final cycle still reported 167 via conflicts and 30 spacing alerts. | Each local edit changed coupled board constraints and could create a new defect elsewhere, so the global defect set did not fall reliably. Why it was missedThe agent lacked a monotonic whole-board objective, checkpoint comparison, repair budget, and rollback rule. | Capture before-and-after repair states and train convergence, regression detection, checkpoint selection, and rollback behavior. |
| Agent | Self-verification | Self-checks missed the main defectFalse completion claim | Task 4 reported 192 passed checks while missing 27 copper crossings. | The agent optimized for abundant, easy-to-run proxy checks and treated their count as evidence that the engineering work was finished. Why it was missedNo coverage map connected the task’s major risks to the checks being run, and no independent verifier challenged the self-report. | Evaluate check coverage, not check count; require independent gates and adversarial known-bad boards before accepting completion. |
| Evaluator | Parsing | The grader misread valid file variantsFalse evaluator signal | Numeric and named KiCad nets initially produced incomparable static results. | A parser assumption converted valid serialization differences into apparent engineering differences until the implementation was corrected. Why it was missedThe evaluator had not been tested for invariance across valid KiCad file variants and versions. | Version graders and run a conformance suite of expert, known-bad, and format-variant boards before scoring model work. |
| Evaluator | Coverage | Automated checks stop short of circuit performancePerformance remains unknown | Registered automation covers DRC, ERC, and schematic parity only. | Structural consistency and CAD-rule compliance do not prove transient response, protection behavior, thermal safety, or hardware function. Why it was missedThe current toolchain has no simulation, fabrication review, or bench instrumentation, so those claims are outside the measured boundary. | Build an evaluation ladder from package checks to native tools, simulation, expert review, manufacturing review, and bench evidence. |
The failures are not random; they cluster around handoffs, coverage, and control.
Completion breaks between stages.
The agent can produce valid-looking pieces without preserving the contract from task brief to project package, schematic, board, and final evidence.
Data opportunityLabel intermediate handoffs and train acceptance checks at every boundary.
Check count is not check coverage.
Task 04 passed 192 self-checks because the checks emphasized file and structure properties while missing a high-risk geometry condition.
Data opportunityMap each requirement and failure risk to an independent check, expert judgment, or explicit unmeasured claim.
The evaluator is part of the system under test.
A grader can create false confidence or false failure when its parser, calibration set, or capability boundary is wrong.
Data opportunityVersion evaluators, test invariances, and calibrate every release on expert and known-bad artifacts.
The agent’s own checks said the board was clean. The independent check disagreed.
This is the most useful research case because the agent did not simply make an error—it produced evidence that made the error look resolved.
Syntax, board outline, placement, file termination, endpoint coverage, drill bounds, and manufacturing-file presence.
Including VIN_PROT ↔ UVLO, ISENSE ↔ GATE, CS ↔ VCC, and CS ↔ UVLO on F.Cu.
Research implicationTrain and evaluate the path from requirements to verified completion. The target is not more self-checks; it is evidence coverage that matches the engineering risks.
Five steps turn this diagnostic signal into a production-grade evaluation.
The pilot identifies useful failure modes. Production requires broader coverage, repeatable scoring, stronger engineering evidence, and operational controls.
Scale and calibrate the native evaluator
- Why it is needed
- Five tasks identify failure modes, not how often they occur. Production needs broader task coverage, repeated runs, native KiCad scoring, and graders tested on expert reference and known-bad boards.
- Evidence to produce
- Stable metrics across task families, repeated runs, and model versions.
Put KiCad checks inside the agent loop
- Why it is needed
- The agent needs project preflight, DRC, ERC, board agreement, project save, and native export before it can claim completion.
- Evidence to produce
- Fewer missing files and hidden geometry or electrical defects at submission.
Control repairs at the whole-board level
- Why it is needed
- Local fixes can create new problems elsewhere. The agent needs checkpoints, a repair budget, regression detection, and rollback.
- Evidence to produce
- Repairs that reduce the full defect set, plus verified before-and-after trajectories for training.
Add expert review and engineering tests
- Why it is needed
- CAD checks cannot judge topology, component ratings, isolation, layout quality, ripple, transient response, efficiency, thermal rise, or protection behavior.
- Evidence to produce
- Expert scores and simulation or stress-test results tied directly to the written requirements.
Validate manufacturing and hardware
- Why it is needed
- Complete-looking BOM, Gerber, drill, and pick-and-place files do not prove supplier readiness or real circuit behavior.
- Evidence to produce
- A manufacturing review followed by bench evidence from a fabricated board.
Covered in this pilot: package completeness, file parseability, schematic-board reference agreement, copper-crossing evidence, repair behavior, and expert-reference calibration. Needed for production: broader task coverage, repeated runs, native KiCad scoring, functional circuit evidence, expert QA, and manufacturing-readiness validation.
Turing combines specialist electrical-engineering expertise, structured task production, automated verification, and two-stage expert review to produce evaluation data beyond this pilot.
Next decision: define task coverage, model conditions, required evidence, and acceptance criteria for a production dataset.