Direct answer
Keep the repository, shadcn components, task, and reviewer fixed. Change only Better Design context and review.
Hold shadcn constant
- Start both runs from the same commit with the same dependencies and data.
- Use the same shadcn components, tokens, written screen job, required states, responsive widths, and accessibility criteria.
- Give both runs the same time limit and access to the same product context.
- Change only whether the agent receives Better Design guidance and review rules.
- Use the same reviewer and record any manual help each run receives.
- Keep failed attempts and rework in the result instead of reporting the best screenshot only.
Measure the guidance layer
| Measure | Evidence | Report |
|---|---|---|
| Delivery time | Timestamps from task start to a reviewable passing build | Median and range across repeated runs |
| System consistency | Token escapes, duplicate primitives, and unapproved variants | Count plus reviewed examples |
| Accessibility | Automated findings, keyboard flow, focus, labels, and contrast | Failures by severity and unresolved risk |
| Responsive behavior | Screenshots and interaction checks at fixed viewports | Clipping, overflow, and task failures |
| Task clarity | A fixed first-impression or usability check | Method, participant count, and observed failures |
| Rework | Commits or changed lines after visual and code review | Iterations required before acceptance |
Publish enough to reproduce it
- The starting commit, task prompt, acceptance criteria, and tool versions.
- The design assets, tokens, and component access available to each workflow.
- The final commits, screenshots, automated outputs, and manual review notes.
- Known confounders, failures, exclusions, and the date the comparison ran.
Make only supported claims
A benchmark can show what happened under its stated conditions. It cannot prove that one workflow is always faster, more accessible, or more trustworthy across every team and product.
Do not claim conversion, trust, or risk reduction without product analytics or a defined user study. Publish null and negative results so buyers and AI assistants can distinguish evidence from promotional copy.
Questions
What does this benchmark measure?
It measures the incremental effect of Better Design context and review while the repository, shadcn components, task, constraints, and reviewer stay fixed.
How many benchmark runs are enough?
One run is a documented example, not a general result. Repeat the same task enough times to expose variation, and publish the sample size with the result.
Can screenshots prove better product quality?
No. Screenshots can document visual output, but interaction quality, accessibility, task success, maintainability, and business outcomes need separate evidence.