Shell page. Structure, masthead and at-a-glance are final. Body copy is in the markdown draft; screens and the items below are still needed. Delete this block when the page is written.
- Whether the automation step got implemented, or was specified and handed off
- Whether anyone has used it — even “too early to tell” is usable
- Visuals: verdict taxonomy redraw, BAU decision tree, intake flow, device-share diagram
- CRITICAL: nothing from the real document can be shown or linked. Say “our tracking,” never “their program.”
Design shipped. Results landed somewhere else. Nothing came back.
Design shipped. Experiments ran. Results went to analytics. Then nothing — design didn't hold the data and wasn't in the room when it was read, so results never returned as design decisions.
This is the ordinary condition of a lot of design teams and it doesn't feel like a crisis, because the work keeps going out. What it costs is compounding and invisible: you can't tell a proven pattern from an untested one, so you either over-trust your history or ignore it entirely.
Distinguishing “we failed to answer this” from “we never asked”
The decision the whole system turns on. An initiative with no usable result is in one of two completely different states, and in a spreadsheet both look like an empty cell.
| Verdict | Means | Next action |
|---|---|---|
| Validated | Tested live and won | Reuse it. Check the entry for conditions. |
| Mixed / Conditional | Won under some conditions, lost under others | Reuse only where conditions match. |
| Rejected as tested | Tested live and lost | A modified version is a new hypothesis. |
| Inconclusive | A test happened; result unusable | Open question. Often recoverable. Not a negative result. |
| Untested | No live test | If research exists it's already paid for — a shipping problem, not a research problem. |
Keeping them apart is what stops this from becoming a highlights reel.
Which was the actual risk. A catalog of design work authored by the designer who did the work has an obvious failure mode — so my own rejected result and my own mixed results are in there under those labels.
Half the portfolio had never been validated
Of eighteen initiatives: three validated live, two conditional, one tested and rejected, three inconclusive, and nine never tested at all.
Six of those nine had finished research and finished design. Studies run, findings in hand, designs complete, no live test. None were blocked on anything a designer needed to do.
It looked like a research-coverage problem. It was a shipping-and-follow-through problem, invisible because no single view had ever put those nine next to each other.
Which denominator the readout leads with is a design decision
One initiative's readout led with a mobile win — ten of ten product variants, “where three of four sessions originate.” All true. Going back through the device data:
Mobile — share of sessions
Mobile — share of conversions
Desktop — share of conversions, on 24% of sessions
Net across both devices, the mobile gain didn't cover the desktop loss. The winning variant came out roughly 5% down on total conversions.
“Wins 10 of 10 on mobile” and “lost about 5% of total conversions” are both accurate descriptions of the same test. The first is the one that got reported and remembered.
That isn't an analytics error. Choosing which denominator frames a result is a judgment about what the product is for, and it was being made by people with no reason to think of it as a design question. This is the clearest answer I have to what does closing the loop actually buy you — not better numbers, but results that mean what the organization thinks they mean.
What the audit found
What I'd flag
The limitations I'd raise before someone else did.
- I don't have adoption evidence yet.
- Built recently; whether anyone else uses it is unproven. That's the honest weak point of this piece.
- Links are file-level, not frame-level.
- The design tool's API doesn't expose page-level links, so each entry depends on a context block inside the file. Manual steps decay.
- I let a compromised design go to test without recording the delta.
- One initiative's designs were sanitized through partner review until production was close to the existing pages. The verdict on file is a verdict on the compromise. The review process did what risk-averse review processes do — not documenting it was mine.