Built a live dashboard tracker for client trials, kept up to date by an AI agent
Head of Product and Design · a health app · 2026 · client:
Problem
Trial data lived in four separate systems and was consolidated retrospectively. brought together once the trial had ended.no daily view
A participant who disengaged stopped using the app in week one showed up in the numbers only after the trial closed.
Solution
A daily consolidated view of engagement and retention, maintained by an AI agent following a documented procedure. The page computes a composite trial score and drafts a weekly report, gated by human approval. and a person says yes before it goes anywhere.about ten minutes a day
It came with a written guide, so a colleague could run it.
What it showed in the first weekit showed everything
Retention, engagement, and what each participant was doing hour by hour, all anonymised.
- Non-returning participants.
- Cross-system differences in counting. Places where two tools counted people and activity differently.
- A logged failure event with no user impact. An error nobody actually saw.
Previously, all three would have surfaced post-trial, at the end, or not at all.
The rules I built in
- Sources with different counting methods are shown side by side instead of added up. They count differently, so a total would be wrong.
- Shared outputs are pseudonymised. No names, only numbers.
- Small groups report counts, not percentages.
- Single-group findings are treated as signals, not conclusions. One group proves nothing yet.
- The AI drafts. A person reads, edits, then sends.