Cut the time per client
It was 20+ hours per client per month, plus coordination overhead, to produce one report.
ResultUnder one hour, end to end.Pandatron · Leader Dashboard
From 20+ hours of manual reporting to an automated, auditable analytics product
I specified, designed and led delivery of a new analytics product that replaced our manual client reporting.
Part of Pandatron's value proposition is to supply transformation leaders with insights and metrics about their transformation programmes. Until this project, insights were produced using unreliable methods prone to hallucinations and inaccuracies. This was a significant burden on our team and a suboptimal experience for our clients.
Two goals
Make findings auditable and trackable over time
Unreliable and hallucinated findings. No data infrastructure for tracking trends over time
ResultAll organisational findings internally traceable to source evidence. New data warehouse for easy insight tracking over time.My primary role was to lead the development of the system as a PM. I also took on significant design and engineering responsibilities, implementing production code including:
- 01Dashboard functionality, frontend and backend
- 02Analytics pipelines
- 03Agent workflows
- 04Evaluations
- 05Data contracts
Product context
What is Pandatron?
Read about the company
Product context
What is Pandatron?
Read about the company
Pandatron Background
Pandatron helps organisations succeed at large transformation programmes
Value 1 · For the employee
Each person gets a tailored AI conversation about the transformation
Scoped to their organisation's context, it helps them find a new angle, clarify a goal, or surface a blocker. Substantially more valuable to the employee than filling in a survey.
Value 2 · For leadership
Those same conversations are analysed at scale
Aggregated and privacy-safe, they become insight into what the workforce is actually experiencing. That second half is what this case study covers.
We worked with, for example:
AI adoption
5000+ employee automotive companyWhere people were using AI, where it was valuable, and what blocked adoption.
Leadership development
Multinational insurerSignals from a distributed leadership-development programme.
Merger integration
Fortune 500 pharmaceutical companyWorkforce experience during a tri-continental merger.
The solution
An organisational intelligence engine for transformation programmes
The system analyses employee conversations to show leaders where a transformation programme is getting stuck, which cohorts are thriving, and where interventions are likely to work.
Technically, this meant:
- 01ETL pipeline: ingesting and versioning completed sessions from our operational MongoDB into a new Postgres database built for analytics.
- 02Analysis agents: agents that read the conversational data and turn it into structured findings.
- 03Guardrails & evals: live code checks on every write, plus an offline LLM-as-judge loop that scores past output to guide prompt improvements.
- 04Analytics API: a NestJS backend that computes metrics and exposes them over a role-scoped API.
- 05Leader Dashboard: nextJS frontend where leaders interact with their organisational insights.
System architecture
data flow
retry / feedback loop
external network call
data store
Click any numbered component in the diagram for detail.
Pandatron platform
Where employees have their conversations. Writes each completed session to MongoDB.
01 · Ingestion
Streams completed sessions from MongoDB into a Postgres database specifically for analytics.
02 · Analysis agents
Custom tools for logging observations and answering the client's tracked questions, all model calls routed through our internal LLM gateway.
03 · Guardrails
Code guardrails validate every output before it is stored: e.g. agent referenced quotes are matched against the source transcript, question IDs are validated. Failed checks are handed back to the agent to correct.
LLM-as-judge
An offline evaluation loop that samples stored findings, scores inference and theme quality against a rubric calibrated to human reviewers, and the results feed back into harness improvements.
LLM Gateway
Every model call from the analysis agents is routed through a centralized internal service rather than hitting a provider directly. It handles provider/model fallback on failure or degradation, so a single provider outage doesn't stop analysis, and lets us swap or A/B different models without touching agent code.
04 · Analytics API
Queries the Postgres analytics database to compute metrics and serve findings to the dashboard, scoped by role so a cohort admin only ever sees their own cohort.
How the evidence is stored
A cohort finding (e.g. “German marketing ICs are held back by X”) is built from user patterns (what seems to be going on for one person), which are built from observations, single things noted in one conversation, which are built from quotes: the employee's own words, taken from a real transcript. Every level traces back to a verbatim quote.
05 · Leader Dashboard
Where client leaders view and track metrics and insights for their programme over time, and configure the questions they want tracked. See examples →
Simplified from the production system. Internal service names and client-specific detail omitted.
Worked example
See the system in action
Follow the system end to end: from a single conversation to a bounded, evidence-linked organisational finding.
↻ shapes next month's conversation
User on the Pandatron platform
Acme Org is running an AI-adoption programme on Pandatron.
Acme employees use Pandatron to discuss their experience with the transformation. Here, one employee in the German marketing ICs cohort sits down to talk about how the organisation is adopting AI. Leadership's standing question for this programme: “What is hindering effective AI use?”
Live session · conversation
PandatronHow has using the new AI tools been going for you this month?
EmployeeI feel conflicted. The written communication says we should experiment with AI, but in meetings it is treated as something people get criticised for using.
Pandatron… conversation continues
Observer agent
Observations are extracted from the raw transcript
The observer agent functions as the analytics engine's nervous system. It works continuously in the background, processing each conversation as it completes and recording raw evidence in a consistent schema. The goal is to gather observations for a given question, not yet to analyse.
Observer Agent TaskLog any evidence of barriers to effective AI use.
Conversation passed to observer agent
PandatronHow has using the new AI tools been going?
EmployeeI feel conflicted. We are told to experiment, but criticised in meetings.
Pandatron… conversation continues
Observer agent
Structured observation log
One conversation can yield many observations. Only evidence in the transcript is recorded.
observation_01 · manager behaviour conflicts with formal AI encouragement
observation_02
observation_03
observation_04
Live session · input
PandatronHow has using the new AI tools been going for you this month?
EmployeeI feel conflicted. The written communication says we should experiment with AI, but in meetings it is treated as something people get criticised for using.
Pandatron… conversation continues
Observer agent
log_observation · output
question_id · AI adoption challenges
observation · Conflicting communications from manager may discourage AI use despite formal encouragement.
supporting_quote · “I feel conflicted. The written communication says we should experiment with AI…”
Zooming out
Across the org, multiple conversations take place during the month.
Hundreds of other employees are in similar AI-adoption conversations right now. Each one being individually processed by the observer.
Employee 014 · observation log
session_014_01 · experimentation concern
session_014_03 · adoption pressure observed
Employee 029 · observation log
session_029_01 · tool access friction
session_029_02 · workflow guidance gap
Employee 037 · observation log
session_037_01 · leadership-facing risk
session_037_02 · psychological safety concern
Analyst agent · employee scope
The analyst agent models the user and their experience of the transformation programme.
For each employee, it has access to notes about their previous conversations. It then compares new observations against the previous hypothesis and updates its model of the user accordingly.
Analyst Agent TaskEvaluate new observations against the user's prior hypotheses.
Employee 0142 sessions
Previous user analysis
Prior hypothesis · low confidence
Manager signals about AI use are mixed.
Manager signals about AI use are mixed.
New observations since last analysis
session_014_01 · experimentation concernsession_014_03 · adoption pressure observed
Analyzing
Analyst reasoning
Both sessions point to the same barrier: formal encouragement conflicts with the signals the employee experiences day to day.
Updated hypothesis
AI use feels unsafe when manager behaviour contradicts policy.
Employee 0292 sessions
Previous user analysis
Prior hypothesis · medium confidence
AI does not fit the employee's current workflow.
AI does not fit the employee's current workflow.
New observations since last analysis
session_029_01 · tool access frictionsession_029_02 · workflow guidance gap
Analyzing
Analyst reasoning
The new workflow-guidance observation explains why the earlier tool friction has persisted, reinforcing a previously logged pattern.
Updated hypothesis
The employee cannot see how AI fits their day-to-day workflow.
Employee 0372 sessions
Previous user analysis
No prior hypothesis
No repeat pattern identified yet.
No repeat pattern identified yet.
New observations since last analysis
session_037_01 · leadership-facing risksession_037_02 · psychological safety concern
Analyzing
Analyst reasoning
Together, these observations suggest a distinct concern about being seen experimenting, not simply a lack of time or access.
New hypothesis
Visibility to leadership makes experimentation feel personally risky.
Analyst agent · context
one employeecohort
High-level prompt
You are an analysis agent. Turn evidence into careful, testable claims about this employee.
Question in focus
What is hindering effective AI use?
Permitted tools
log_hypothesissearch_user_history
This run
Generate a new hypothesis, or evaluate whether evidence is accumulating for an existing one.
Observations in context
manager behaviour · todayworkload concern · 3 weeks agounclear guidelines · 3 weeks agotool access friction · 2 weeks agotime pressure · 1 week ago
One analyst
Changes the scope of the evidence it can inspect.
Changes the scope of the evidence it can inspect.
one employeecohort
Observation · manager behaviour (example session)
Observation · workload concern (3 weeks ago)
Observation · unclear AI guidelines (3 weeks ago)
Observation · tool access friction (2 weeks ago)
Observation · time pressure (1 week ago)
Analyst agent · one employee
Analyze the observations and generate hypotheses about the employee.
Employee pattern · output
“The employee experiences a contradiction between formal AI-adoption messaging and manager behaviour, creating hesitation about AI use.”
rationale · The manager-behaviour observation is the direct driver; the workload, unclear-guidelines and time-pressure observations recur across three separate sessions, reinforcing rather than explaining the hesitation.
New hypothesisEmployee pattern · existing
“The employee is unsure how AI fits their day-to-day workflow.”
new evidence · Tool-access friction and time pressure reinforce a pattern first logged three weeks ago.
Evidence accumulatingSame analyst agent · cohort scope
User hypotheses are synthesised into group-level hypotheses.
Analyst agents then compare the anonymised employee patterns and generate new hypotheses. Existing findings get updated.
Cohort Analyser TaskEvaluate new user hypotheses against the cohort's prior hypotheses.
German marketing ICs10 qualifying users
Previous cohort analysis
Prior hypothesis · low confidence
AI experimentation may lack practical and social support.
AI experimentation may lack practical and social support.
New user hypotheses since last analysis
Employee 014 · mixed manager signalsEmployee 029 · AI does not fit workflowEmployee 037 · experimentation feels risky
Cohort analyser thinking
Cohort analyser reasoning
The incoming user patterns independently point to two connected gaps: employees need clearer workflow guidance and safer local signals to experiment.
Updated cohort hypothesis
AI experimentation is constrained by unclear workflow guidance and inconsistent local leadership signals.
The same analyst
Now compares anonymous employee patterns, not conversations.
Now compares anonymous employee patterns, not conversations.
one employeecohort
Employee pattern · Employee A
Employee pattern · Employee B
Employee pattern · Employee C
Employee pattern · Employee D
+ 6 more
Analyst agent · cohort scope
Cohort finding · output
“Employees are formally encouraged to use AI, but inconsistent middle-management behaviour is contributing to an environment where experimentation still feels discouraged.”
Transformation Intelligence
Findings come with suggested interventions whose effectiveness can be measured over time.
Acme leaders at different scopes can act on the intelligence and track intervention effectiveness.
Cohort admin viewGerman marketing ICs
cohort scopeFinding
AI experimentation is constrained by unclear workflow guidance and inconsistent local leadership signals.
Recommended intervention
Align managers on explicit safe-to-experiment language, then run a role-specific AI workflow pilot.
01 · BaselineRecord confidence and workflow-fit signals.
02 · ActRun the manager alignment session and pilot.
03 · CheckRe-measure at 30 and 60 days to see if the barrier falls.
Organisation admin viewCross-cohort findings
organisation scopeComparison
German and Japanese marketing teams share adoption friction, but the source of friction differs.
German marketing ICsLocal leadership signals make experimentation feel risky.
Japanese marketing ICsRole-specific workflow guidance is missing, despite manager support.
Organisation recommendation
Use a shared measurement framework, but tailor interventions: manager alignment in Germany, localised workflow playbooks in Japan. Compare outcomes before scaling either.
A cohort-admin opens their dashboard and sees a bounded, evidence-linked finding, never the raw transcript or a single employee's words in isolation.
Leader dashboard · finding card
Question · What is hindering effective AI use?
Cohort · German marketing ICs
Finding · “Employees are formally encouraged to use AI, but inconsistent middle-management behaviour is contributing to an environment where experimentation still feels discouraged.”
That one employee's conversation never left their own session. What reached leadership is a pattern, backed by evidence, that they can act on.
Note: this is a representative, fictionalised example. The system details have been simplified and details about the user have been modified to ensure privacy.
Product Gallery
Examples from the Live Dashboard
Sensitive details have been obscured.