In the boardrooms of organizations investing in AI tools, the question is no longer "How many people are using it?" but "What has this investment delivered?" To answer it, most organizations lean on one of two data sources: they either survey employees and ask how much time they have saved, or they pull usage and productivity data from their systems. Either approach on its own can give incomplete, even misleading, results.
- Measuring the return on AI investment with a single data source is misleading. Employee surveys show perception, experience and reasons; observed work data shows behavior and output.
- The time savings employees report can differ systematically from actual savings. That is why the question "How much time has AI saved you?" should not on its own be the basis of an ROI calculation.
- Observed work data has blind spots too: it does not show quality, rework, unofficial tool use or where the time saved actually goes.
- The most reliable approach is a measurement model that combines a baseline measurement, a comparison group, observed work indicators and an employee survey.
In this post we explain what employee surveys and observed work data do and do not show when measuring the return on AI investment, why self-reported time savings should be read with care, and a measurement model that combines the two. If you are looking for example questions to measure how AI use affects the employee experience, see our post on measuring AI effectiveness with employee surveys.
Why is AI ROI hard to measure?
The formula for calculating the return on AI investment is simple: (Benefit − Cost) / Cost. The difficulty lies in filling in both sides of the formula correctly.
The cost side is usually underestimated. The license fee is a visible line item, but integration, data preparation, training, the time employees spend learning the tools, security and governance processes, and the time spent checking AI output are all part of the cost.
The benefit side is hard to measure for three reasons:
- The benefit is scattered. AI does not affect a single process but hundreds of small tasks: drafting an email, summarizing a report, writing a piece of code.
- The benefit is mixed up with other effects. A new process, a new team structure or a seasonal change in workload during the same period also affects productivity.
- Time saved is not automatically money saved. If the time saved is not redirected to more valuable work, it does not turn into a financial return.
What do employee surveys show, and what do they not show?
An employee survey shows how employees experience AI. There is no other way to get this information:
- Which tasks AI genuinely helps with and which it does not,
- The barriers that keep employees from using the tools: lack of skills, lack of trust, unclear rules,
- How much they trust the quality of AI output and how much correction it needs,
- Where the time saved goes,
- Unofficial use of tools the organization does not provide.
But surveys have an important limitation: the savings employees perceive and the actual savings may not be the same.
Why can self-reported time savings mislead?
AI tools can make work feel smoother and less tiring. That feeling is often confused with the feeling of working faster. In 2025, a randomized controlled study by the research organization METR with experienced software developers showed this gap strikingly: participants believed they finished their work about 20% faster with AI tools, yet the measured times showed that tasks took longer with these tools.
This result cannot be generalized to every role and every tool; in many jobs AI delivers measurable time savings. But it is an important warning: asking employees "How many hours a week has AI saved you?" and multiplying the answer by an hourly rate is not a sound ROI calculation.
Making survey questions more reliable
To make surveys more reliable on time savings:
- Ask about concrete behavior instead of estimates. Instead of "How much time have you saved?", ask "How many hours did you spend on [task] last week?", and ask the same question before the tool is rolled out.
- Focus on specific tasks. Rather than a general perception of productivity, ask separately about the specific tasks AI is used for (preparing reports, responding to customer requests, writing code).
- Ask about correction time too. The question "How much do you need to correct AI output before you can use it?" helps you understand the net value of the savings.
- Protect anonymity. Employees may worry that if they say AI makes their work easier, their workload will grow or their role will be questioned. This concern can distort answers in either direction. For details, see our post on anonymity in employee surveys.
What does observed work data show, and what does it not show?
Observed work data is the trail employees leave in systems as they do their work: processing times, the number of completed tasks, customer request resolution times, error and return rates, software delivery times and AI tool usage logs. This data is strong exactly where surveys are weak: it shows actual behavior and output, not perception.
But observed work data has its own blind spots:
- It does not always show quality. Customer requests closed faster may be reopened more often. More code written may contain more bugs.
- It does not see unofficial use. If employees use a tool they found themselves instead of the one the organization provides, that use does not show up in system data.
- It does not show the reason. If productivity has not improved in a team, is it because the tool is unsuitable, because training is lacking, or because the rules are unclear? System data cannot answer this question.
- What gets measured can become the target. Once usage rates become a performance target, employees may start using the tools not to create value but to hit the metric.
- Not every job produces measurable output. In strategy, management, relationship management and creative work, observable output is limited.
Comparing the two data sources
| Comparison | Employee survey | Observed work data |
|---|---|---|
| What does it measure? | Perception, experience, trust, barriers, reasons | Behavior, output, time, usage |
| Strength | Answers "why?" and "for whom?" | Unaffected by perception bias |
| Weakness | Self-reported savings can differ from actual savings | Does not show quality, reasons or unofficial use |
| Suitable for which roles? | All roles | Process-based roles with measurable output |
| Privacy risk | Low when anonymous | Can create a perception of individual monitoring; must be managed carefully |
As the table shows, the two sources are not alternatives but complements. The right question is not "survey or work data?" but "which question will I answer with which data?"
A measurement model that combines both data sources
To measure the return on AI investment reliably, we recommend the following steps:
- Define the use cases. Instead of "the return on AI", focus on specific tasks: first responses to customer requests, preparing monthly reports, contract review. For each use case, write down in advance what success looks like.
- Take a baseline measurement. Before the tools are rolled out, collect both work data (time, volume, quality) and survey data (workload, time spent on tasks, experience). Any assessment made without a baseline is guesswork.
- Set up a comparison group if possible. Roll out the tools to some teams first and compare them with other teams doing similar work. This is the most reliable way to separate out the effect of other changes in the same period.
- Pair observed indicators with quality indicators. Put a quality indicator next to every speed indicator: reopening rate next to resolution time, number of corrections next to report preparation time.
- Measure reasons and experience with the survey. Use the survey to ask which use cases work, what barriers exist and where the time saved goes.
- Compare the two data sources. Do the perceived savings in the survey match the observed savings in the work data? Large gaps show where both the measurement and the implementation need to be reviewed.
- Convert to financial value cautiously. Count time saved as financial return only to the extent you can show it has been redirected to more valuable work. Report the rest separately as "potential capacity".
When the survey and the work data disagree
When the two data sources give different results, it is usually not a measurement error but the most valuable finding:
| Survey | Work data | Possible interpretation |
|---|---|---|
| High perceived savings | No clear change | The tool makes work easier but not faster, or the savings are spread across other tasks |
| Low perceived savings | Clear improvement | Employees do not notice the benefit, or rising speed expectations are creating pressure |
| High perceived savings | Speed rising, quality falling | Outputs are being used without enough checking |
| Low perceived savings | Low usage | There are barriers: skills, trust, rules or an unsuitable tool |
Measurement timeline
Measuring the return on AI too early captures the cost of the learning period and misses the benefit. The recommended timeline is:
- Before rollout: Baseline measurement (work data and survey).
- Month 3: First measurement. At this stage, read barriers and adoption rather than return.
- Month 6: First meaningful assessment of return.
- Month 12: Annual assessment and review of investment decisions.
To measure where your organization stands on its AI journey, you can also take a look at our post on measuring AI maturity.
Common mistakes
- Counting license usage as return. The number of active users is an indicator of adoption, not of value.
- Multiplying self-reported time by an hourly rate. This calculation ignores both perception bias and the question of where the time saved goes.
- Skipping the baseline. Once the tools have been rolled out, reconstructing the "before" is almost impossible.
- Looking only at speed. If speed comes at the expense of quality, the return is lower than it seems.
- Creating a perception of individual monitoring. Turning work data into individual performance tracking leads employees to hide their tool use and answer surveys cautiously. Run analyses at team and use-case level.
In short, to measure the return on AI investment you do not have to choose between surveys and observed work data, and you should not. Work data shows what has changed; the survey shows why and for whom. A model built on a baseline, a comparison group, quality indicators and a comparison of the two sources is the foundation of a reliable answer you can take to the board.
Frequently asked questions
How is the return on AI investment (ROI) calculated?
The basic formula is the benefit AI delivers minus the total cost, divided by the total cost. Cost should include not only licenses but also integration, training, learning time and governance. Benefit is made up of items such as time savings, quality gains, fewer errors and revenue impact, and should be measured against a comparison group where possible.
Can the time AI saves be measured with a survey?
A survey gives a useful signal about time savings, but it is not enough on its own. The savings employees perceive can differ from the actual savings. For a more reliable result, ask concrete questions such as "How many hours did you spend on this task last week?" and compare the survey results with observed work data.
What is observed work data?
It is the trail employees leave in systems as they do their work: the number of completed tasks, processing times, customer request resolution times, error and return rates, software delivery times and AI tool usage logs. This data shows behavior and output, but not reasons or experience.
What is the most common mistake in measuring AI ROI?
Converting license usage rates or employees' self-reported time savings directly into financial return. Time saved does not mean a financial return unless it is redirected to more valuable work. In addition, measurements made without a baseline and a comparison group attribute effects that have nothing to do with AI to AI.
When should the return on AI be measured?
Take a baseline measurement before the tools are rolled out, then repeat it once the initial learning period is over (usually in months 3 and 6). Measurements in the first weeks can understate the return because of the learning curve.
