How to evaluate the benefits of AI for employees: measurement methodology during the pilot
The benefit of an AI assistant for employees is measured by the total task time — including preparing the request, reviewing and correcting the response — not by the model’s speed. Below is a practical methodology for the pilot: what to measure, how to compare “before” and “after,” how to calculate the savings, and by what signs to stop the project.
Why “like” and “responds quickly” are not yet an assessment
Author — Alexey Igonkin, General Director of RESTART Group. The material complements the author's column on «RBC Companies» (link — at the end of the article): there — the position, here — the workflow.
Initial feedback on the AI assistant is almost always good: employees find it interesting, responses appear in seconds, and the demonstration looks convincing. But neither "liking it" nor "responding quickly" indicates whether it has become easier for the department to work.
Model speed is response time. The benefit for an employee is the time it takes to complete the entire task: formulate the request, receive the answer, verify it, correct it, and only then apply it. If verification and corrections take as much time as the whole job used to take, there is no gain, no matter how quickly the model responds.
Hypothetical example. A lawyer receives a draft legal opinion on a contract from an assistant in less than a minute, and then spends half an hour checking every reference to a clause of the contract, because they are unsure of the accuracy. Response time has decreased radically, task time — hardly at all.
Therefore, the benefits should be assessed on real tasks, taking verification into account and with a clear point of comparison. Below is a methodology that can be applied in a pilot of any AI assistant for employees: document search, preparing responses, or reviewing contracts or requests.
What to measure
Five metrics are sufficient for the pilot. All of them are calculated by task, not by requests to the model, and are understandable to a manager without technical terminology.
| Indicator | How to calculate | What it shows |
|---|---|---|
| Total task time | From the start of work on the task to the finished result: preparing the request, working with the response, verification, corrections | Main indicator of usefulness; compared to the same type of tasks without AI |
| Share of responses accepted without edits | The number of tasks where the AI result was used as is, relative to the total number of tasks with AI | To what extent can an employee rely on the assistant |
| Rework rate | Tasks where the answer had to be discarded and the work redone, relative to the total number of tasks with AI | Hidden time losses: such a task costs more than without AI |
| Errors found later | Errors in the result discovered after being passed on: during approval, in the report, at the client's side | The cost of insufficient verification; one such error can outweigh the saved hours |
| Reviewer workload | Time spent by a manager or expert reviewing the results of employees who worked with AI | Has the work shifted from the performer to the reviewer |
What not to measure in the pilot. The number of requests to the assistant and user «activity» — a large number of requests can mean both benefit and unsuccessful attempts to get the right answer. Overall «satisfaction» without linkage to tasks. Model speed separately from the task. And savings at the scale of the whole company: the pilot tests one process, and it is too early to transfer its result to other departments.
How to build a measurement
The comparison is based on the «before and after» principle on identical tasks.
- Starting point (baseline) — a measurement of the same tasks without AI, against which the result is later compared. It is taken 2–4 weeks before the pilot, or from a sample of already completed tasks if the work time is visible in the ticketing or document management system.
- Identical tasks. Only tasks of the same type and complexity can be compared: a supply contract review — with a supply contract review, and not with an answer to a simple question.
- Control group — a subset of employees who, during the same period, perform the same tasks without AI. It helps separate the assistant’s effect from seasonal fluctuations and from the fact that people have simply become more attentive to their time. If it is not possible to isolate a group, compare against the baseline, but record what changed in the process during this time.
- Real documents and questions. Not a demo set, but the tasks that employees tackle every day, along with complex and non-standard cases.
- Who records the time. The most convenient way is to use timestamps in the work system. If there are none, the employee records the start and end of the task in a simple form, and the reviewer records the review time. The fewer manual entries, the more honest the data.
- Volume. The target is several dozen tasks of each type in each group. In several tasks, the output is determined by random outliers.
Measurement table template. One line — one task:
| Column | What to write down |
|---|---|
| Task number and type | For example, «review of a supply contract», «response to a client's request» |
| Group | With AI or without AI |
| Executor | Who performed the task |
| Preparation | Time to formulate the request and select source materials, minutes |
| Work | Time to complete without AI or to obtain and parse the AI response, minutes |
| Examination | Executor and verifier time to reconcile the result, minutes |
| Corrections | Time to refine the result, minutes |
| Total time | Sum of the previous four columns |
| Accepted without edits | Yes or no |
| Rework | Yes, if the AI's response had to be discarded |
| Error found later | Yes or no, where and when it was discovered |
| Comment | Why edits were needed: missing required document, inaccurate link, outdated data |
How to calculate savings
Savings are calculated based on the total task time, not the model response time:
Hours saved per month = (task time without AI − task time with AI, including review and corrections) × number of such tasks per month.
The time for a task with AI is taken as the average across all tasks in the group, including rework: if the answer had to be discarded, this task includes both the time spent working with the assistant and the time it took to redo it.
Hypothetical calculation example (the figures are made up for illustration, not the result of a project). Without AI, a task takes an average of 40 minutes. With AI: 5 minutes for the request, 10 minutes for review, 5 minutes for corrections — 20 minutes. There are 200 such tasks per month: (40 − 20) × 200 = 4 000 minutes, about 67 hours. If only the time to receive the answer were taken into account, the «savings» would come out to about 117 hours — almost twice as much as in reality.
Therefore, a measurement without verification and corrections always overstates the result. The same applies to errors found later: fixing them also takes time, and sometimes involves direct losses that must be deducted.
Money. The saved hours are converted into money at the hourly rate of the employees who perform the task. This amount is compared with the cost of the loop: licenses, computing resources, maintenance, and expert time for updating the knowledge base and verifying answers. A decision to scale is made only if the savings are consistently higher than these costs, not just in the best month of the pilot.
When to stop or change the scenario
Measurement is needed not only to confirm the benefit, but also to stop in time. Signs that the scenario should be closed or replaced:
- More edits than savings. The time spent on a task with AI, including verification, is no less than without it, and does not decrease from week to week.
- Rework does not decrease. The proportion of responses that have to be discarded remains high after configuring sources and queries.
- The data is not ready. The required documents are not available in an accessible form, are outdated, or have no owner responsible for keeping them up to date.
- Blocked by information security. The security service does not allow processing of the required data in the selected environment, and the environment cannot be changed within the pilot.
- There is no process owner. There is no one to accept the result and be responsible for how the assistant is used after the pilot.
- Late mistakes are more costly than savings. Errors found after delivering the result cost more than the time saved.
An honest “we’re not adopting this scenario” is a normal outcome of a pilot. Sometimes it helps to change the scenario: choose a process where a person already verifies the result before applying it, or a task with better-prepared data. But continuing a pilot just because it has already started is the most expensive option.
How it works at RESTART
At RESTART, the path to an AI assistant for employees is divided into stages, and each stage has its own «continue or not» decision.
- AI-discovery — ten working days to select a scenario. Scenarios are compared by four criteria: value, data readiness, access mode, and reversibility; for the main scenario, success and stop criteria are prepared, and the outcome is a decision to launch the pilot, postpone it, or abandon it. To assess the economics, the customer confirms the baseline metrics for transaction volume and labor costs — this is the basis for the future measurement.
- RAG pilot on a corporate basis — testing on real documents, users, and questions, in the four- and six-week versions. The technical acceptance criteria and the usefulness hypothesis are separated in it. The task time measurement from this article can be added to the pilot criteria along with an assessment of answer quality.
- A corporate platform — when there are several scenarios and shared roles, sources, logs, and operations are needed. How a pilot differs from a platform — in comparison; the platform itself — Restart AI Enterprise Platform.
The entire implementation route is on the page AI implementation in business.
Article on RBC
12.10.2026 · «RBC Companies» · expert column. Alexey Igonkin — How to Assess the Benefits of AI for Employees and Reduce Unnecessary Workload in the Workplace. In the column — the author's position; in this article — the measurement methodology, table template, and calculation. All publications of the group — in the section "Media about us".
Frequently asked questions
How long does the measurement take?
The baseline is collected 2–4 weeks before the pilot or from a sample of already completed tasks, if working time is visible in work systems. Then they work with AI for the same amount of time. The first days with the assistant are best left out of the calculation: employees are still getting used to the tool.
How many tasks are needed to draw a conclusion?
The benchmark is several dozen tasks of each type in each group. If there are fewer tasks, the conclusion is determined by individual cases: one difficult task or one successful response.
What should I do if employees do not log their time?
First, see what already exists in the working systems: request creation and closure times, and document approval stages. If there are no timestamps, a simple “started — finished” form is sufficient for pilot tasks, and the reviewer records the review time. There is no need to record every minute of the working day.
Can the benefit be assessed without a pilot?
Only as a preliminary assessment: based on the volume of operations, labor costs, and typical task durations, it is possible to estimate the potential of the scenario and choose what to test. The real benefit is demonstrated by working on your documents and tasks — with verification and corrections.
How does measuring usefulness differ from evaluating the quality of answers?
Quality assessment answers the question of how good the answers are on a test set of questions. Measuring usefulness is about how much time and effort employees spend on the task as a whole. Both are needed: good answers that take a long time to verify deliver no value.
Is a control group mandatory?
It is desirable, but not mandatory. If the employees cannot be separated, compare against the baseline on the same types of tasks and record what else has changed in the process during this time: new regulations, changes in team composition, seasonal workload.
Let's discuss your environment
Describe the task, current systems, constraints, and expected results. We will offer a practical first step: diagnostics, pilot, audit, roadmap or project team.
