AI Transformation Playbook: The Definitive Guide to Measuring, Rescuing, Prioritizing, and Scaling AI Transformations

AI is impossible to ignore right now. But despite ever-increasing adoption, only four percent of companies are able to consistently generate value from their AI investments. Some of that’s because the technologies most orgs are experimenting with are so new.

But a lot of it is because most organizations don’t have the knowledge or tools to gather leading indicators of AI success.

In this playbook, we’ll lay out everything you need to know to measure the effectiveness of your AI implementations so you can rescue failing tools, prioritize future projects, and scale successful investments.

Follow these recommendations, and you’ll be able to achieve your transformation objectives faster and with less risk to the business.

Table of contents

• Frameworks for AI Transformations
• How to Measure AI Transformations: 5 Keys to staying ROI Positive
• Data Frameworks: How Surveys About Work Let You Achieve Transformation Goals Faster
• How to Rescue ROI-Negative AI Implementations
• How to Prioritize Future AI Investments
• How to Scale Future AI Investments

Frameworks for AI Transformations

Before we get into the hands-on tactics for making the most of your AI investments, let’s clarify what we mean by “AI transformations.” Broadly, we’re talking about digital transformations that involve an organization bringing in some third-party AI tool or software to improve operations.

In practice, we’ve found that these transformations tend to fall into one of three buckets:

Transformation Type	How it Works	Example
Internal services transformation	Implementing AI tools that help employees navigate company services (HR, IT, etc.)	A global organization streamlines enterprise services with AI-powered tools and saves $2.3 million annually
Employee work tool transformation	Implementing AI tools that employees use to do their jobs (chatbots, copilots, etc.)	A financial services company deploys an AI chatbot to increase developer efficiency and saves $5.4 million annually
Customer-facing transformation	Deploying AI solutions in customer-facing functions (chatbots, etc.)	An insurance provider deploys a self-service AI chatbot to speed up the claims process.

Read more about AI transformation frameworks.

Regardless of the type, AI transformations are different from other digital transformations in one important way: they are bottom-up rather than top-down. To succeed, they require individual employees across your organization to use them on a regular basis.

This means they have to make work easier. If a new AI tool doesn’t make work easier, employees will not use it and the ROI of the transformation will be negative.

Luckily, there’s a way to overcome this problem, and it involves a key concept in maintaining positive ROI in AI transformations: work friction.

To measure work friction, you have to look at specific work moments (aka specific tasks employees do during the day) and the touchpoints involved in each moment. Touchpoints include things like people and technology required to get the work done.

Much of the rest of this playbook focuses on how to identify work friction in and around AI tools. When you do that regularly, you’ll have the information you need to remove obstacles, improve productivity, and improve the ROI of all your AI transformations.

How to Measure AI Transformations: 5 Keys to Staying ROI Positive

Summary:

1. Measure leading indicators of ROI
2. Identify problem areas
3. Identify problem causes
4. Address problems
5. Re-measure

The problem many organizations face when measuring the ROI of AI transformations is that they’re only able to gather data on lagging indicators of success – things like…

• Decreased reliance on support teams (and lower support costs).
• Reduced employee time spent on tasks (and increased productivity).
• Stable or improved employee experience.

And when you have to wait for lagging indicators, it’s often too late to make changes to things that aren’t working.

To stay ahead of the ROI question, organizations need a way to measure leading indicators of AI transformation success. This means measuring the work itself – and the friction present in that work.

Leading indicators are things like…

• Relevance of work required to complete tasks.
• Effort required to complete tasks.
• Time required to complete tasks.
• Enjoyability of tasks.

To measure this, you have to run surveys that ask employees about what happens during specific work moments as they work with various touchpoints. In Figure 1, you’ll see what this looks like in FOUNT’s interface. (For more on how our data models work, jump to the next section.)

Figure 1: Example survey questions asking about moments and touchpoints

Work moments for a developer might include, for example, writing code, debugging code, writing documentation, finding an answer about the code base, reviewing pull requests, and so on. Touchpoints might include things like the developer portal, senior developers, external tech resources, an AI chatbot, etc.

Once you’ve collected this data, it’s time to identify problem areas. You can do this by plotting the importance of a moment against its impact on work overall (Figure 2).

Figure 2: A visualization of moments, plotted by importance to overall work vs. employee satisfaction

Moments that are of high importance that have low satisfaction numbers are problem areas. This is where your work friction exists!

If, during an AI transformation, anything related to your AI tool is generating work friction, the transformation is at risk for negative (or less-than-projected) ROI. As soon as you know this information, you can take steps to reduce the work friction and get the project back on track.

To identify what’s causing problems, look to the free-form answers from the survey (Figure 3).

Figure 3: Free-form survey answers and sentiment analysis

You can then estimate the ROI of addressing each problem (Figure 4).

Figure 4: Our process for calculating potential ROI for fixing problem areas

Once you have that information, you can develop solutions, implement them, and run the survey again to see if the work friction has resolved.

One thing to keep in mind: once you introduce AI, the nature of your employees’ work will change. Surveying them about their work is important not only to assess whether the solutions you implement work but also to identify new sources of friction that emerge from their changing role.

For more on this, check out How AI Tools Change Your Team’s Work (And What to Do About It).

To summarize: when you measure leading indicators of AI ROI (relevance of work, effort required, time required, enjoyability), you get a sense of ROI early in an implementation, when you still have time to change course.

Now let’s take a look at why these metrics are so powerful and why FOUNT’s system (which we’ve been showing via screenshots) works so well.

Data Frameworks: How Surveys About Work Let You Achieve Transformation Goals Faster

If you read that heading and thought, “The last thing my team needs is more surveys,” stay with us. 

First: Traditional employee experience surveys aren’t anchored to key work activities. They may offer valuable high-level insight, but they are rarely actionable when you’re trying to assess the ROI of an AI transformation.

Second: Traditional surveys are long and aim for 100 percent participation.

Third: Most organizations lack a structured framework for when and why surveys are sent. Employees may receive two or three surveys a day – one about submitting a ticket, another about using a tool, a third from a department lead doing their own research. Not only is this disjointed – it’s exhausting. And that’s not just survey fatigue.

Fourth: While survey fatigue is often blamed for disengagement, the deeper issue is a lack of visible action. Even when employee feedback drives decisions behind the scenes, those outcomes are rarely communicated. The result? Employees assume their input doesn’t matter, and engagement drops even further.

How FOUNT’s surveys are different

Our surveys (Figure 5) are different from traditional EX surveys in three ways:

1. They ask about the work itself: effort, time spent, friction experienced, and satisfaction in the context of real employee tasks.
2. They deliver meaningful insights with a small, focused sample. To get statistically significant results, you only have to survey about 53 employees.
3. They’re fast to complete – FOUNT surveys take between one and five minutes per person to complete.

And perhaps most importantly: Our model is built for action.

Figure 5: FOUNT’s data model translates worker experience into actionable data

Using actionable data to achieve transformation goals faster

Transformation happens when workers adopt ways of working that make them more productive, efficient, effective, or all three. To reach transformation goals, then, it’s essential to measure the actual work employees are doing.

For more on how FOUNT’s data fuels transformations, check out The Origins of and Statistical Models Behind FOUNT’s Use of Data.

How to Rescue an ROI-Negative AI Transformation

Let’s imagine now that you launched an AI product or tool at some point in the past. When you first checked your lagging indicators, they showed that the project was on its way to being ROI-negative – that is, the numbers showed…

• Increased reliance on support teams (or higher support costs);
• Increased employee time spent on tasks (or decreased productivity); or
• Worsening employee experience.

Don’t worry: it’s not too late to rescue the implementation.

Before we walk through the how-to, though, it’s important to resurface the concept of bottom-up transformations.

AI transformations are bottom-up because their success hinges on individual workers seeing the value in and therefore using AI tools. If workers don’t see the value in these tools, they’ll likely find a way to work around them.

This is terrible for ROI.

Even worse: once workers lose faith in an AI tool, it’s hard to recover that faith. So the sooner you can measure impact and adjust course as needed, the better. If it’s already been several months since the introduction of the unsuccessful AI tool, you’ll need a comms plan to accompany any changes you eventually make.

Now to the good stuff – how to rescue an ROI-negative AI transformation:

1. Identify work tasks (aka moments) affected by the AI tool.
2. Spin up surveys to assess how the AI tool affects those moments. If you work with FOUNT, you can choose from one of our many templated surveys and then tweak it to fit your team’s needs.
3. Distribute the survey to an appropriate selection of workers (often as few as 53).
4. Review results to identify areas of work friction (aka those with high importance and low satisfaction) (Figure 6).
5. Review freeform text responses to understand the causes of the friction.
6. Address the friction.
7. Re-survey to determine whether your fix worked.

For example, imagine an organization that introduced a coding copilot to its IT team to improve coding efficiency. When the team lead ran surveys to figure out why the team hadn’t yet reached the anticipated efficiency improvements, they found that three areas had high work friction (aka high importance but low satisfaction) (Figure 6):

• Writing technical documentation
• Reviewing pull requests
• Finding an answer about the code base

Figure 6: Bottom-ranked moments by importance vs. satisfaction

The team lead then considered the comments developers had written in the free-text portion of the survey and discovered that the words “portal” and “chatbot” came up over and over.

When they dug deeper, they found that developers wanted better documentation of internal knowledge in the portal – something that would make the chatbot trained on internal data far more effective.

This aha moment gave the team lead a clear goal for how to improve not only the effectiveness of the AI chatbot but also the overall efficiency of the team. They were able to rethink the portal, improve information sharing, and ultimately save $5.4 million in productivity across the IT team.

How to Prioritize Future AI Investments

Every day, AI leaps forward. New tools hit the market. Models hit new benchmarks. It’s no wonder, then, that 41 percent of CFOs struggle to prioritize AI projects because of uncertainty.

One strategy many business leaders take is to find AI tools for the parts of the business that drive the most revenue – the sales team, for example. While this sounds reasonable at first glance, it often leads to ho-hum results.

Why? Because the purpose of AI is to automate work. It functions best when it removes friction from existing workflows.

If you start from a revenue perspective, you may not choose a tool that addresses existing friction. Because of that, it may not offer much value to users, which means they may not actually use it – and then you’re looking at a negative ROI.

A better way to prioritize future AI investments is to start with friction.

Survey your employees to identify moments of work friction, research tools that can remove it, then repeat (Figure 7).

Figure 7: Flowchart of steps for prioritizing future AI projects

Note: In some cases, the best tool for the job won’t be an AI tool. That’s okay. If you position your overall AI strategy as one to improve efficiency and productivity, then it’s natural that identifying where not to use AI is as important as where to use it.

How to Scale AI Transformations

Once you achieve productivity or efficiency increases with an AI tool, your board will no doubt clamor for more across the organization. But AI success can be notoriously difficult to scale.

Why? One reason is that, once you introduce an AI tool, the nature of your employees’ work inevitably changes.

For example, if you introduce an AI chatbot to a call center to handle all straightforward customer questions, the result is that call center employees now handle exclusively complex questions. This means they may need additional training or support. It may mean you need to adjust your recruitment and hiring processes to prioritize workers with different skill sets.

In fact, it may mean a lot of things. This is why it’s important to measure work repeatedly, to understand which areas of friction are being resolved and where new friction points are cropping up.

For example, maybe the AI chatbot is good at most simple queries but bad at anything involving product returns. Those calls always end up escalating to human agents, and by the time the agent is on the line, the customer is frustrated, making the call more challenging.

A work friction survey would uncover this problem right away. You could then reroute return-related calls directly to agents while you adjust the AI chatbot and redeploy it.

If you’re always measuring employees’ work, you’re always seeing new opportunities for AI to improve it. This means you’ll always have a strategy for identifying your next AI deployment – which, across the organization, means you have a plan for scaling AI.

Protect Your AI Investment: Gather Data You Can Act On

AI tools are a major investment: in addition to their cost in dollars and cents, they require organizations to invest in organizational shifts, new ways of working, and even new ways of thinking.

When all that is on the line, a wait-and-see approach is not sufficient. By gathering data about the specific things AI changes, leaders can identify what’s working, adjust what’s not, and therefore keep projects on track to realize budgeted ROI.

Still have questions about your AI transformation? Get in touch – we’d love to help you ensure your current and future AI investments deliver the ROI you planned for.