The traditional approach to testing a new IT system is to delay end-user involvement until the final test phase, typically referred to as a User Acceptance Test. Until then, testing remains the responsibility of the project team and exposure to the new system is limited. The premise behind this approach is straightforward…don’t let the users see the new system until it’s stable and works well. But doing so sacrifices opportunities to enhance testing and change management efforts. A better approach is to involve users in the testing process, but do it in a controlled way.
There are many benefits to involving end-users in system testing. Some are obvious but others may not be readily apparent. Let’s review the key ones.
End-users approach a new system differently than the project team because it is a tool to perform their jobs. This perspective helps identify issues while the project is still underway, when it’s easier and cheaper to fix defects.
Including users in testing and helping shape the new system is a powerful way to create positive change agents. It gives users a sense of ownership in the process. Building a core support group within the user base will help increase acceptance once the system goes live.
Being part of testing also gives users an opportunity to get accustomed to the new system in a safe environment. They get to practice before it counts toward their job performance. Once the system goes live, not only will those users be more prepared, but they can help others adapt. This will mitigate the typical post go-live performance dip and lessen the load on technical support.
Finally, since testing usually begins prior to training, users that are involved in testing will have a solid knowledge base and get more out of the training experience.
So how do you effectively involve users in testing? The key is to proactively manage every aspect of the experience. Here are some guidelines to follow:
Involve users early in the process but not too early. If the project adopts a typical multi-phase test plan, the ideal time to involve users is phase two when testing usually becomes more functional in nature, rather than purely technical. It’s possible to involve users from the start but only if the system is already fairly stable.
Identify a cross-section of areas for users to test. But shield them, at first, from those that are not as stable. As testing progresses, broaden the areas in which they’re involved.
Prepare. Understand that users will likely have little knowledge of the system by that point. Train users to ensure they understand objectives, test procedures and tools. Define what testing success means and how to spot it. Distribute supporting documentation.
Monitor progress closely. Communicate with users frequently to help them understand testing results and gauge their experience. Solicit feedback but stick to requirements and designs. Testing is not an opportunity to raise new requirements. Listen to their issues and desired modifications but adhere to a predefined change control process.
Consider users’ inexperience when creating the test schedule. Allow more time to execute test scripts. Plan for project members to spend time with users to address issues and enforce good practices.
Have a clear process to report defects. Ensure that project members are on hand to clarify and coach users on what they’re seeing and how it relates to specific system functionality.
Start with a small group of users then expand. Do not rely on users as your core test team at the beginning. As testing progresses, gradually layer on more responsibilities.
Risks and Mitigation Strategies
Involving end-users in testing doesn’t come without risk. Let’s review the common ones and how to avoid them.
Users get exposed to problem areas and report back to peers and management, creating unnecessary change management problems. The way to mitigate this risk is to control the areas to which users are exposed and start with those that are robust and stable. As defects arise, ensure users understand the implications and how big or small a problem really is. Maintain a positive attitude and remind users that the system improves every day.
The rigor of user testing may be inadequate. Most business users typically have little or no IT testing experience. Don’t take for granted that users understand the purpose of testing and what a defect is. The tools to mitigate this risk are communication and education.
If users are not dedicated to testing on a full-time basis, they can be distracted by their day-to-day job responsibilities. Account for this when building the test plan. Be prepared to adjust workloads and assignments as necessary. Test execution plans typically have little slack and missed deadlines can easily have a snowball effect. It is important to build contingency into the test execution plan.
Including end-users when testing a new system can help a project immensely, and build goodwill across the organization. The critical success factor is to adopt a hands-on approach to managing user testing and facilitate the best possible experience.