Change Project 911: Counting what Counts: Addressing the challenge of incomplete data collection

Mat Roosa, LCSW-R
NIATx Coach

“Help! We don’t know if our change is an improvement!”

 

At the foundation of all quality improvement work lies data.

 

Imagine driving down a twisty road at night and having your headlights turned off for a portion of the journey. That’s what happens when we try to manage a change project without consistent data access.

 

It can be helpful to think about the data needed to steer a change in three stages:

 

Data at the beginning of the change journey: Baseline

The only way that we know if a change is an improvement is by measuring before the change, and comparing that measure to ongoing data collection during and after the change. We all know this. And yet, too often teams rush to implement changes and fail to collect baseline data. They are then left confused about the impact of the change and may be at risk of sustaining new activities that soon demonstrate little or no benefit.

 

Data during the change journey: Data-driven change management

As we drive along, we keep gathering data by looking down the road as far as we can see. Each turn in the road reveals new data to interpret and incorporate into our effort to steer safely. A failure to regularly collect data blinds a change team’s effort to interpret the change as it evolves.  

 

Data toward the end of the journey: Sustainment

At the conclusion of the change project, the team must ask whether they want to abandon, adopt, or adapt the change project based upon the data collected. The best way to sustain a successful change is through regular data checks that ensure that the new practice is firmly established and continues to have the desired effect.

 

This focus on data can all seem like a lot of work. However, focusing on a few key factors can help ensure that data collection continues for the duration of the project and beyond. The following tips can help you make sure that you count what counts:

Keep the data simple: If you have the choice between a perfect measure that is complex and a “good enough” measure that is simple, pick the good enough measure. To keep the entire team engaged in the project, keep the data clear and understandable to all team members. A simple line graph helps the team to track the trend.

Use existing data sources: Most teams have access to a range of existing data sources that they can use to steer the project without adding any additional burdens to the system. 

Assign a data coordinator. Placing one team member in charge of managing the data can ensure accountability. Each time the team meets, the data coordinator can make sure that the data is available and current. It can also help to have a second party assigned to the data coordination task, so that the data production process does not stop if the coordinator is not available.

Frontload the data effort. Too often, the data measurement components of a change project are developed and implemented in a mad scramble as a change project commences. Careful consideration of data well before the project begins will ensure consistent data.

Train people in data collection. There’s often an assumption that all participants in the change project have a clear understanding of data collection definitions and procedures. “Oh, I thought we were counting it this way!” is a frequent comment as teams realize that they failed to train the team on the specific data collection details. This confusion can result in a need to restart the change or throw away a portion of the data collected.

There are many different ways to count things. Engage the team in generating ideas about what data metrics to use and how to collect them. Even simple measures can be collected in different ways and require team dialogue. The team can help to consider how best to measure your change to make sure that you count what counts.

Make data collection an essential part of your change project from the beginning, and you and your team will have a built-in tool for seeing if your change is an improvement!  

 

About Change Project 911

Change Project 911 is a monthly blog post series covering common change project barriers and how to address them. Has your change project hit a snag that you’re not sure to tackle? Share your issue in the comments section below, or email Change Project 911 at matroosa@gmail.comWe’ll offer solutions from our team of change project experts!

About our Guest Blogger

Mat Roosa was a founding member of NIATx and has been a NIATx coach for a wide range of projects. He works as a consultant in quality improvement, organizational development and planning, and implementing evidence-based practices. His experience includes direct clinical practice in mental health and substance use services, teaching at the undergraduate and graduate levels, and human service agency administration. You can reach Mat (Change Project SOS) at matroosa@gmail.com.