What is our purpose? How do we make a difference? How can we tell that we’re making a difference?
These are questions all nonprofits and mission-driven organizations discuss and revisit over time. They also form the backbone of Theory of Change methodology and a process of data gathering, querying, and evaluation. This article discusses Theory of Change in practice and a few examples of metrics for use in responding to that last question: “How can we tell that we’re making a difference?”
What is Theory of Change?
In a nutshell, a Theory of Change is an organization’s set of beliefs and hypotheses about how their activities lead to outcomes that contribute to the overall mission and vision of the organization. A fully articulated Theory of Change includes references to data that can be collected to use for reporting on metrics, or quantitative measures used to monitor the progress and results of an organization’s efforts.
Example: Humane Society of Boulder County
The Humane Society of Boulder County has the mission “to protect and enhance the lives of companion animals by promoting healthy relationships between pets and people.” They do this by:
- Providing shelter and care for more than 7,000 animals per year
- Running a successful adoption program, which allows them to open their doors to thousands of animals in the region
- Running a behavior modification program, which helps shelter dogs and cats overcome behavior problems, a shelter medicine program, a full-service veterinary clinic, and reward-based, positive reinforcement training and behavioral courses which are open to the public
Metrics on all these activities can be quantified and tracked over time. This allows the Humane Society of Boulder County to keep track of their progress internally, and to report externally about their contribution to the overall mission of protecting and enhancing the lives of companion animals by promoting health relationships between pets and people.
Metrics for Monitoring
Quantitative metrics should never be the sole source of information about an organization’s progress and impact, but they can be useful proxies for monitoring the outcomes of an organization’s activities. Some “obvious” metrics like funds raised as a percent of the goal, or proportion of registered volunteers who attend a function, are indeed very useful to track. But sometimes a more nuanced metric is necessary or useful for more accurately understanding the progress of the organization according to its Theory of Change. Here are a couple examples.
Financial
Fundraising is a core function of many nonprofit and religious organizations. A congregation, for example, may want to track its members’ pledges of financial support against its overall annual fundraising goal. Supposing the annual goal was $100,000, the results in both of the following graphs show fantastic results, with about 99% of the goal pledged.
But there is a valuable difference, made apparent by plotting individual pledges this way. In the first pledging scenario, fewer than 10 individual pledges constitute over 75% of the total raised. This means that the congregation is heavily reliant on a few families or individuals for most its funding. Although the funding goal was met for the year, this situation is precarious. If some of these top 10% donor individuals or families move, retire, or otherwise decide to contribute less, the congregation will find it much more difficult to meet future fundraising goals.
In the second scenario, there is a much stronger “mid-level” donor pool. Here, it’s the top 33 donors who are responsible for the same pledge total as the top nine donors in the first scenario. This congregation is in a more stable position for future fundraising. If some families or individuals move, or their financial situations change, there is still a significant pool of mid-level donors who will pledge again.
It is worth tracking not just funds raised compared to an overall goal, but also characteristics of the donor base such as total number of contributors, and proportion of the donor pool at low, medium, and high donation levels.
Participation
For many nonprofits – and certainly for congregations and institutes of higher education – participation, whether in events, services, or classes, is a core metric. Identifying participation goals — and tracking data against those goals — is sometimes quite straight-forward. For example, if your organization hosts parenting classes and has 30 people enrolled, you might reasonably expect that the goal is perfect attendance and you track data on who shows up to classes against your list of enrolled participants.
Things are trickier, however, for programs with short terms goals that relate to high rates of participation, and long-term goals that are self-expiring. Homelessness response, job-seeking support, and English language learning are examples of programs where the long-term goal for any one individual is that they are eventually able to stop participating.
A job-seeking support workshop might also enroll 30 people at the beginning of the year. In the first weeks of the program, the program leaders might push hard to have all 30 people attend the workshop sessions. After three months, however, it is reasonable to expect that some people will stop attending, not because they’re too busy or uninterested, but because they have accomplished the goal of the program – they have found a job.
For programs like this, the best way to track progress is to keep track of individuals. Simply tracking number of attendees per week is not as valuable as tracking individuals: when they enroll and why, how often they attend, AND for what reason they leave. Ideally, their post-program progress can also be monitored if contact information is kept up to date.
Considerations
Numbers aren’t everything
It’s important to note that quantitative metrics do not give the entire picture of an organization’s progress. Case studies, qualitative interviews, and even informal anecdotes can all provide valuable context and powerful narrative to accompany statistics.
Metrics are not static
Neither is a Theory of Change permanent once it has been articulated. Organizations change focus, join forces, face new challenges, and sometimes simply decide that what seemed like a core metric previously is no longer salient.
Be Careful What you Measure
…And watch for unintended consequences.
There is a now infamous story of a call center who generated a goal of reducing call time to improve productivity. They were forced to abandon this goal when they realized that their representatives were rushing callers or just hanging up on them in order to “perform to the goal.” Wells Fargo came under significant scrutiny when widespread fraudulent account creation was noticed. It turned out that the goal of “Eight is Great” (i.e., that every household should have eight Wells Fargo products) led to pressure on retail staff to meet their internal sales goals by signing up more accounts than were desired by their customers.
These stories serve as reminders that although developing clear goals and the metrics to monitor progress toward them can be powerful forces for good in an organization, they should be revisited periodically. If they are not serving the organization well – either because they do not correlate well with long-term success according to the Theory of Change, or because unintended negative consequences outweigh the benefits – the best action is to try again with a thoughtful approach to goal setting and metrics development.
If you would like help thinking through how your organization’s current practice of data collection and analysis correspond to your key metrics, short and long term goals, and Theory of Change, please reach out to Cloud for Good.
You may also be interested in these related Cloud for Good blog posts for more insight:
