Right now, in community development and in the nonprofit/charitable/public service sector more broadly, there is a push to incorporate more hard numbers, more metrics, into our evaluation of our performance.
In community development, many leading thinkers in the same breath as advocating for more performance metrics will talk about how community development has failed because the poverty rate is the same now as it was 50 years ago.
The prime metric for community development is the poverty rate—the success or failure for community development is the extent to which it moves the needle on poverty. But is this the right metric?
To explain further, I'm going to use the example of the movie Moneyball. Moneyball is about the application of Society of American Baseball Research (SABR) thinking to mine market inefficiencies in assembling a baseball team in an extremely resource-constrained environment. This piece is a nod to the SABR metric approach in the spirit of revisiting a traditional statistic, examining its flaws, and asking for new, better metrics.
One of the most radical and counter-intuitive assertions of the SABR set is that, in measuring a pitcher’s performance, it doesn’t matter if a ball, once it leaves the batter’s bat, is a hit or an out. This is a crazy notion. Isn’t the whole point of the pitcher’s job is to get outs and to prevent hits? But take the example a ground ball hit in the general direction of the shortstop.
In terms of what is within the purview of the pitcher, there is negligible difference between whether the ball is just out of reach of the diving shortstop’s glove, or whether the shortstop fields it cleanly and easily, or whether the ball finds an easy hole through the infield. While the outcome of the game varies hugely whether the ground ball is a hit or an out, the randomness of the path of the ball and the positioning and quality of the fielders are not under the pitcher’s control and therefore should not factor into the evaluation of the pitcher.
Part of the nebbishy beauty of baseball is that there is over a century’s worth of data that can be crunched and reformulated and correlated and analyzed to test assumptions about what stats relate to what outcomes. It turns out that the SABR fielding-independent approach to pitcher evaluation is significantly better than traditional stats at predicting future pitcher performance and that these statistics are more stable over the course of a pitcher’s career.
Community Development and Poverty
Poverty is a useful statistic in that it is a familiar, conventional indicator of economic need. It is good enough for descriptive purposes, especially when conveying a general sense of economic need, but it is a crappy evaluative statistic, particularly for talking about a specific set of programs and activities like community development.
Poverty is too complicated. There are too many factors that influence poverty and not enough clear relationships between inputs and outcomes. Poverty rates have some relationship to the larger economy, but correlate poorly with employment, GDP growth, per capita GDP growth, etc. This is the rising tide does not lift all boats point that I always pound away at.
Poverty rates have a stronger (inverse) correlative relationship with broader social investment (i.e., poverty rates fall as levels of social investment rise), but this includes a diverse mix of investments—education, social services, gender equity, childcare, healthcare, public safety, infrastructure, civil society, on and on—that have been inconsistently applied.
It is impossible to make statistically significant conclusions about specific programs and activities in isolation. And when you look at poverty and social investments, you have to realize you are not dealing with closed systems or with the same policies and conditions acting on a static population of people over time. Within the units of geographic analysis, there are inflows and outflows of people—internal migration, immigration—and there are changing policy priorities and shifting public and private investment trends.
Lessons From Moneyball
The lesson from SABRmetrics is to evaluate performance on factors that are actually under the control of the actor being evaluated and to create metrics that put these factors at the forefront. Don’t use statistics that are the responsibility of the entire team (e.g., don’t evaluate the pitcher for things that are about the quality of his fielders or, worse yet, the offensive performance of his teammates). And don’t rely on the same old statistics just because we always have.
Metrics should be more narrowly tailored to the specific programs and results: look at impacts on housing costs and neighborhood stability for affordable housing, look at school enrollment for youth and afterschool programs, etc. Make sure that you have significant sample sizes before you draw your conclusions. These are SABR’s lessons. These lessons tell us that it doesn’t make any kind of sense to condemn community development just because the poverty rate is the same now as it was 50 years ago.
I have enough of a soft, bleeding heart to strongly believe that community development needs better metrics and that numbers don’t fully capture what is most important about community development: community, empowerment, and social change.
People who know me well must think I talk out of both sides of my mouth on this. I like numbers. I think we should use more of them. But I also believe we have a hard time quantifying what is most important, most magical and precious.
The numbers paint a fuller picture of baseball than they do of real life. Baseball has a constrained number of inputs and outcomes, a more limited field of possibility. There are only so many places a ball can go once it leaves the pitcher’s hand. The pitcher’s mound and home plate are always, practically speaking, the same distance from each other. The rules and goals and objectives are mostly defined. But in real life, things are much messier. The field of play is much more open, varied, and variable.
So, at the same time I say that that SABRmetrics teaches us the power of better, more targeted metrics, I want to resist the absolute, universal application of quantitative methods to all decisions, to all evaluation.
We need to develop a better set of community development metrics but we also need to understand the limitations of these metrics. We still need a place for heart and grit. For stories. For dreams of empowerment and social change and for dreams of a better world. We still need a place for community.
(Photo by Alan English CC BY NC)
About the Author
Comments