Methodology for Developing Rubric

Methodology for Developing Rubric

CHIP’s contribution to this work began with an effort to design a preliminary rubric that will help evaluate 100 & Change proposals for structural inequality. This rubric is being refined through feedback from faculty at Penn’s School of Social Policy and Practice. The following describes our process for arriving at an initial “strawman” rubric.

Defining Structural Inequality

Our research starts by defining structural inequality. To arrive at a definition, we consulted the most highly cited academic literature on the subject. To place this theoretical view of inequality in an applied contemporary context, we supplemented our review of academic literature with commentary on structural inequality from popular sources with expertise on the domains of philanthropy and economics.

Structural Inequality

Structural inequality describes disparities in wealth, resources, and other outcomes that result from discriminatory practices perpetuated through institutions such as legal, educational, business, government and health care systems. Structural inequalities result from imbalances in the distribution of political and economic power, where one group has historically set the rules in order to exclude others from access to wealth and resources.[1] Positions of economic and political power, called ‘rents’ in social science literature, allow holders to reap benefits without effort to create value. For example, a feudal lord who hangs a chain across a river and demands payment from any boat that wants to pass it has created a rent. Because benefits accrue to those closest to such positions in social structures, rents lead to sum-zero competition for proximity to power where those similarly located are incentivized to strategically align and create barriers that reduce competition for status from those in other positions. Those barriers are the basis for structural inequality.[2]

Structural inequality differs from inequality in outcomes that result from individual agency and instead focuses on how dominant social institutions shape outcomes for individuals without regard to their personal decisions, efforts, talent or needs.[3] Social institutions are the system of roles, norms and values lodged in social structures and organizing human behavior. In other words, institutions are the rules, written or unwritten, that humans play by as they pursue education, employment, household formation, financial security, among other things. Schools and hospitals are social institutions, as are more abstract concepts like public policy, or systems of bribery and favor-trading.[4]

When social institutions are created by and designed to serve the interests of one group to the exclusion of another, structural inequalities result. The effects of segregation and disenfranchisement of black Americans are the starkest examples of structural inequality in the U.S. Globally, international borders are major structural drivers of inequality, as illustrated by comparisons of development indicators in towns on the Mexico-U.S. border.[5] More insidious forms of structural inequality include the combined effect of low-density zoning policies that stymie development of affordable housing in rich neighborhoods and property tax-financed public schools. This creates major disparities in per-pupil spending across school districts, leading to worse educational outcomes, and fewer opportunities for employment and higher education for students in low-income areas. Further examples include the family and medical leave policies of many employers which fail to accommodate the needs of mothers. These policies create barriers to career advancement leading to underrepresentation of women in leadership positions, perpetuating bias in company policy-making and the behavior of corporations writ large.

Given this understanding of structural inequality, CHIP’s research sought a more comprehensive picture of the factors that constitute and contribute to the condition both in the United States and globally. To do so, we identify frameworks that and indices conceptualize aspects of inequality and suggest metrics for measuring it.






Initial Framework Scan

Once we defined structural inequality, we conducted a comprehensive scan to identify existing frameworks with potential for use in philanthropic competitions. We looked for frameworks that fit the following criteria:

  • Evidence-based: Is the framework based on scientific literature, field experience, and/or informed opinion of key stakeholders?
  • Comprehensive: Does the framework allow for consideration of all the facets of the complex field of behavioral health? Is the framework applicable to all persons suffering from and/or affected by behavioral health problems?
  • Accessible: Can a lay audience easily understand the framework?
  • Practical: Does the framework provide actionable guidance for those evaluating a program’s suitability for a philanthropic prize?
Analysis of Frameworks

In our initial scan, the CHIP research team identified 15 frameworks that conceptualized structural inequality or some aspect of it. Three more were added by referrals from faculty interviews. These frameworks include both general and subject-specific (e.g. inequality in health or education) literature from academic sources, think tanks and advocacy groups, or governmental organizations, and are designed to serve a variety of purposes. Academic research employs a conceptual framework in order to identify factors with causal or correlational relationships with inequality. Various indices provide ways of measuring inequality to give policy makers and nonprofits more information about community need. Some frameworks are more prescriptive, and suggest policies or strategies for reducing structural inequality. A table at the end of this document summarizes the 18 frameworks we reviewed.

Among the academic articles reviewed, two focused on analyzing the source of inequities in health. These identified income and healthcare supply (availability of doctor, pharmaceutical insurance, etc.) as having the biggest effect in health inequalities among individuals. An examination of drivers of racial disparities in health outcomes pointed to neighborhood factors, immigration enforcement, perceived racism institutions, socioeconomic status, and workplace factors. Another academic article unpacked the variety of sources of racial inequality, focusing on how cultural traits (attitudes, habits, etc.) are shaped by structural factors, thereby underscoring the importance of understanding that relationship in prescribing interventions. Finally, a geographic study of economic mobility in the U.S. shows that 76% of the variation in intergenerational mobility among census tracts is explained by segregation, income inequality, school quality, social capital, and family stability.

Frameworks focused on measuring inequality include Pew Research’s summary of measures of economic inequality which include income, expenditure ratios, and household wealth. We also examined two indices for understanding overall inequality that conceptualized inequality as including economic, healthcare, educational, and community factors in one case, or basic human needs, foundations of well-being (education and healthcare), and opportunity in another. An index measuring inequity in higher education focused on six key questions: who enrolls, what types of institutions they enroll in, how they pay, whether financial aid is available, how household characteristics predict attainment, and how the U.S. compares internationally. The Social Vulnerability Index is produced by the CDC to measure how susceptible an area is to public health threats and focuses on socioeconomic status, household composition, race/ethnicity/language, and housing/transportation type.

Yet another category of frameworks for understanding structural inequality are policy-focused analyses. These more prescriptive frameworks range from guidance for philanthropists on how to engage with marginalized communities, to policy recommendations written by the UN for national policymakers. They also include subject-specific analyses focused on health (the Social Determinants of Health) and education (the Achievement Gap Initiative.) The list of policy recommendations is summarized in detail in a separate framework analysis document, but is too wide-ranging and exhaustive to synthesize concisely. Example guidance ranges from the importance of elevating existing community leadership in philanthropic efforts, to the need for land reform in low and middle-income countries. These expert recommendations can be useful in assessing an intervention’s approach to addressing a structural issue.

Following initial desk research, referrals to other frameworks were solicited from 1:1 interviews with SP2 faculty. These referrals included Child Welfare Risks and Protective Factors, Critical Race Theory, and Theories of Action for Community Development. These referrals provide insights narrowly on the fields of community development, social work, and child welfare, and are intended as tools for practitioners within those fields.

Taken together these three types of frameworks can be used in a variety of ways in constructing our own rubric. First, we can consult the framework components for ideas on how to break down structural inequality into more targeted problems that a nonprofit program might target. The index literature is particularly valuable for this analysis, and suggests categories such as economic/employment factors, healthcare, education, neighborhood/community factors, and family characteristics. Second, academic analyses suggest what factors might be prioritized in our rubric/framework. For example, neighborhood characteristics were emphasized as drivers on inequality in several studies and so our rubric might emphasize the importance of place-based interventions. Finally, the policy recommendations inform us as to what experts see as the key barriers to overcoming structural inequality. These insights can be applied at the micro-level such as ensuring nonprofit leadership is diverse and representative of communities served, or to more macro-issues such as how social safety net programs are financed.

Analysis of Prize Rubrics

With this more nuanced understanding of structural inequality, the CHIP team turned to thinking about how these concepts could be translated into a method of evaluating of nonprofit programs that sought to address inequality. For this effort, we consulted the rubrics and calls for proposals from other philanthropic competitions. We summarized and synthesized the criteria used in these competitions as well as how they related to the competition’s impact goals.

Analysis of Rubrics

To provide insight on how our understanding of structural inequality might be translated into criteria with which to evaluate nonprofits, we also reviewed rubrics and calls for proposals from philanthropic prizes. We started with 100& Change’s own evaluation criteria, but also included rubrics from 17 other philanthropic prizes or other evaluation processes. A table summarizing these rubrics is included at the end of this document.

The 100 & Change competition includes four stages of evaluation that includes peer review, “wise-head” (informed generalists) panel review, review by a technical expert in the issue the proposal addresses, and final selection by the MacArthur Board of Directors. Proposals are evaluated to assess how meaningful they are: whether they target a significant and urgent problem and could significantly improve lives; how verifiable they are: whether there is evidence that the proposal could be effective; how feasible they are; whether the organization/team has the skills, resources, and capacity to deliver the solution; and finally how durable they are: whether the proposed solution has staying power. A 0-4&5 scale is defined for each trait.

Looking across other prize rubrics, there is significant overlap in the criteria they define for selecting an outstanding proposal. Often the first criteria listed is the creativity or promise of the idea or solution that underlines the proposal. This is often evaluated in the context of the competition’s goals, and suggests that our rubric should start with an evaluation of the promise of the idea as a solution to structural inequality.  Feasibility with respect to a project plan and budget is another near-universal inclusion in rubrics. The credibility of the organization or coalition is also frequently included. Most rubrics include evaluation of scalability especially those with bold impact goals. Finally, sustainability or durability of the program’s effects is a frequently occurring component.

There are other less common rubric criteria that might be particularly suited to the evaluation of a proposal’s potential to address structural inequality. One is the focus on a systems-level issue where an intervention can catalyze large-scale change. Both Co-Impact and Rodenberry prizes emphasize this criterion though neither defines it in significant detail. Another potentially important criteria to include given a focus on structural inequality is participation. The MIT Inclusive Innovation Challenge asks, “Does the organization enable more people, especially those in socially disadvantaged groups (based on race, gender, age, socioeconomic status), to be included in the digital economy?”, and defines a 0-5 scale for that question. A full summary of all rubric criteria is included in our framework analysis document.

Faculty Interviews

From analysis of frameworks and rubrics, the CHIP team outlined an initial rubric with five criteria that could be used to assess a proposal’s potential to counteract structural inequality. This rubric served as a basis of discussion with faculty at Penn’s School of Social Policy and Practice in 1:1 interviews. These interviews asked faculty for their feedback on our definition of structural inequality and evaluation criteria, and called our attention to literature and frameworks not included in our initial scan.

Faculty Interviews

After using literature review to develop a deeper understanding of structural inequality, we designed an initial rubric that could evaluate a proposal designed to counteract it. The rubric served as the basis of our interviews with four faculty from the School of Social Policy and Practice at the University of Pennsylvania. These interviews asked faculty for their feedback on our definition of structural inequality and evaluation criteria, and called our attention to literature and frameworks not included in our initial scan.

Feedback on Initial Definition of Structural Inequality – Feedback on our definition emphasized the need to stress that structural inequality was not a passive phenomenon. By describing “biases embedded into social institutions” our definition could be interpreted as something subconscious, when in fact these biases reflect intentional choices made by those in power to discriminate against marginalized groups. Faculty noted that the criminal justice system did not fit easily into the common components identified through analysis of frameworks. Several faculty members also referred frameworks to our review, including Risks and Protective Factors for Child Welfare, and Critical Race Theory for social work practitioners.

Feedback on Initial Rubric – Faculty expressed general agreement with the five criteria included in our initial rubric, but did make several suggestions in how we defined and evaluated these criteria. Notably, faculty emphasized the need to consider evidence-base and an intervention targeting structures that entrench inequality in our Theory of Change criteria. It was also suggested that evaluation of Direct Impact include “implementation science”, or plans to scale the intervention and adapt it to other contexts. As a result of comments, we updated our criteria for Systems Level Impact to include a question as to whether the intervention leveraged existing community-based efforts. A number of comments helped us clarify the definitions of “Inclusivity” and “Durability of Power” including suggestions for specific ways of sharing power, such as by giving beneficiaries authority to make financial and hiring decisions.  We incorporated this feedback into our Strawman Rubric below.