Assessing Outcomes through Data Collection and Establishing Benchmarking Goals

• Metro South/West Regional Workforce Investment Board, MA
• Seattle/King County Workforce Development Council, WA
• Illinois Workforce Investment Board and Illinois Department of Employment Security, IL
• Maryland State Workforce Investment Board

Overview

The Metro South/West Regional Employment Board adopted the MA Self-Sufficiency Standard as a goal to drive both program and evaluation design. In the first phase of its use, self-sufficiency benchmark data is being used for self-evaluation and program improvement, without any attendant funding penalties. The longer-term goal is to establish the Standard as a formal benchmark accompanied by data collection that could be used to determine funding of service providers.
The decision of the Metro South/West Regional Employment Board (REB) to formalize the MA Standard in its evaluation process was preceded by a pilot study in 2000. The purpose of the study was to measure the effectiveness of the REB's low-income adult and dislocated worker programs. The assessment of these programs was based on the Standard, which had already been calculated for communities in Massachusetts. Based on the report's findings, the authors observed that a "paradigm shift" is created by setting a higher goal for wage outcomes: i.e., to move customers toward a self-sufficiency wage. The Board concluded that setting high wage goals can lead to program improvements, a greater emphasis on helping workers’ enhance their educational skills, and measure progress in meeting employers’ needs for skilled labor. In addition to Metro South/West REB seven other WIBS within Massachusetts have requested data on how their outcomes compare to the Standards for their areas.

top ^

History

The 2000 Metro South/West Regional Employment Board pilot study (described in the overview) measured the effectiveness of the REB’s low-income adult and dislocated worker programs. The report, Defining Success in the New Economy: A Case Study of Metro Southwest Massachusetts Self-Sufficiency as a Benchmark for Workforce Programs [PDF, 338KB], found that while participants in the low-income adult programs improved their earnings, most did not earn enough to meet the Standard including many dislocated worker program participants who were unable to maintain their standard of living.

In addition to the decision to utilize the concept of self-sufficiency to effectuate the "paradigm shift" described above, the report contained. two other key recommendations that drove much of the policy changes at the Metro South/West Regional REB. One of these recommendations called for a new benchmark focusing on self-sufficiency:

"Set a higher goal for workforce programs by using the Self-Sufficiency Standard as a benchmark to understand the current state of the workforce development system in moving its customers toward self-sufficiency. Suggest the Self-Sufficiency Standard as a local option that can be adopted by local workforce investment boards."

The report noted that a self-sufficiency benchmark should be used as an indicator against which actual program performance may be compared, without penalties or WIA Title I performance bonus money. This policy distinction would allow self-sufficiency benchmark data to be used to provide information to local WIBs for self-evaluation and program improvement, without the threat of losing funds. By analyzing the benchmark data and related information local WIBs could determine which program strategies most effectively move people toward self-sufficiency.

In response to the report’s findings and recommendations, the Metro South/West Board established a new goal of moving all WIA customers toward wage outcomes that meet the Self-Sufficiency Standard. The REB adopted a service delivery model that moves customers toward self-sufficiency through on-going education and training.
The Metro South/West REB Director credits the shift to a higher self-sufficiency standard with transforming the way the board and staff think about service delivery and outcome goals. "The whole idea of self-sufficiency through career ladders is firmly embedded in what we do. I don’t know if that ever could have happened when I was just using 200% of poverty [as the self-sufficiency threshold]."

top ^

Partners and Funding

Funding for the report was provided by the Commonwealth Corporation, a quasi-public agency that conducts and funds a wide range of workforce development research and consulting.

Challenges

The Metro South/West board hesitated about adopting the Self-Sufficiency Standard because it could make all of their programs seem as if they were falling short. However, the Board ultimately concluded that the Standard was essential to measuring the workforce development system’s impact on lower-income customers.

top ^

Keys to Successful Implementation

• A willingness to analyze current wage outcomes in comparison with established Self-Sufficiency Standard and to make necessary changes in response to the results.
• Using a self-sufficiency benchmark as a goal, performance indicator, and tool for program evaluation and improvement while not penalizing service providers if customers fall short of the self-sufficiency goal.

top ^

Model Materials

• Defining Success in the New Economy: A Case Study of Metro Southwest Massachusetts Self-Sufficiency as a Benchmark for Workforce Programs [PDF, 338KB]

top ^