Why is measuring worker outcome so challenging for workers’ compensation systems?

Those who make laws and policy do so to achieve certain outcomes.  Unfortunately, meaningful measurement of outcomes in workers’ compensation is sadly lacking. 

Workers’ compensation outcomes seem straightforward enough.  Safe workplaces.  Fair compensation for work injuries.   Effective treatment and rehabilitation.   Safe, timely and durable return to work.   There are others, but even these basic worker outcomes (or indicators related to them) are rarely reported. 

The most obvious barrier to reporting worker outcomes is that they are hard to measure.  Unlike “dollars spent” or “new claims received”, which are very objective input and process measures, worker outcome measures require a great deal of time and effort to develop, track, and report.  

Measuring “safe, timely and durable return to work” , “impact of disability on future earnings”, or “worker satisfaction with claim process”  for example, require specific definitions for these important terms and a mechanism for consistently assessing cases and reporting outcome on a timely basis. 

Often, the only way to get the data is through interviews well after the last temporary disability payment has been sent and the claim closed.  Many organizations are just not willing to put the time, staff resources and money into getting the data needed to produce a credible outcome measure on a timely basis.

One way to overcome this barrier is to be selective in the population under study.  Focusing on a few injury types, industries, and occupations can make the process easier.  Sampling techniques can reduce the number of cases (and related time, effort, resources and costs) needed to get a representative result.  Often, the lower precision of the outcome measure is an acceptable price to pay for increased timeliness of the analysis. 

The second barrier is a little less obvious but vitally important.  Worker outcome measures are meaningless without an appropriate basis on which to assess the reported result.  Even if you put the money, time and effort into developing and reporting on a given worker outcome, how is a policy maker or other stakeholder to know if the reported outcome is good or bad?  Without a credible parallel measure to compare with, an outcome measure may only provide year-over-year change data.  Comparator data is notoriously hard to come by.

No matter what population you decide to focus your outcome measure on, you are going to need something to compare your result against.  Where are you going to find that data?  You can’t just use another jurisdiction’s data without first adjusting for factors that might otherwise impact the outcome—factors like age, gender, industry, and occupation.  Not only that, privacy rules are likely to add to the data acquisition challenge.

Consider “duration of temporary disability”—arguably an important outcome for injured workers (who suffer the financial and physical losses while away from work) and employers (who pay the claim costs and consequences of worker absence including backfilling costs, lost productivity, etc.).  One would expect cases of similar work-related injury in similar occupations and industries would have similar outcomes assuming other factors are also similar.  Differences in outcomes across jurisdictions selected for comparison can be a fabulous starting point for exploring policy and practice impacts on worker outcomes.  But I can tell you from experience, getting timely comparable data from jurisdictions outside your own is a herculean task.

This example raises a third barrier, the “Do I really want to know?” challenge.  Outcome measurements that are rigorously developed and reported with appropriate comparator start discussions and raise questions that some may not want to consider. Fear that the result of outcome measurement will make a jurisdiction “look bad” may be the biggest unspoken reason for avoiding the whole process or the real reason behind stated objections against involvement (funding, providing data) in worker outcome research.  This barrier applies both to the jurisdiction developing the measure and any other jurisdictions approached to participate as a comparator.

At this point, you can understand why outcome measures are rarely reported in workers’ compensation.  Yet, if you are interested in improving workers’ compensation systems, outcome data across jurisdictions is essential.  When you come across well developed outcome measures from multiple jurisdictions it is like finding a vein of pure policy gold in the mountains of workers’ comp statistics, reports and data out there.  

A couple of recent studies demonstrate how the commitment of participating jurisdictions and the dedication of researchers have overcome these barriers.  These research products provide credible, useful outcome measures and analysis that policy makers and stakeholders can use to evaluate system performance and improve workers’ compensation.  Each study involves very large sample sizes and matched data sets that control for variations in many factors ( such as injury type, industry mix, age, gender, etc.).

Alex Collie, Tyler J Lane, Behrooz Hassani-Mahmooei, Jason Thompson , and Chris McLeod,  “Does time off work after injury vary by jurisdiction? A comparative study of eight Australian workers' compensation systems”, BMJ Open 2016;6:e010910 doi:10.1136/bmjopen-2015-010910

This study examines more than 90,000 claims and controls for demographic, worker and employer factors; it shows conclusively that jurisdiction in which an injured worker makes a compensation claim has a significant and independent impact on duration of time loss.  (Free on-line article).

Bogdan Savych and Vennela Thumula. “Comparing Outcomes for Injured Workers in …” WCRI, May 2016

This study (or, more accurately, a series of parallel studies) examines worker outcomes for each of the 15 states:  (Arkansas,  Connecticut, Florida,  Georgia, Indiana, Iowa, Kentucky, Michigan, Massachusetts, Minnesota, North Carolina, Pennsylvania, Tennessee, Virginia, Wisconsin)  using claim and interview data from very large samples in each jurisdiction. Each study controls for “mix” of industry and financial severity of the claim.  In the “Data Book” supplements for each jurisdiction, the authors provide worker outcome data for the unadjusted for case mix and additional detail on return-to-work accommodations provided in both successful and unsuccessful cases. (Limited free viewing and  free policy-maker registration for webinar;  low cost for others).

Neither of these examples identifies the specific policy features that may account for the outcome differences—that was never their purpose.  System features such as the presence (and length of the waiting period), rate of compensation, mandatory reinstatement laws, specific vocation rehabilitation programs, and insurance arrangements (exclusive state fund, competitive state fund, or private provision) are a few candidates for stakeholders and policy makers to consider.   

Another series of studies that do attempt to evaluate the impact of legislative changes on the specific worker outcome of post injury earnings and the adequacy of compensation are highlighted in a recent research summary: 

Emile Tompa, R Saunders , C Mustard, and  QLiao  “ Measuring the adequacy of workers’ compensation benefits in Ontario: An update”   IWH Issues Briefing,   March 2016.

This summary updates the analysis of benefits adequacy in Ontario by looking at more recent cohorts of permanently impaired workers’ compensation beneficiaries following the 1998 changes to Ontario’s workers’ compensation legislation.  This update does not directly compare any other jurisdictions although the methods and prior research but the study demonstrates the complexity of analysis necessary in outcome analysis.  Prior research in this series used data from British Columbia and Ontario as well as comparable data from taxation data sets.  Previous work by RAND and other research groups on data from California, New Mexico and Washington state, among others.  (Issue briefings and previous IWH research available for free online viewing).

By the way, although the results of these high-quality, peer-reviewed research efforts may be “free” to stakeholders and policy makers everywhere, the research itself has significant costs.  Not all workers’ compensation jurisdictions financially support workers’ compensation research, but thanks to those that do, every system can benefit from research findings. 

Getting those findings takes qualified researchers familiar with the data, jurisdictions committed to providing data and data definitions, and hours of work by analysts not to mention the infrastructure necessary to secure and maintain the integrity of the data, review research, publish and other activities transfer that knowledge to those who can act on it.  Understanding workers’ compensation is not a trivial undertaking for anyone including skilled and capable researchers.  It is doubtful that any of this research could have been undertaken without the knowledge base and experience with workers’ comp data evident among the researchers leading these studies.

Bottom line:  Worker outcome measures are challenging but essential to assessing and improving workers’ compensation systems. A few jurisdictions have invested their time and resources demonstrating  just how valuable this sort of research can be.  Every jurisdiction should actively pursue worker outcome research by contributing their data to comparative efforts,  funding workers’ compensation research,  and developing the research talent in workers’ compensation.