A retrospective case review by ECRI Institute of its own forecasts on major technologies found that 75% of early predictions were still sound when the technologies reached routine clinical care. The organization’s findings were published in a case study in the February 2015 “Biomedical Innovations” themed issue of Health Affairs. In it, ECRI co-authors Jeffrey C. Lerner, Diane C. Robertson, and Sara M. Goldstein compare very early predictions with updated ones made after the technologies entered clinical care.
As described in an ECRI press release, the article addresses three questions central to the usefulness of forecasting:
- Are early forecasts accurate enough to help providers acquire the most promising technology and payers to set effective coverage policies?
- What variables contribute to inaccurate forecasts?
- How can forecasters manage the variables to improve accuracy?
The article compares ECRI forecasts published between 2007 and 2010 on four technologies with updates published in late 2013 and 2014. The technologies included single-room proton beam radiation therapy, digital breast tomosynthesis imaging, transcatheter aortic valve replacement, and minimally invasive robot-assisted surgery.
The authors found that 15 of 20 early predictions about the four technologies were accurate when compared with the updated forecasts. The inaccuracies involved two technologies more affected by “time-sensitive variables.”
According to Lerner, ECRI President and CEO, “We found that frequent revision of forecasts could improve accuracy, especially for complex technologies whose eventual use signals a paradigm shift in clinical care for a disease. We were surprised that so few forecasters examine their own work, and we hope this initial self-evaluation leads to increasingly robust, independent efforts.”
An abstract of the article, “Case Studies on Forecasting for Innovative Technologies: Frequent Revisions Improve Accuracy,” is available on the Health Affairs website. Subscribers to the journal may access the full article.