Knowing what “good” looks like in the digital health space has proved challenging for most solutions. Net promoter scores (NPR) don’t tell a consumer about efficacy or effectiveness. On one level, given the nature of funding for these companies, developing a sham version of the product to test as part of a Randomized Control Trial (RCT) is unlikely. Currently, we see a lot of “wait-list controls” as a control condition.
A new paper by Chaohui Guo and colleagues from Imperial College London, published in Nature Digital Medicine, reviews the evaluation issues at hand in digital health. The authors to anchor to FDA definitions released in 2020. While conventional methodologies like RCTs are the gold standard, digital health evaluations have not matched this standard. In part, due to the fact, a digital solution keeps iterating as it learns more about what its end-user finds beneficial.
This space is noisy due to a lack of evaluation. This lack doesn’t inspire confidence from health systems that look for efficacious and effective solutions to support diagnosing. monitoring or treating a clinical condition toward a clinically benchmarked standard. Most digital health tools fall under the wellness category and thus outside of the regulatory framework where claims of benefit and outcomes must be demonstrated. It is not uncommon to see health systems be part of the evaluation cycle to determine whether a solution is credible and meets standards and needs. This system isn’t sustainable, given how many digital solutions are funded, and a more evolved evaluation framework is necessary. The evidence base rarely compares like products against like, another challenge for healthcare systems in determining best in class.
The table below shows the current state of the challenges facing innovative companies to demonstrate their solution’s value and where companies need to move toward to be taken seriously beyond the consumer space.
Source: https://doi.org/10.1038/s41746-020-00314-2
The conventional evaluation methods don’t take the agile methods used by innovators to develop software. To succeed, multiple parties need to collaborate to define pragmatic ways to show value. The longer tail of study design, intervention testing, and outcomes generation is at odds with digital health methods, so a new path is necessary. Companies will have to value evidence generation from product inception versus waiting a few years into the development cycle. Partnerships with academic institutions are growing in the partner space, which lends credibility to development efforts. The reality is evidence generation with new products has historically been weaker. It may be enough for early adopters to step into the testing space but is unlikely to garner most of the majority’s attention. To date, a gap exists between lower-cost approaches early in the development cycle and higher cost approaches required to garner market share. The authors propose that simulation approaches might be a step forward; this allows clinicians access to the solutions with simulated patient actor’s responses demonstrate how the product may be useful. Creating scenarios that simulate real-world clinical interactions can support more rapid solution development.
To move beyond digital health’s dilemma of “no implementation without evidence, no evidence without implementation,” innovative, pragmatic approaches must be developed.
Thanks for reading – Trina
(Opinions are my own)
References
Guo, C., Ashrafian, H., Ghafur, S. et al. Challenges for the evaluation of digital health solutions—A call for innovative evidence generation approaches. npj Digit. Med. 3, 110 (2020). https://doi.org/10.1038/s41746-020-00314-2
