We applied the RAND method, which uses domain (subject matter) expert guidance, and a modified Ottawa method, which uses quantitative and qualitative signals. For both methods, we conducted focused electronic literature searches of recent studies using the key terms from the original reports. We assessed the agreement between the methods and qualitatively assessed the merits of each system.
Agreement between the two methods was 鈥渟ubstantial鈥?or better (kappa > 0.62) in three of the four systematic reviews. Overall agreement between the methods was 鈥渟ubstantial鈥?(kappa = 0.64, 95%confidence interval [CI] 0.45-0.83).
The RAND and modified Ottawa methods appear to provide similar signals for the possible need to update systematic reviews in this pilot study. Future evaluation with a broader range of clinical topics and eventual comparisons between signals to update reports and the results of full evidence review updates will be needed. We propose a hybrid approach combining the best features of both methods, which should allow efficient review and assessment of the need to update.