Rayyan—a web and mobile app for systematic reviews
Tóm tắt
Synthesis of multiple randomized controlled trials (RCTs) in a systematic review can summarize the effects of individual outcomes and provide numerical answers about the effectiveness of interventions. Filtering of searches is time consuming, and no single method fulfills the principal requirements of speed with accuracy. Automation of systematic reviews is driven by a necessity to expedite the availability of current best evidence for policy and clinical decision-making. We developed Rayyan (
http://rayyan.qcri.org
), a free web and mobile app, that helps expedite the initial screening of abstracts and titles using a process of semi-automation while incorporating a high level of usability. For the beta testing phase, we used two published Cochrane reviews in which included studies had been selected manually. Their searches, with 1030 records and 273 records, were uploaded to Rayyan. Different features of Rayyan were tested using these two reviews. We also conducted a survey of Rayyan’s users and collected feedback through a built-in feature. Pilot testing of Rayyan focused on usability, accuracy against manual methods, and the added value of the prediction feature. The “taster” review (273 records) allowed a quick overview of Rayyan for early comments on usability. The second review (1030 records) required several iterations to identify the previously identified 11 trials. The “suggestions” and “hints,” based on the “prediction model,” appeared as testing progressed beyond five included studies. Post rollout user experiences and a reflexive response by the developers enabled real-time modifications and improvements. The survey respondents reported 40% average time savings when using Rayyan compared to others tools, with 34% of the respondents reporting more than 50% time savings. In addition, around 75% of the respondents mentioned that screening and labeling studies as well as collaborating on reviews to be the two most important features of Rayyan. As of November 2016, Rayyan users exceed 2000 from over 60 countries conducting hundreds of reviews totaling more than 1.6M citations. Feedback from users, obtained mostly through the app web site and a recent survey, has highlighted the ease in exploration of searches, the time saved, and simplicity in sharing and comparing include-exclude decisions. The strongest features of the app, identified and reported in user feedback, were its ability to help in screening and collaboration as well as the time savings it affords to users. Rayyan is responsive and intuitive in use with significant potential to lighten the load of reviewers.
Tài liệu tham khảo
Chalmers TC, Smith H, Blackburn B, Silverman B, Schroeder B, Reitman D, Ambroz A. A method for assessing the quality of a randomized control trial. Control Clin Trials. 1981; 2(1):31–49.
Uman LS. Systematic reviews and meta-analyses. J Can Acad Child Adolesc Psychiatry. 2011; 20(1):57.
Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009; 151(4):264–9.
Wallace BC, Trikalinos TA, Lau J, Brodley C, Schmid CH. Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinforma. 2010; 11(1):55.
Tsafnat G, Glasziou P, Choong MK, Dunn A, Galgani F, Coiera E. Systematic review automation technologies. Syst Rev. 2014; 3(1):74.
O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev. 2015; 4(1):5.
Ruby on Rails. http://rubyonrails.org. Accessed Jan 2016.
Heroku. http://heroku.com. Accessed Jan 2016.
Heroku Postgres. https://www.heroku.com/postgres. Accessed Jan 2016.
Logentries. https://logentries.com. Accessed Jan 2016.
NewRelic. http://newrelic.com. Accessed Jan 2016.
Pusher. https://pusher.com. Accessed Jan 2016.
HireFire. https://www.hirefire.io. Accessed Jan 2016.
EMBASE screening project to filter randomised controlled trials. http://screening.metaxis.com/EMBASE/. Accessed Jan 2016.
Cortes C, Vapnik V. Support-vector networks. Mach Learn. 1995; 20(3):273–97. doi:10.1007/BF00994018.
Cohen A, Hersh W, Peterson K, Yen PY. Reducing workload in systematic review preparation using automated citation classification. J Am Med Inform Assoc: JAMIA. 2006; 13(2):206–19. doi:10.1197/jamia.M1929.
Pedrazzi V, Oliveira-Neto JM, Sequeira P, Fedorowicz Z, Nasser M. Hand and ultrasonic instrumentation for orthograde root canal treatment of permanent teeth. Cochrane Database Syst Rev. 2008;4(CD006384). doi:10.1002/14651858.CD006384.pub3.
Fedorowicz Z, Nasser M, Sequeira-Byron P, de Souza RF, Carter B, Heft M. Irrigants for non-surgical root canal treatment in mature permanent teeth. Cochrane Database of Syst Rev. 2012;9(CD008948). doi:10.1002/14651858.CD008948.pub2.
Paynter R, Banez LL, Berliner E, Erinoff E, Lege-Matsuura J, Potter S, Uhl S. EPC Methods: An Exploration of the Use of Text-Mining Software in Systematic Reviews. Research white paper. AHRQ publication 16-EHC023-EF. Agency for Healthcare Research and Quality. 2016. https://www.ncbi.nlm.nih.gov/pubmed/27195359.