CrowdCARE: Crowdsourcing critical appraisal to support evidence-based practice
Laura Downie, University of Melbourne
Co-authors: Michael Pianta
Evidence-based practice (EBP) is a dominant paradigm in healthcare that aims to deliver the highest quality patient care. EBP requires clinicians to integrate the best-available, current research evidence, with their own clinical expertise, and to consider patients’ needs and preferences, when making clinical decisions. Consideration of the ‘best’ evidence requires clinicians to evaluate the scientific quality of published studies (i.e., undertake critical appraisal); however, recognised barriers to this process include a lack of skill, a lack of time, and the quantity of published research.
To overcome these established barriers to EBP, we have developed a free, online tool that teaches critical appraisal and facilitates the sharing of appraisals amongst a global community of clinicians (CrowdCARE, Crowdsourcing Critical Appraisal of Research Evidence: crowdcare.unimelb.edu.au). After completing training modules, clinicians can contribute to, and benefit from, a responsive and evolving stream of appraised research evidence, generated from an interdisciplinary group committed to practicing EBP. CrowdCARE is distinct from other evidence databases (e.g., PEDro, speechBITE), as it trains appraisal, eliminates the need to pay experts, is not discipline specific and can be rapidly updated as new studies are published.
In this presentation, we will outline how CrowdCARE provides clinicians with the essential skills to appraise research quality and contributes to making EBP more efficient by removing the substantial duplication of effort made by individual clinicians across the globe. We will share analyses relating to the quality of data generated using this crowdsourcing approach for critical appraisal and provide examples of how the system can be mined to reveal deficiencies in reporting within the clinical literature. Further, we will discuss how the CrowdCARE data-stream can support efficient and rapid evidence synthesis for clinical guidelines and systematic reviews, to inform practice and/or policy, based upon the best-available research evidence.
Otitis Media Guidelines 2001 to 2017: an upGRADE and an OM-app to improve clinical care of Aboriginal and Torres Strait Islander children
Amanda Leach, Menzies School of Health Research
Co-authors: Kelvin Kong, Peter Morris, Peter Richmond, Hasantha Gunasekera, Ngiare Brown, Deborah Lehmann, Paul Torzillo, Kim Mulholland, Anne Chang and the OM Guideline Technical Advisory Group (OM-TAG, alphabetical by first name): Amanda Leach, Chris Perry, David Isaacs, Deborah Lehmann, Harvey Coates, Hasantha Gunasekera, Jarod Pak, Jennifer Reath, Jessica Sommer, Judith Boswell, Katherine Jarosz, Kathy Currie, Kelvin Kong , Paul Torzillo, Peter Morris , Peter Richmond, Sam Brophy-Williams, Samantha Harkus, Stephen O’Leary.
Otitis media (OM) and associated hearing loss causes speech and language delays, social isolation, behaviour problems and educational disadvantage resulting in lifelong impacts on employment and wellbeing. In remote areas 90% of children <3 years have OM, and up to 20% have chronic suppurative OM. The NHMRC-funded Centre for Research Excellence Ear and Hearing Health of Aboriginal and Torres Strait Islander children is a national research collaboration dedicated to reducing OM through high quality innovative clinical research, Indigenous leadership, and more effective and sustainable research translation.
Aims: Using GRADE, to update “Recommendations for Clinical Care Guidelines on the Management of Otitis Media in Aboriginal & Torres Strait Islander Populations” (the 2010 Guideline), and produce a multi-platform OM-app to enhance translation into practice.
Method: A multidisciplinary, geographically representative Technical Advisory Group (TAG) including Indigenous leaders was formed. Phase I: search strategies applied and Cochrane, Revman5 and GRADEpro used to assemble Evidence Profiles including quality and effect sizes, and Summary of Findings tables with “what happened” and number needed to treat. The TAG determined strength of recommendations for priority questions. Phase II: baseline stakeholder survey of 2010 Guideline, preferences of GPs, nurses and Aboriginal Health Practitioners in urban, rural and remote settings for 2017 version and OM-app. The digital multi-platform OM-app will include diagnostic aides (manual, video and image gallery including tympanograms). Phase III: finalise OM-app. Phase IV: evaluate and monitor impact of the OM-app, secure ongoing support for education and real-time updates according to GRADE standards. Phase V: incorporate a diversity of culturally appropriate information and advice for families.
Results and conclusions: Phase I is complete including prototype OM-app. We have 43 Summary of Findings tables and 169 outcomes for prevention and treatment of five clinical conditions. The GRADE approach requires substantial investment and expertise, and should be evaluated.
Adaptation of public health and clinical practice guidelines to local contexts: Review of frameworks
Zhicheng (Jeff) Wang, University of Sydney
Co-authors: Quinn Grundy, Lisa Bero
Background: The implementation of evidence-based guidelines can improve clinical and public health outcomes by helping health professionals practice in the most effective and efficient manner. The adaptation of guidelines to local context is crucial to successful guideline implementation. A variety of methods for adaptation (frameworks) have been developed in recent years. A review of different adaptation frameworks could provide insights into the current environment of guideline adaptation. This review aims to explore the current literature to examine the recent developments in the field of guideline adaptation.
Method: A scoping review of MEDLINE, Embase and CINAHL was conducted. Data extraction was done using an author-generated, a priori data collection form. The categories of analysis were formed by an inductive, thematic exploration of the studies.
Results: From the 2343 articles found in the search, 10 studies were classified as “description and critique of adaptation frameworks”. These studies described and analysed eight adaptation frameworks. The studies focused on the perspectives of the adapters and did not have any suggestions to making guidelines more adaptable. Frameworks have evolved over the years, by building upon each other and improving on the flaws of previous frameworks. Most of the frameworks have been applied in high resource settings; only one was developed for a low-resource setting.
Discussion: Future studies should focus on adaptation in low resource settings and determine what is required for a guideline to be adaptable. These studies could inform guideline developers, such as the World Health Organization, to develop guidelines with considerations for adaptation.
Comparing risk of bias between prospectively and retrospectively registered trials in the Australian New Zealand Clinical Trials Registry (ANZCTR)
Lene Seidler, NHMRC Clinical Trials Centre, University of Sydney
Co-authors: Kylie Hunter, Melina Willson, Ailsa Langford, Ava Tan-Koay, Slavica Berber, Lisa Askie
Background: Prospective trial registration is an important tool for reducing reporting bias. Registration prior to enrolment of the first participant helps ensure transparency by publicly documenting key trial characteristics and outcomes prior to any data collection or analysis. It has been proposed that systematic reviews should only include prospectively registered studies but it is currently unclear whether risk of bias is generally higher for retrospectively registered studies.
Objectives: To compare study characteristics and overall risk of bias for prospectively and retrospectively registered studies.
Methods: We included all 1753 studies registered on the ANZCTR in 2016. Registrations before enrolment of the first participant were defined as prospective; registrations after were defined as retrospective. We analysed whether timing of registration was related to allocation method (randomised versus non-randomised), blinding and target sample size. Additionally, we combined these criteria to assess overall risk of bias: studies that were randomised, blinded and had an above median sample size were classified as low risk of bias.
Results: Prospectively registered trials were more likely to be randomly allocated (OR = 1.29, 95% Confidence Interval (CI) 1.01-1.64), and to blind their participants to treatment allocation (OR = 1.59, 95% CI = 1.27-1.98). There was no statistically significant difference in target sample size (Median (Interquartile Range) prospective = 72 (157); Median (IR) retrospective = 70 (129), p = .39). Of the studies classified as low risk of bias, 41% were retrospectively registered and 59% were prospectively registered, this difference was statistically significant (OR = 1.53, 95% CI = 1.18-1.98).
Conclusion: While on average prospectively registered studies were less likely to be biased, four out of ten of all studies that had an overall low risk of bias were retrospectively registered. Excluding retrospectively registered studies from systematic reviews would result in the exclusion of large numbers of low risk of bias studies that would otherwise add important information. Other potential solutions will be presented.
Improving the usability and efficiency of trial registration and updating processes on the Australian New Zealand Clinical Trials Registry (ANZCTR) by implementing new logic rules
Kylie Hunter, NHMRC Clinical Trials Centre, University of Sydney
Co-authors: Anna Lene Seidler, Ailsa Langford, Slavica Berber, AvaTan-Koay, Lisa Askie
Background: Prospective trial registration is widely accepted as an essential process to improve transparency and reduce research waste. Trial registries are an important tool to identify trials to be included in systematic reviews, and to assess and reduce risk of reporting bias. In the registration process, ANZCTR staff are required to review all registration and update submissions and email queries to the Registrant if the WHO minimum dataset items are not completed to an acceptable standard, or if there are inconsistencies or errors in the information provided. The Registrant must address these queries and submit the record for further review. Often a number of querying rounds are required. An easy and intelligent trial registration process can facilitate registration and help obtain high rates of prospectively registered trials.
Objectives: To implement automatic processes which improve the usability and efficiency of the ANZCTR registration and updating procedures, without compromising quality and completeness.
Methods: ANZCTR administrative and Information Technology (IT) staff devised new logic rules to increase automation when completing and updating the registration form. For example, if Registrants selected that their study was ‘Not yet recruiting’, they were not able to enter any data in the ‘Date of first participant enrolment’ field. We compared querying patterns for a period prior to, and following, implementation of the new logic rules in order to evaluate their effectiveness.
Results: Since implementation of the new logic rules on 4 October 2016, there has been a decrease in the amount of querying required by staff, particularly for update submissions. From January to September 2016 (the pre-logic rules period), an average of 55% of submitted updates required at least one round of querying by staff. From October 2016 to January 2017 (the post-logic rules period), this proportion decreased to 38%.
Conclusions: Implementation of logic rules within the ANZCTR registration and updating process was effective in reducing the amount of querying required by staff, thereby improving efficiency and usability.
Adapting critical appraisal methods to facilitate complex clinical decision making within the New Zealand Accident Compensation Corporation (ACC)
Melissa Barry, New Zealand Accident Compensation Corporation
The New Zealand Accident Compensation Corporation (ACC)* delivers NZ’s no fault accident insurance scheme. Three core functions of ACC are: injury prevention, rehabilitation and compensation. Building robust and pragmatic evidence resources may include critical appraisal of literature to inform business decisions that deliver these functions. We show that as demand for critical appraisal increases we can adapt literature analyses to meet the requirements of a fast moving dynamic organisation.
The ACC Research team works on projects that may determine the efficacy of medical interventions, exploring if an injury results from an accident or natural consequence, and horizon scanning of future technologies. Academic resources are critically appraised using techniques similar to those of the Cochrane Group, and inclusion/exclusion criteria for studies are adapted to meet the complexity of the question related to the business decision. Questions regarding efficacy of a medical intervention for a purchasing policy may require a rapid systematic review; whereas for the implications of newly developed technology, a brief scan describing the robustness of literature may be sufficient as only a small number of lower quality studies exist for that topic.
For complex clinical guidance ACC has developed an Agile-like approach. This involves Expert Working Groups (EWGs) of researchers, health professionals and consumer representatives meeting on a regular basis. Small targeted critical appraisals are performed for contentious parts of the guidance. These are then presented, debated then discussed about how the decision is implemented operationally. This approach is also useful for disseminating and obtaining agreement to projects where specific professional sectors have a vested interest, and making the process more efficient. Although cognitive computing may change complex clinical decision guidance in the future, we have shown at ACC how critical appraisal can be used and implemented to meet business needs now.
*ACC provides comprehensive, no-fault personal injury cover for all NZ residents and visitors to NZ including cover for some injuries caused by medical treatment
Increasing dissemination of Cochrane evidence using social media
Lucy Hardie, University of Auckland
Co-authors: Cindy Farquhar, Vanessa Jordan
Background: Social media is currently defined as Internet-based tools that allow individuals and communities to share information and ideas (Ventola 2014). These tools include Twitter, Facebook, LinkedIn, YouTube and many more. As of the first quarter of 2017 Twitter had 328 million active users and Facebook had over 1.94 billion (Statista.com). Social media has been identified as a potential way to promote health behaviours and interact with healthcare practitioners and consumers (Ventola 2014).
Objectives: To explore the use of social media to increase engagement between Cochrane New Zealand and healthcare practitioners, consumers and healthcare organisations
Methods: This ongoing project began in September 2016. We set out to identify the key elements for posts that would encourage engagement in social media. The initial phase of the project concentrated on the use of Twitter and this has subsequently been followed by the second phase of the project concentrating on Facebook. Hootsuite, a scheduling and analysis software program, was utilised to plan social media posts and allow consumer engagement to be monitored.
Results: We identified the key components of engaging posts as; a brief and intriguing tagline, the inclusion of related images, tags of the subject and local organisations with a relevant interest and finally an abbreviated link to the plain language summary. The number of Twitter followers has increased from by 267% (from 133 to 355). The total engagement (sum of interactions) is 466; 19 quotes, 225 retweets, 211 likes and 11 replies. New followers to Cochrane New Zealand on Twitter who are engaging with posts include Ageing Well New Zealand, Arthritis New Zealand, New Zealand Doctor, Otago Medical School and Plunket New Zealand which demonstrate the reach to local organisations.
Conclusions: Social media is a useful tool to increase dissemination of Cochrane evidence. This platform enables the timely transfer of the most recent findings to local organisations and consumers.
Ventola CL (2014). Social media and health care professionals: benefits, risks, and best practices. Pharmacy & Therapeutics 39(7), 491-520.