Problems with evidence: publication bias and outcome switching

Researchers and practitioners in evidence-based fields rely upon quality evidence, and attempt to follow recommendations indicated by successful interventional and observational studies.  Two problems that prevent the true picture of evidence from reaching a wider audience are publication bias and outcome switching.

Publication bias is the tendency of research studies with positive results to be more easily published than research studies with null or negative results.(Sterling, 1959; Hopewell et al., 2009; Joober et al., 2012; Hedin et al., 2016)  Generally, academic journals favour studies with positive results, and researchers are also loath to publish research that has failed, especially embarrassingly so.  Because of the lack of comparative, unsuccessful studies on an intervention, readers and practitioners will infer an exaggerated and false level of viability of that intervention.  In medical research, publication bias is wide-spread and increasing.(Hopewell et al., 2009; Hedin et al., 2016; Mlinarić, Horvat and Smolčić, 2017)

Outcome switching occurs when researchers perform an intervention, and then receive results that do not support the original research aims, and so they ignore, or retrospectively alter, their research aims to match those results.  In so doing, they obscure the negative results and exaggerate (or totally falsify) the success of their intervention.  In medical research, there are many examples of outcome switching (Goldacre et al., 2016); perhaps the most notorious example is Study 329, dealing with the drug Paxil.(Doshi, 2015; Le Noury et al., 2015; Mulder, Rucklidge and Toop, 2016)

The problems in medical research and practice

Because evidence-based medicine relies on published research as that evidence, and publication bias and outcome switching results in tainted evidence, the impact on patient care can be devastating.(Whittington et al., 2004; Joober et al., 2012; Goldacre et al., 2016)

In response to these problems, Section 801 of The Food and Drug Administration Amendments Act of 2007, known as FDAAA 801,(FDA, 2007) established the legal necessity for protocols of all clinical trials in the United States to be registered before implementation.  This would mean that the full scope, goals, aims and methods of the research would be available publicly on a website beforehand, and any published results stemming from the trial could be compared to the original protocol.  In addition, if no results were published, this could be questioned.(Zarin et al., 2016, 2017)

Even though this law is applicable to the United States only, its strength is reinforced by the practice of medical journals’ requiring authors to have previously registered their trials; manuscripts that do not cite a protocol registration number are rejected even before peer-review.

Although researchers can register their trials on many such sites, one of the largest sites on which these trials are registered is ClinicalTrials.gov (https://clinicaltrials.gov/) on which researchers register clinical trials (or interventional studies) and observational studies.(ClinicalTrials.gov, 2018) 

This solution has not entirely solved the problems of publication bias and outcome switching: industry-sponsored journals do not always follow suit, the legal language has frequently been misunderstood, (Zarin et al., 2016) and sometimes trials are registered well after they have begun (Zarin et al., 2017).  Nevertheless, it has been a successful step in the right direction, and this success will increase as more medical journals adopt the practice.

Although publication bias and outcome switching do not appear to be well-studied in medical education, publication bias has been noted in social sciences for more than 60 years (Sterling, 1959; Franco, Malhorta and Simonvits, 2014), and there is no reason to believe that it does not exist in medical education.  In spite of this, no such protocol registration sites exist for medical education research, and medical education journals do not generally require prior registration of studies.  

One might question the need for such a medical education site, on the grounds that few medical education studies are trials, and that the stakes for medical education are not as high as they are for medical research.  I hold, however, that these arguments are weak:

• Similar protocol registries already exist for several social sciences, including general education.(Anderson, Spybrook and Maynard, 2019)  Medical education’s absence is distinctly, and embarrassingly, noteworthy.

• Although the name MedicalTrials.gov implies randomised, controlled trials, the site accepts registration of any interventional and observational study.  Many medical education studies involve new interventions or implementations of interventions under different circumstances; if medical education practitioners read only about successful interventions and are not aware of the many failures, the result will be further failed attempts to implement medical education interventions that were doomed from the start.

• In medical research, there is no doubt that the stakes are high: patients’ well-being and their lives.  But it is not true to say that, in medical education research, the stakes are not as high.  They are identical; they are just not as obvious.  The chain is longer, but attempting medical education interventions that were already doomed to failure is poor teaching, will lead to poorly-prepared medical students, and, in turn, will lead to poorly-trained health professionals; ultimately, there will be a negative impact on patients’ well-being and their lives.  The chain is longer, but it is no less sure.

• Such a registry is in keeping with current publication trends in all fields towards far greater openness and transparency.

Finally, it is with some embarrassment that we note that, although medical journal editors and publishers had known about publication bias and outcome switching for many years, it was only after legal pressure in the form of FDAAA 801 that these journals required pre-registration.  It would be a sad day for medical education if, in spite of the obvious existence and impact of these on medical education and healthcare, medical education journals were to wait for the lawyers to instruct them on what is, essentially, an ethical decision. 

In response to this need, the following is proposed:

• The creation of a website, MedEdTrials.org, in which all medical education researchers can pre-register any medical education interventional study.  (For convenience, I refer to “medical”, but this site should be applicable to all human and animal health sciences education).  Although the details of the registration process and requirements for each intervention registration will need to be established by all stake-holders, especially medical education journal editors (see next point), the site’s characteristics should be:
  • Free registration of researchers and protocols.  (Researchers register themselves with a user name and password in much the same way that they would register on any other web site).
  • Researchers are identified by unique, publicly verifiable identifiers such as ORCID (https://orcid.org/) numbers.
  • The project lead researcher registers the intervention before performing the research.  Ideally, this should be done after ethics approval (if required).  The project must be allocated a unique identifier.
  • When fields have been completed in the initial registration, they cannot be edited, although extra notes and documents can be added for clarification.
  • Annual updates on the progress of the research until publication, or a decision to not publish.  If the decision is made to not publish, reasons must be given.
  • All entries to be date- and time-stamped.
  • Free and open access of the information to the public, without registration requirements.

• Agreement by editors of all medical education journals (and any journals publishing medical education manuscripts) that any interventional- and observational-study manuscripts submitted for publication should indicate the unique project identifier issued by MedEdTrials.org (or any other similar website), and should provide a link to the protocol.  Where authors have deviated from the protocol, this should be suitably justified, and part of the editorial or reviewing process should include a comparison of the protocol with the manuscript.  Again, this includes all human and animal health sciences education publications.

• Institutional Ethics’ Committees and Review Boards (IRBs) should be encouraged to request their researchers to stipulate where they intend to register their intervention.

There are two limitations of the registry that need to be borne in mind:

First, because publication bias is a journal issue, having the registry will not altogether stop the bias.  Having the registry should, however, reduce the impact of publication bias, as educationists and the general public will be able to find a public record of other research in which the intervention has been attempted but not published; as a result, they will have a more balanced picture of the effectiveness of the intervention.

Second, educational research can take many different forms, and not all educational research follows a somewhat positivist pattern of Aims – Intervention – Results; therefore, in the same way that not all educational research requires ethics’ approval, not all research should require protocol registration.  Whether or not a study requires registration would depend upon its structure, and will need to be determined by the researchers. 

Publication bias and outcome switching have long been recognised as having a harmful impact on medical research and healthcare.  These problems have been partially addressed by legislation requiring pre-registration of medical interventional and observational studies’ protocols, and agreements by medical journal editors to publish only manuscripts that indicate such registration. 

This paper calls for a similar registration site and agreement for medical education.  Medical educators should not wait for legislation, but should embark on this venture because it is an ethical necessity required to enhance medical education scholarship, and will impact positively on all aspects of healthcare education, and, ultimately, healthcare delivery.