Commentary
Open Access

MedEdTrials: Protocol registration for medical education research [Version 2]

Ken Masters[1][a]

Institution: 1. Sultan Qaboos University
Corresponding Author: Dr Ken Masters ([email protected])
Categories: Professionalism/Ethics, Scholarship/Publishing, Research in Health Professions Education
Published Date: 05/08/2019

Author Revision Notes

• Reviewer 1 [FS]: No changes required.
• Reviewer 2 [BA]: (1) How to register and whom to register? A parenthetical explanation has now been added to the first bullet point under the MedEdTrials Heading. (2) Can the researchers individually also can register themselves before proceeding for the medical education researches? Bullet point three under the same heading has now been modified to make this clearer.
• Reviewer 3 [GDR]: (1) Will not stop publication bias. (2) Very Positivist approach to the paper. Rather than break the overall flow of the paper, a new section (entitled “Limitations and scope”) has been added, in which these two points are addressed.
• Reviewer 4 [HH]: No changes required.

Abstract

Publication bias (the tendency for studies with positive results to be more easily published than studies with null or negative results) and outcome switching (the adjustment of study aims to match results) have long been problematic in medical research.  Recent legislation, protocol-registration sites, and agreements by medical journals have led to a reduction of these problems in medical research.  In medical education specifically, however, there is no such legislation, registration, or agreement.  This paper argues for the creation of such a registration site and agreement by medical education journals as a matter of ethical necessity. 

 

Keywords: publication bias; outcome switching; medical education; medical education journals; negative results; Clinicaltrials.gov; Study 329; ethics

Introduction

Problems with evidence: publication bias and outcome switching

 

Researchers and practitioners in evidence-based fields rely upon quality evidence, and attempt to follow recommendations indicated by successful interventional and observational studies.  Two problems that prevent the true picture of evidence from reaching a wider audience are publication bias and outcome switching.

 

Publication bias is the tendency of research studies with positive results to be more easily published than research studies with null or negative results.(Sterling, 1959; Hopewell et al., 2009; Joober et al., 2012; Hedin et al., 2016)  Generally, academic journals favour studies with positive results, and researchers are also loath to publish research that has failed, especially embarrassingly so.  Because of the lack of comparative, unsuccessful studies on an intervention, readers and practitioners will infer an exaggerated and false level of viability of that intervention.  In medical research, publication bias is wide-spread and increasing.(Hopewell et al., 2009; Hedin et al., 2016; Mlinarić, Horvat and Smolčić, 2017)

 

Outcome switching occurs when researchers perform an intervention, and then receive results that do not support the original research aims, and so they ignore, or retrospectively alter, their research aims to match those results.  In so doing, they obscure the negative results and exaggerate (or totally falsify) the success of their intervention.  In medical research, there are many examples of outcome switching (Goldacre et al., 2016); perhaps the most notorious example is Study 329, dealing with the drug Paxil.(Doshi, 2015; Le Noury et al., 2015; Mulder, Rucklidge and Toop, 2016)

 

The problems in medical research and practice

 

Because evidence-based medicine relies on published research as that evidence, and publication bias and outcome switching results in tainted evidence, the impact on patient care can be devastating.(Whittington et al., 2004; Joober et al., 2012; Goldacre et al., 2016)

 

In response to these problems, Section 801 of The Food and Drug Administration Amendments Act of 2007, known as FDAAA 801,(FDA, 2007) established the legal necessity for protocols of all clinical trials in the United States to be registered before implementation.  This would mean that the full scope, goals, aims and methods of the research would be available publicly on a website beforehand, and any published results stemming from the trial could be compared to the original protocol.  In addition, if no results were published, this could be questioned.(Zarin et al., 2016, 2017)

 

Even though this law is applicable to the United States only, its strength is reinforced by the practice of medical journals’ requiring authors to have previously registered their trials; manuscripts that do not cite a protocol registration number are rejected even before peer-review.

 

Although researchers can register their trials on many such sites, one of the largest sites on which these trials are registered is ClinicalTrials.gov (https://clinicaltrials.gov/) on which researchers register clinical trials (or interventional studies) and observational studies.(ClinicalTrials.gov, 2018) 

 

This solution has not entirely solved the problems of publication bias and outcome switching: industry-sponsored journals do not always follow suit, the legal language has frequently been misunderstood, (Zarin et al., 2016) and sometimes trials are registered well after they have begun (Zarin et al., 2017).  Nevertheless, it has been a successful step in the right direction, and this success will increase as more medical journals adopt the practice.

Medical Education

Although publication bias and outcome switching do not appear to be well-studied in medical education, publication bias has been noted in social sciences for more than 60 years (Sterling, 1959; Franco, Malhorta and Simonvits, 2014), and there is no reason to believe that it does not exist in medical education.  In spite of this, no such protocol registration sites exist for medical education research, and medical education journals do not generally require prior registration of studies.  

 

One might question the need for such a medical education site, on the grounds that few medical education studies are trials, and that the stakes for medical education are not as high as they are for medical research.  I hold, however, that these arguments are weak:

 

  • Similar protocol registries already exist for several social sciences, including general education.(Anderson, Spybrook and Maynard, 2019)  Medical education’s absence is distinctly, and embarrassingly, noteworthy.
  • Although the name MedicalTrials.gov implies randomised, controlled trials, the site accepts registration of any interventional and observational study.  Many medical education studies involve new interventions or implementations of interventions under different circumstances; if medical education practitioners read only about successful interventions and are not aware of the many failures, the result will be further failed attempts to implement medical education interventions that were doomed from the start.
  • In medical research, there is no doubt that the stakes are high: patients’ well-being and their lives.  But it is not true to say that, in medical education research, the stakes are not as high.  They are identical; they are just not as obvious.  The chain is longer, but attempting medical education interventions that were already doomed to failure is poor teaching, will lead to poorly-prepared medical students, and, in turn, will lead to poorly-trained health professionals; ultimately, there will be a negative impact on patients’ well-being and their lives.  The chain is longer, but it is no less sure.
  • Such a registry is in keeping with current publication trends in all fields towards far greater openness and transparency.

Finally, it is with some embarrassment that we note that, although medical journal editors and publishers had known about publication bias and outcome switching for many years, it was only after legal pressure in the form of FDAAA 801 that these journals required pre-registration.  It would be a sad day for medical education if, in spite of the obvious existence and impact of these on medical education and healthcare, medical education journals were to wait for the lawyers to instruct them on what is, essentially, an ethical decision. 

MedEdTrials

In response to this need, the following is proposed:

  • The creation of a website, MedEdTrials.org, in which all medical education researchers can pre-register any medical education interventional study.  (For convenience, I refer to “medical”, but this site should be applicable to all human and animal health sciences education).  Although the details of the registration process and requirements for each intervention registration will need to be established by all stake-holders, especially medical education journal editors (see next point), the site’s characteristics should be:
    • Free registration of researchers and protocols.  (Researchers register themselves with a user name and password in much the same way that they would register on any other web site).
    • Researchers are identified by unique, publicly verifiable identifiers such as ORCID (https://orcid.org/) numbers.
    • The project lead researcher registers the intervention before performing the research.  Ideally, this should be done after ethics approval (if required).  The project must be allocated a unique identifier.
    • When fields have been completed in the initial registration, they cannot be edited, although extra notes and documents can be added for clarification.
    • Annual updates on the progress of the research until publication, or a decision to not publish.  If the decision is made to not publish, reasons must be given.
    • All entries to be date- and time-stamped.
    • Free and open access of the information to the public, without registration requirements.
  • Agreement by editors of all medical education journals (and any journals publishing medical education manuscripts) that any interventional- and observational-study manuscripts submitted for publication should indicate the unique project identifier issued by MedEdTrials.org (or any other similar website), and should provide a link to the protocol.  Where authors have deviated from the protocol, this should be suitably justified, and part of the editorial or reviewing process should include a comparison of the protocol with the manuscript.  Again, this includes all human and animal health sciences education publications.
  • Institutional Ethics’ Committees and Review Boards (IRBs) should be encouraged to request their researchers to stipulate where they intend to register their intervention.

Limitations and scope

There are two limitations of the registry that need to be borne in mind:

 

First, because publication bias is a journal issue, having the registry will not altogether stop the bias.  Having the registry should, however, reduce the impact of publication bias, as educationists and the general public will be able to find a public record of other research in which the intervention has been attempted but not published; as a result, they will have a more balanced picture of the effectiveness of the intervention.

 

Second, educational research can take many different forms, and not all educational research follows a somewhat positivist pattern of Aims – Intervention – Results; therefore, in the same way that not all educational research requires ethics’ approval, not all research should require protocol registration.  Whether or not a study requires registration would depend upon its structure, and will need to be determined by the researchers. 

Conclusion

Publication bias and outcome switching have long been recognised as having a harmful impact on medical research and healthcare.  These problems have been partially addressed by legislation requiring pre-registration of medical interventional and observational studies’ protocols, and agreements by medical journal editors to publish only manuscripts that indicate such registration. 

 

This paper calls for a similar registration site and agreement for medical education.  Medical educators should not wait for legislation, but should embark on this venture because it is an ethical necessity required to enhance medical education scholarship, and will impact positively on all aspects of healthcare education, and, ultimately, healthcare delivery.

Take Home Messages

  • Publication bias and outcome switching have long been problematic in medical research. 
  • Recent legislation, protocol-registration sites, and agreements by medical journals have led to a reduction of these problems in medical research. 
  • In medical education, there is no such legislation, registration, or agreement. 
  • This paper argues for the creation of such a registration site and agreement by medical education journals as a matter of ethical necessity. 

Notes On Contributors

Ken Masters, PhD, FDE is Associate Professor of Medical Informatics at Sultan Qaboos University, Sultanate of Oman.  He has been involved in education and medical education research for some 20 years.  ORCID: https://orcid.org/0000-0003-3425-5020

Acknowledgements

I wish to thank the reviewers of the previous version of this paper for their valuable comments.

Bibliography/References

Anderson, D., Spybrook, J. and Maynard, R. (2019) ‘REES: A registry of efficacy and effectiveness studies in education’, Educational Researcher, 48(1), pp. 45–50. https://doi.org/10.3102/0013189X18810513.

 

ClinicalTrials.gov (2018) Why Should I Register and Submit Results? Available at: https://clinicaltrials.gov/ct2/manage-recs/background (Accessed: 20 May 2019).

 

Doshi, P. (2015) ‘No correction, no retraction, no apology, no comment: Paroxetine trial reanalysis raises questions about institutional responsibility’, BMJ (Online), 351(September), p. h4629. https://doi.org/10.1136/bmj.h4629.

 

FDA (2007) Food and Drug Administration Amendments Act of 2007, Public Law 110-85. USA. Available at: http://www.gpo.gov/fdsys/pkg/PLAW-110publ85/pdf/PLAW-110publ85.pdf (Accessed: 13 May 2019).

 

Franco, A., Malhorta, N. and Simonvits, G. (2014) ‘Publication bias in the social sciences: Unlocking the file drawer’, Science, 345(6203), pp. 1502–1506. https://doi.org/10.1126/science.1255484.

 

Goldacre, B., Drysdale, H., Powell-Smith, A., Dale, A., et al. (2016) The COMPare Trials Project. Available at: http://compare-trials.org/ (Accessed: 21 May 2019).

 

Hedin, R. J., Umberham, B. A., Detweiler, B. N., Kollmorgen, L., et al. (2016) ‘Publication Bias and Nonreporting Found in Majority of Systematic Reviews and Meta-analyses in Anesthesiology Journals’, Anesthesia and Analgesia, 123(4), pp. 1018–1025. https://doi.org/10.1213/ANE.0000000000001452.

 

Hopewell, S., Loudon, K., Mj, C., Ad, O., et al. (2009) ‘Publication bias in clinical trials due to statistical significance or direction of trial results (Review)’, Cochrane Database of Systematic Reviews, (1), p. MR000006. https://doi.org/10.1002/14651858.MR000006.pub3.

 

Joober, R., Schmitz, N., Annable, L. and Boksa, P. (2012) ‘Publication bias: What are the challenges and can they be overcome?’, Journal of Psychiatry and Neuroscience, 37(3), pp. 149–152. https://doi.org/10.1503/jpn.120065.

 

Mlinarić, A., Horvat, M. and Smolčić, V. Š. (2017) ‘Dealing with the positive publication bias: Why you should really publish your negative results’, Biochemia Medica, 27(3), p. 030201. https://doi.org/10.11613/BM.2017.030201.

 

Mulder, R., Rucklidge, J. J. and Toop, L. (2016) ‘Restoring Study 329: Paroxetine neither effective nor safe for adolescents’, Australian and New Zealand Journal of Psychiatry, 50(9), pp. 922–923. https://doi.org/10.1177/0004867416657412.

 

Le Noury, J., Nardo, J. M., Healy, D., Jureidini, J., et al. (2015) ‘Restoring Study 329: Eficacy and harms of paroxetine and imipramine in treatment of major depression in adolescence’, BMJ (Online), 351, p. h4320. https://doi.org/10.1136/bmj.h4320.

 

Sterling, T. D. (1959) ‘Publication decisions and their possible effects on inferences drawn from tests of significance - or vice versa’, Journal of the American Statistical Association, 54(285), pp. 30–34. https://doi.org/10.2307/2282137.

 

Whittington, C. J., Kendall, T., Fonagy, P., Cottrell, D., et al. (2004) ‘Selective serotonin reuptake inhibitors in childhood depression: systematic review of published versus unpublished data’, Lancet, 363(9418), pp. 1341–1345. https://doi.org/10.1016/S0140-6736(04)16043-1.

 

Zarin, D. A., Tse, T., Ph, D., Williams, R. J., et al. (2016) ‘Trial reporting in ClinicalTrials.gov — The Final Rule’, New England Journal of Medicine, 375(20), pp. 1998–2004. https://doi.org/10.1056/NEJMsr1611785.

 

Zarin, D. A., Tse, T., Williams, R. J. and Rajakannan, T. (2017) ‘Update on trial registration 11 years after the ICMJE Policy was established’, New England Journal of Medicine, 376(4), pp. 383–391. https://doi.org/10.1056/nejmsr1601330.

Appendices

None.

Declarations

There are some conflicts of interest:
I am Associate Editor of MedEdPublish. This paper has been processed through the normal channels.
This has been published under Creative Commons "CC BY-SA 4.0" (https://creativecommons.org/licenses/by-sa/4.0/)

Ethics Statement

This is an editorial. No human or animal subjects or tissue were used, so no ethics approval is required.

External Funding

This article has not had any External Funding

Reviews

Please Login or Register an Account before submitting a Review

Dave Wilson - (07/08/2019)
/
An interesting idea, but fails to convince in terms of medical education. As the author points out neither publication bias or outcome switching has been investigated in terms of medical education. For this reason, the establishment of a registry at this stage would seem to be premature. Given the ability to provide online supplemental information for readerships of most education journals, would it be simpler to ask for the authors to upload the Research Ethics approval form?
Possible Conflict of Interest:

I have no conflicts of interest

Felix Silwimba - (06/08/2019)
/
this paper has provided me with a better understanding of research governance. it is educative and a commendable piece of work.
BALAJI ARUMUGAM - (06/08/2019) Panel Member Icon
/
Thanks for the invite once again to review this article.
Good wok by the authors and now the registration process for MedEd trials is clear and easy user friendly which will enhance the researchers to register. I hope the publication bias will definitely go down considerably if its ethically followed and properly registered under this weblink.
Congrats and Thanks
Possible Conflict of Interest:

NIL

Tan Nguyen - (06/08/2019) Panel Member Icon
/
I found this paper highly commendable. There certaintly has been maturity in the medical education research space. As noted, there are limitations to academic work that often does not require ethics approval. Furthermore, protocols, whilst offers transparency, may require amendments during the research being undertaken for various administrative or practical reasons. However, there is value in such an approach. It also enables the reduction in duplicating work that may be performed elsewhere and can foster international collaboration. We should also be mindful to ensure academics are not burdened by the process.