Clinical and translational research in the United States is supported by numerous federal, industrial and academic organizations, and many other stakeholder groups (Callard, Rose and Wykes, 2012; Martinez et al., 2012; Trochim, Rubio and Thomas, 2013; Joosten et al., 2015). The NIH National Center for Advancing Clinical and Translational Science (NCATS) is a distinctive among them as it funds a broad network of research support centers, Clinical and Translational Science Awards (CTSAs), embedded in over 50 research intuitions located across the country (NCATS, 2018). A key strategic goal of these CTSAs regards their development of the entire clinical and translational workforce through dedicated research training programs (NCATS, 2017). Clinical Translational Research (CTR) training programs are providing highly valued instruction on relevant research skills and so demands for rigorous evaluations that demonstrate their impact on the research workforce also continue to grow (Bonham et al., 2012; Calvin-Naylor et al., 2017). 

Rigorous evaluations of CTR training programs typically require the periodic measurement of the research skills, or competencies, of the participants (Misso et al., 2016). However, the responsibility for evaluating the CTR programs often rests with small groups of investigators, research managers and administrators with little to no experience measuring research competencies despite having achieved many research accomplishments themselves. This work provides concrete steps they can take to integrate competency assessments into generalizable frameworks for developing and implementing rigorous evaluation plans of CTR training programs (Centers for Disease Control and Prevention, 1999; Trochim, Rubio and Thomas, 2013).

In this work we provide guidelines for CTR investigators and administrators responsible for evaluating research training programs in which the assessment of competence is a necessary component. The twelve guidelines discussed in this paper should inform the work of these individuals, but have been tailored to the role of the CTR training program directors. To ensure the relevance of these guidelines to this role, the authors carefully considered the typical demographics, job duties, motivations, skills and experiences of a typical administrator charged with guiding the evaluation and improvement of these training programs.  

Review team roles and expertise related to trainee’s professional development

The responsibility for evaluating CTR training programs is often carried out by personnel in a number of positions and roles in any given program. The collaborative review of these roles can be facilitated by creating personas, which are defined as archetypes with distinctive needs, goals, technical skills and professional characteristics (Adlin and Pruitt, 2010). Creating a persona that defines who will be conducting assessments can help program teams and stakeholders discuss and negotiate changes to the ways this work is distributed and coordinated. Table 1 provides examples of personas of clinical research professionals who are likely to share some responsibility for administering a CTR training program. This process can be carried about by CTR program leads and administrators to help focus collaborative efforts on measuring the research knowledge and skills of researchers in ways that enable a rigorous evaluation of a CTR training program to be conducted.

Integrate competency frameworks into evaluation planning

Map the existing CTR training curriculum to competency-based education (CBE) frameworks (Dilmore, Moore and Bjork, 2013). It may be necessary to partner with subject matter experts who understand CBE when navigating the competency mapping process. There are multiple evidence-based competency frameworks applicable to CTR education and training (NCATS, 2011; Calvin-Naylor et al., 2017; Sonstein et al., 2018). Table 2 shows training opportunities that have been mapped to one domain of an established CTR competency framework (Joint Task Force, 2018).

The products of this mapping exercise should be shared with programmatic stakeholders to facilitate the collection of their feedback about the breadth and depth of the existing or potential CTR training opportunities. Collecting stakeholder feedback about the content of CTR training programs is an essential first step in many guides to evaluating health research training programs, including the U.S. Center for Disease Control and Prevention’s (CDC) guide for the evaluation of public health programs (Centers for Disease Control and Prevention, 1999).

Engage stakeholders in identifying critical knowledge and skill outcomes 

Engage programmatic stakeholders in conversations focused on identifying the most important knowledge and skills imparted to CTR trainees as soon work on an evaluation plan has begun. Partner with instructional designers and evaluators to develop valid and measureable lists of competencies which can be prioritized in collaboration with various stakeholder groups. When identifying which specific stakeholder groups to involve in this phase of the evaluation process it is important to ensure that those with divergent recommendations of which CTR skills are in greatest need of development and assessment are included (Callard, Rose and Wykes, 2012; Martinez et al., 2012; Trochim, Rubio and Thomas, 2013; Joosten et al., 2015).

Diverse stakeholder feedback can be systematically collected and synthesized using a variety of methods ranging from standard survey methods to more intensive methods such as interviews, focus groups or Delphi panels (Brandon, 1998; Geist, 2010). The opportunities created to collect feedback can also be used to collect stakeholder opinions about other short-term outcomes, such as those regarding participant experiences and behaviors (Kirkpatrick and Kirkpatrick, 2006). The collection of stakeholder feedback on all types of programmatic outcomes, including the knowledge and skills accrued through the program, is a necessary to the development of rigorous program evaluations (Centers for Disease Control and Prevention, 1999; Trochim, Rubio and Thomas, 2013).

Develop models depicting the links between program operations and outcomes

Logic models should be created in order to enrich and advance conversations with stakeholders and other administrators about the operation and impact of a CTR training program. Logic models are figures that typically depict the relationship between key programmatic A) inputs, B) activities, C) outputs and D) outcomes, often using itemized lists arranged into columns under each of these headers (McLaughlin and Jordan GB, 1999). Some also include references to relevant important contextual or environmental factors affecting all or select progrmmatic outcomes. The choice of which of these elements to represent in the figure should be motivated by the need to visualize the links between programmatic operations and outcomes that are likely to be of greatest interest to key stakeholders. Many funders ask that logic models be included in program proposals, and the production of these figures are standard practice in the evaluation of training programs across the health sciences (Centers for Disease Control and Prevention, 1999; Centers for Disease Control and Prevention, 2018).

Whenever possible logic models for CTR training programs should include the identification of short-, intermediate-, and long-term outcomes. The acquisition of critical research knowledge and skills are often represented as outputs or short-term outcomes of these training programs in logic models. In contrast, downstream impacts such as the production of research grants and peer-reviewed publications are often represented as intermediate- or long-term outcomes. To enhance the efficiency of this process, include lists of competency domains, each of which cover sets of related competencies, as outcomes rather than numerous specific competencies in logic models. Exemplars of logic models that include lists of CTR competency domains have been published and can be used inform development of logic models for similar programs (Rubio et al., 2010). Figure 1 shows a logic model that can be used a basic template for enabling the planning, implementation and evaluation of a CTR training program.

Figure 1. Sample Logic Model for an Evaluation of a CTR Training Program

Distinguish programmatic outcomes used for formative and summative evaluation

The choice of when outcomes should be measured should be informed by the intent to use the results for formative or summative evaluations (Newman et al., 1995; Yudkowsky, Park and Downing, 2019). The results of formative evaluations are used to improve programs and projects during their implementation. Outcomes chosen for the purpose of formative evaluation are often represented as short- or intermediate-term outcomes in logic models. The results of summative evaluations are used to produce valid and objective measures of programmatic impact at the end of the implementation process. Footnotes can be added to logic models to differentiate the use of certain metrics for these two distinct purposes, as shown in the template above (Figure 1). 

Measures of knowledge and skill can be used for both the formative and summative evaluation of CTR training programs. The results of relevant pre-program assessments, and of assessments conducted during the course of a program, enable formative evaluation when they are used to improve the experience of actively participating trainees. For example, the results of subjective or objective skill assessments can be shared with respondents to inform their own development of individualized training plans or used to inform modifications to training curricula to address perceived or objectively-measured gaps in knowledge and skill. The results of post-program skill assessments enable summative evaluation when compared to relevant benchmarks, including the results of pre-program skill assessments or measures of skill acquisition produced by similar training programs (Newman et al., 1995; Centers for Disease Control and Prevention, 1999).

Select validated assessments to measure critical knowledge or skills

The validation of knowledge or skill assessment requires that evidence be marshalled to advance the claim that the assessment actually measures what it was designed to measure (Kane, 1992). Peer-reviewed publications that demonstrate the validity of a competency-based assessments will include the results of tests suggesting the assessment provides reliable and accurate measures of knowledge or understanding among the types of learners targeted by the training program. The use of validated competency-based assessments for program evaluation lends credibility to this work and typically requires fewer resources than does the development of novel assessments of knowledge or skill, although both types of assessments can be used simultaneously.

More validated self-assessments of CTR skills for investigators and research professionals have been published in recent years (Bakken, Sheridan and Carnes, 2003; Streetman et al., 2006; Bates et al., 2007; Ellis et al., 2007; Mullikin, Bakken and Betz, 2007; Lowe et al., 2008; Cruser et al., 2009; Cruser et al., 2010; Lipira et al., 2010; Murphy et al., 2010; Poloyac SM et al., 2011; Robinson et al., 2013; Ameredes et al., 2015; Awaisu et al., 2015; Robinson et al., 2015; Sonstein et al., 2016; Jeffe et al., 2017; Patel et al., 2018; Hornung et al., 2019). When choosing between validated assessments it is critical to select ones which are most closely aligned with the competency framework chosen for a given CTR program and which have been validated using learners with similar credentials and research experience to those participating in that program. Be sure to obtain all the necessary permissions from the creators of any validated assessments before using the instruments for evaluation purposes.

The design and purpose of a CTR training program may require the use of subjective and objective assessments. Subjective assessments through which participants rate their own knowledge or skills can provide, for example, valid measures of self-confidence in ones’ abilities, but have not been shown to correlate with the results of objective measures (Hodges, Regehr and Martin, 2001; Davis et al., 2006). Subjective and objective assessments of CTR knowledge and skill can be used simultaneously, but only the results of the latter type should be used to make inferences about the actual knowledge and skills currently possessed by CTR participants.

Clinical and translational research training programs that confer any level of certification which are formally recognized by professional institutions or organizations may require that objective assessments be used to verify the actual research capabilities of the graduates. In these cases, the specific assessments that should be used by CTR training programs may have already been identified by these associated professional groups. When multiple or conflicting assessments are required by these groups conversations with programmatic stakeholders will be needed before any final determination about the use of any competency-based assessment can be made. 

Estimate the time and effort required for implementing an evaluation plan

Evaluation plans take many different forms, but all detail how evaluation data will be collected, analyzed, reported and used (Trochim, Rubio and Thomas, 2013). The costs of implementing rigorous evaluation plans can be substantial, so it is essential that the resources available for this work are accurately estimated and budgeted for.  Some evaluation activities, such as the administration of publically-available skill assessments using free online platforms, have comparatively low costs. The costs of other evaluations involving focus groups for example, can be considerably more.

The effort required for each step of the evaluation plan can be estimated in a table (Table 3). When reviewing an evaluation plan, carefully consider the risks and benefits of proposed assessments and choose those that are feasible to administer given the resources available. Collaborate with stakeholders to ensure that key evaluation activities are aligned with other project timelines and plans that guide the allocation of financial, human, and institutional resources needed to implement a CTR training program (Centers for Disease Control and Prevention, 1999).

Train evaluation team members to collect assessment data in reliable ways.

Once an evaluation plan has been developed, and a formal evaluation team has been assembled, it is important that they understand the steps required for the reliable collection of data using competency-based assessments. For example, use of an CTR assessment of regulatory compliance may require that all persons administering the assessment be perfectly consistent with the use of the instrument as well as their subsequent scoring of individual’s performance. Even objective scoring systems include risks related to subjective interpretations (Van der Vleuten et al., 2010).  And research has shown that individuals in apparent positions of power may dissuade respondents from giving honest responses on tests of their knowledge or skills (Taut and Brauns, 2003; Van der Vleuten et al., 2010). When evaluating CTR training programs, it is essential that the appropriate members of the evaluation team receive and demonstrate their understanding of the relevant procedures.

Use technology platforms that best facilitate data collection, analysis and reporting

To maintain consistency, accuracy and accessibility of the assessment results, it is recommended to use one platform to administer, analyze and report survey results whenever possible. For example, common online survey platforms, such as REDCap, Qualtrics^xm, and SurveyMonkey®, provide automatically generated results reports that include options for basic analyses, such as descriptive statistics and cross-tabulations of response options. If necessary, the resultant data can also be extracted from these platforms so that further analyses can be performed.

The deliberate, coordinated use of a platform for data collection, management and analysis can also facilitate further validity tests that can be used to measure CTR knowledge and skills. Many standard quantitative validation tests are included in statistical software packages like STATA, SAS and R, including exploratory and confirmatory factor analysis (Levine, 2005; Osborne and Costello, 2005), which are commonly used to identify and to validate the accuracy of the competency domains that structure many competency-based assessments. While there are many valuable validity tests (Kane, 1992), these are the ones used most commonly to validate skill assessments. Software programs for qualitative analysis, such as Dedoose or NVivo can be used to code transcripts obtained from focus groups and interviews in ways that reveal patterned relationships between the content of participant responses.

Consult with subject matter experts to interpret assessment results

The results of competency-based assessments of CTR skill may not be readily interpretable, particularly when no established criteria or rubric is associated with the assessment. In fact, many validated assessments do not prescribe how the resultant scores should be interpreted by respondents to better understand their own training needs or by training program administrators to enable programmatic improvements. In these cases, it is important to consult with subject matter experts in clinical and translational research, psychometrics and statistical analysis while conducting analyses of the assessment results.

The need to consult with these types of subject matter experts is particularly acute with subjective and objective assessments are used to evaluate CTR training programs. Subjective tests of knowledge and skill have been shown to be poorly correlated with objective measures (Hodges, Regehr and Martin, 2001; Davis et al., 2006). There is evidence suggesting that while subjective measures of CTR knowledge and skill often increase between pre- and post-program tests the scores obtained through such objective tests do not increase at a similar rate (Ellis et al., 2007; Cruser et al., 2010). Measurement and educational experts can help ensure that assessment results are interpreted in ways that are justified by the design and administration of the assessment instrument.

Collect stakeholder feedback about options for programmatic improvement

An essential step of program evaluation involves sharing of evaluation results with stakeholder groups in order to facilitate the collection of their feedback about programmatic improvement (Wandersman et al., 2000). For example, in the four overlapping and iterative phases of the Plan, Do, Check, and Act (PDCA) quality improvement cycle, the third stage typically involves studying the outcomes of a given initiative in ways that enable the articulation of what was learned through the implementation process (Juran and DeFeo, 2010; Kleppinger and Ball, 2010). The involvement of stakeholders in this step of the process is critical to the rigorous evaluation of any CTR training program (Trochim, Rubio and Thomas, 2013).

Reports of evaluation results should be customized to speak to stakeholder subgroups whenever it is not possible or productive to share the same report with all of them. For example, stakeholders with distinctive interests in the scientific content or pedagogical approach of a CTR training program may be most interested in reports showing how the results of competency-based assessments are being used to help participants identify and address their personal research learning challenges (Chatterji, 2003). In contrast, stakeholders who value training programs as an institutional resource enabling the CTR enterprise may be more interested in the research careers or achievements of the participants (Frechtling and Sharp, 2002). Whenever possible thoroughly document stakeholder feedback so that it can be used to inform future discussions about programmatic improvement and impact.

The guidelines presented here are intended to support the work of all clinical research professionals who are charged with the administration and evaluation of CTR training programs. In particular, this work fulfills a need for guidelines that clinical research investigators and administrators can follow to integrate competency assessments into their evaluation plans. Doing will better enable research centers to collaborate with programmatic stakeholders efficiently and effectively in order to measure and improve the quality and impact of CTR training using the results of competency-based assessments of research knowledge and skill.