Research article
Open Access

Validating an adapted questionnaire to determine perceptions of the clinical learning environment

Koot Kotze[2], Elize Archer[2]

Institution: 2. Centre for Health Professions Education (CHPE), Faculty of Medicine and Health Sciences, Stellenbosch University
Corresponding Author: Dr Kotze (
Categories: Research in Medical Education, Teachers/Trainers (including Faculty Development), Curriculum Evaluation/Quality Assurance/Accreditation
Published Date: 01/06/2017



The clinical learning environment is a key component of undergraduate medical students’ curriculum. At our institution there was a need for an electronic instrument to evaluate student perceptions of their clinical learning environment.


The Uppsala Instrument, which had previously been used to improve student perceptions of their clinical learning environments, was adapted for the context of a South African Academic Hospital by a panel of experts and then completed by a group of students. The results were then used to determine the validity and reliability of the instrument using principal component analysis.


The instrument proved to be highly reliable at measuring a single underlying component, ‘perception of the clinical learning environment‘, and due to this, further determination of validity was limited. The findings were similar to those reached by other authors’ studies of instruments evaluating the clinical learning environment in that most variance in responses were attributable to a single component.


The fact that the instrument measures the clinical environment as a whole is useful, however it does not allow for deeper analysis of educational constructs within the clinical learning environment.

Keywords: Questionnaire Validation, Clinical Learning Environment, Medical Education, Measuring Instruments, Clinical Teaching and Learning


The clinical learning environment (CLE) forms an important part of all health sciences curricula. Learning in the clinical learning environment differs from the formal curriculum: there is no clear timetable, with learning often happening in an unstructured way when as the opportunity presents (Swanwick, 2014).

Through obtaining student evaluations of the clinical learning environment, clinical educators can enhance the curricula of institutions (AlHaqwi, Kuntze, & van der Molen, 2014; Snell et al., 2000). Prior to this study, our institution did not have an evaluation instrument aimed at evaluating student perceptions of the CLE.

Various measuring instruments have been developed for the evaluation of clinical learning environments, as well as clinical teaching (AlHaqwi et al., 2014; Stalmeijer, Dolmans, Wolfhagen, Muijtjens, & Scherpbier, 2008), however, there is no evidence that any of these are objectively superior to others (Beckman, Ghosh, Cook, Erwin, & Mandrekar, 2004). In terms of selecting a particular evaluation instrument, previous researchers have suggested that the instrument should be “consistent with the culture of the organization, acceptable to teachers, easy to administer, and applicable to all levels of teachers” (Nation, Carmichael, Fidler, & Violato, 2011).

Several evaluation instruments have been developed specifically with the clinical learning environment in mind, for example, the Dundee Ready Education Environment Measure (DREEM) ( Roff et al., 1997), the Postgraduate Hospital Educational Environment Measure (PHEEM) ( Roff, McAleer, & Skinner, 2005), the Clinical Learning Environment Inventory (CLEI) (Chan, 2002), and the Clinical Learning Evaluation Questionnaire (CLEQ) (AlHaqwi et al., 2014). In addition, there are some instruments that are aimed at evaluating clinical teachers and clinical teaching, such as the Stanford Faculty Development Program Clinical Teaching Framework (Hesketh et al., 2001), Clinical Teaching Effectiveness Index (CTEI) (Van der Hem-Stokroos, Van der Vleuten, Daelmans, Haarman, & Scherpbier, 2005) and the Clinical Tutor Evaluation Questionnaire (CTEQ) (Kember & Ginns, 2012; Nation et al., 2011).

It is important to acknowledge that the terms “clinical teaching” and “clinical learning environment” are used somewhat interchangeably in the literature  (AlHaqwi et al., 2014), and clear, widely accepted definitions for either are not forthcoming. For the purpose of this study, we refer to the clinical learning environment as a concept that encompasses clinical teaching, but also  the “quality of supervision, the quality of teachers and facilities and atmosphere” (Boor et al., 2007, p. 27).

Some of the difficulties that have been observed when evaluating clinical teaching is that many clinical teachers do not supervise enough students at one time or do not supervise those same students over a sufficiently long period of time in order to be evaluated accurately (Kihlberg, Perzon, Gedeborg, Blomqvist, & Johansson, 2011). This is certainly the case at our institution where students rotate through various departments and wards, and have numerous teachers, each for a relatively short time periods. Due to these difficulties, some authors have elected to evaluate clinical rotations (or departments) using clinical teaching evaluation instruments (Kihlberg et al., 2011) in a context similar to where clinical learning environment evaluation instruments have been used (Boor et al., 2007)

When using an instrument to gather information, the length of a questionnaire has been found to be inversely correlated with the response rate, therefore this was taken into consideration when selecting an evaluation instrument suitable (Bogen, 1994). For example, the CLEQ consists of 40 items, whereas the instrument developed for use at Uppsala University, Sweden, consists of only 10 items (Kihlberg et al., 2011).

Ultimately, the decision was made to use the latter instrument mainly because of its focus, length, and the success which the authors reported they had with improved student ratings of various departments, after providing the results of the instrument to the relevant departments and faculty leaders (Lewisson, Hellgren, & Johansson, 2013).

The Uppsala Instrument is based on the Stanford Faculty Development Program Clinical Teaching Framework (SFDPCTF), which has been extensively researched with large sample sizes at different sites, and found to be both valid, and reliable (Nation et al., 2011). One limitation of the Uppsala Instrument, however, is that, to our knowledge, it has not  been tested for validity and reliability since its adaptation (Kihlberg et al., 2011).

Each of the 10 items that forms part of the Uppsala Instrument aims to assess one or more domains (interchangeably referred to by the authors as “behavioural anchors”) (Kihlberg et al., 2011). It is worth noting here that the meaningfulness of the underlying domains for instruments on the clinical learning environment and clinical teaching has been questioned before (AlHaqwi et al., 2014), and that the principal component analysis of at least one other instrument has failed to support this notion entirely. Instead it was found that the instrument in question had only one underlying dimension (Boor et al., 2007).

The 10 questions originally used in the Uppsala evaluation instrument (translated from the original version in Swedish), as well as the proposed “domain” or “behavioural anchor” of each original question is displayed in Table 1.

Our study set out to determine the validity and reliability of the adapted questionnaire, aimed at measuring student perceptions of the clinical learning environment, after being adapted to suit the context of a South African Academic Hospital.


The Uppsala Instrument was reviewed and commented on by five experts in medical education from within our faculty for adaptation to our setting.

In the second phase of review, final year medical students were invited via email and social media (on a closed class Facebook group) to complete the newly adapted questionnaire anonymously, via Google Forms. The respondents could use any device capable of accessing the web page. Respondents selected the clinical rotation they wished to evaluate and answered each question on a Likert scale from 1 to 6. Table 1 presents the original items, as well as the expanded questionnaire which was finally used.

Next the principal component analysis was performed and results were interpreted in light of the steps proposed by Schönrock-Adema et al (2009). A scree plot of eigenvalues was generated, and considered alongside an eigenvalue cut-off of >1.5 and “the proportion of variance accounted for” minimum was set at 5%. Interpretability criteria as proposed by the above authors were also applied in identifying the underlying component, including requiring a component to load (show correlation with the given component) with a value of at least 0.4 on the component matrix for at least three questions. Convergent and divergent validity could further have been determined if multiple components were identified.

Following this the Cronbach’s alpha was determined for all Likert scale questions with the intention to determine Cronbach’s alpha for the identified component(s).


Figure 1: Schematic illustration to explain the process



Participants were final year medical students who were invited to participate in the study via email, social media and class visits. Responses were collected during 2015. Students were asked to complete the electronic questionnaire after completion of every clinical rotation between January and July.

Ethical Approval

This study was approved by the Stellenbosch University Health Research Ethics Committee and was conducted in accordance with the international declaration of Helsinki (S14/09/185) and also received institutional permission. Participants provided informed consent for the study. Participants were informed that the purpose of the study was to validate and determine the reliability of a questionnaire aimed at evaluating the clinical learning environment. No identifying information was gathered from student respondents. Furthermore, only the researcher had access to the password protected data sets which were generated. These data sets were sent to the biostatistician using password protected email accounts.


Face Validity

Expert feedback on the Uppsala Instrument’s face validity expanded the questionnaire from 10 items to 20, separating some of the original questions into more specific questions, several alterations to phrasings, as well as proposing a few new questions. (Table 1, columns 3 and 4)

Principal Component Analysis

The adapted questionnaire received 168 complete responses that could be used for analysis (n=168). The principal component analysis revealed only one component which satisfied the set criteria (Schönrock-Adema, Heijne-Penninga, Van Hell, & Cohen-Schotanus, 2009). The scree plot below supports this notion, as the slope after the first item is gradual. The point of transition does not count as a component (Brown, 2009b).

Figure 2: A scree plot of the eigenvalues for the data. The transition is clearly at the second point.


This first component accounted for 54.52% of variance and had an eigenvalue of 8.72. A second component accounted for 6.26% of variance, but had an eigenvalue of only 1 (Table 2). The first component loaded significantly on all questions, ranging between 0.538 and 0.853. The second component did not load more than 0.4 on any question. (Table 3)

After varimax rotation with Kaiser extraction, instead of attaining simple structure, as is the intention with rotation (Brown, 2009a), multiple questionnaire items now loaded on both the first (accepted) and second (rejected) component (table 4). From this it could be concluded that the second component did not represent an interpretable component.

Therefore, the PCA failed to support any of the proposed underlying components (domains or “behavioural anchors”) for the original Uppsala Instrument, but instead showed that most variance resulted from a single underlying component (component 1).


The Crohnbach’s alpha coefficient for Likert-scale type questions was 0.943. Because all Likert-scale questions belonged to the same component, the alpha for this component was 0.943, indicating very high reliability, and possible redundancy of certain questions.

This further supported the finding that a single component explained most of the responses, and that the original proposed components (domains), were not demonstrable.


The results described showed that the items addressed in the Uppsala Instrument was acceptable to clinical teaching experts although  ten  additional questions (including demographic information) and the expansion of certain original items into clearer more directed  questions were added. The problem, however, was that some of the added questions could not easily be answered on a Likert scale.

The principal component analysis failed to support the presupposition that this questionnaire meaningfully assessed several underlying components or domains, as was suggested in previous studies utilizing the Uppsala Instrument (Kihlberg et al., 2011; Lewisson et al., 2013). Rather, it seems that the questionnaire only addressed a single component; namely perception of the clinical learning environment. While this component is vague the evidence presented above is sufficiently rigorous to suggest that this is what the instrument actually measures.

These results found are somewhat similar to that published by Boor et al.(2007) while investigating the Postgraduate Hospital Educational Environment Measure (PHEEM), where it was also found that the variance in answers to the 40 items was due to a single component. The authors of the Clinical Learning Evaluation Questionnaire (CLEQ) made a similar finding after they did their principal component analysis. They found that all the identified components (referred to as “factors” in their study), still related to each other (as there was no divergent validity), raising the question of whether it is “meaningful to differentiate between these”.(AlHaqwi et al., 2014, p. 6) It could also mean that each of the proposed domains are very closely related in the clinical learning environment, and that high ratings on one necessarily mean high ratings in all others.  To use the “behavioural anchors” originally proposed by Lewisson et al.(2013) as an example, a rotation which spends time on “evaluation and feedback”, would probably have  “good control of session” and “communication of goals”, which in turn will inspire “self-directed learning” in students (to quote the proposed domains from the original questionnaire). There is unfortunately no clear way of proving this hypothesis with the current data set.

The fact that the instrument only measures a single component, does not devalue its use; in fact the  original instrument which  was used effectively to improve student perceptions of the CLE (Lewisson et al., 2013). Therefore it can still be of use, but will unfortunately not allow for deeper analysis of educational constructs within the clinical learning environment.

Furthermore the instrument’s reliability compares well to that of instruments used to evaluate clinical teaching and clinical learning environments and the values determined are in line with those reported on other instruments based on the SFDPCTF instrument (with Crohnbach’s alpha coefficient ranging from 0.82-0.98) (Nation et al., 2011). In light of this, the high value obtained here is not surprising.


There is a need for evidence based instruments to evaluate learning in the clinical environment and the various factors which influence it. This study found that variance in responses to an instrument aimed at evaluating perceptions of the clinical learning environment (CLE) amongst students was due to a single underlying component. The findings were similar to the outcomes of other studies evaluating instruments aimed at evaluating the CLE. This calls into question whether meaningful study of various domains is possible using this or similar instruments.

Limitations of the study

Due to limitations in time, certain portions of the research were conducted in an expedited fashion. It would have been preferable to have used a formalized method such as the Delphi method for the face validity (or “expert validation”) step in the research.

The response rate was not as high what had initially been envisaged. Furthermore, given the software used (Google Forms), it was not possible to maintain participant anonymity at the same time as preventing multiple entries by an individual for the same rotation, so this created a possibility for students to answer more than once. This risk was taken due to the ease of distribution offered by this platform (over other platforms, which allow only one response per link to prevent multiple entries), and therefore the ability to distribute the questionnaire on social media. This medium allowed easy communication and distribution.

Take Home Messages

This study found that variance in responses to an instrument aimed at evaluating perceptions of the clinical learning environment amongst students was due to a single underlying component. This calls into question whether meaningful study of various domains is possible using this or similar instruments.

Notes On Contributors

Koot Kotze is a medical doctor with an interest in medical education. He conducted this research whilst a student at Stellenbosch University, under the Centre for Health Professions Education.

Elize Archer is a health professions educator with a special interest in clinical teaching and learning. She is the manager of the Simulation and Clinical Skills Centre at Stellenbosch University, South-Africa.


The authors would like to thank the Dr. Moleen Zunza from the Biostatistics Unit of the Centre for Evidence Based Health Care at the Faculty of Medicine and Health Sciences of Stellenbosch University, for providing expert input into the analysis of the data gathered. They would also like to thank the Stellenbosch University Undergraduate Conference Presentation Fund for funding the presentation of this research at a the South African Association of Health Educationalists Conference in 2016, allowing discussion and development of this research prior to publication.





AlHaqwi, A. I., Kuntze, J., & van der Molen, H. T. (2014). Development of the clinical learning evaluation questionnaire for undergraduate clinical education: factor structure, validity, and reliability study. BMC Medical Education, 14(1), 44.

Beckman, T. J., Ghosh, A. K., Cook, D. a, Erwin, P. J., & Mandrekar, J. N. (2004). How Reliable Are Assessments of Clinical Teaching? Journal of General Internal Medicine, 19(9), 971–977.

Bogen, K. (1994). The Effect of Questionnaire Length on Response Rates - A Review of the Literature. US Bureau of the Census.

Boor, K., Scheele, F., Van Der Vleuten, C. P. M., Scherpbier, a. J. J. a, Teunissen, P. W., & Sijtsma, K. (2007). Psychometric properties of an instrument to measure the clinical learning environment. Medical Education, 41(1), 92–99.

Brown, J. D. (2009a). Choosing the Right Type of Rotation in PCA and EFA. Shiken: JALT Testing & Evaluation SIG Newsletter, 13(November), 20–25.

Brown, J. D. (2009b). Questions and answers about language testing statistics : Choosing the Right Number of Components or Factors in PCA and EFA Choosing the Number of Components or Factors to include in a PCA or EFA Examples Illustrating the Five Stopping Rules. JALT Testing & Evaluation SIG, 13(May), 19–23.

Chan, D. (2002). Development of the clinical learning environment inventory: using the theoretical framework of learning environment studies to assess nursing students' perceptions of the hospital as a learning environment. Journal of Nursing Education, 41(2), 69–75.

Hesketh, E. A., Bagnall, G., Buckley, E. G., Friedman, M., Goodall, E., Harden, R. M. … Newton, R. (2001). A framework for developing excellence as a clinical educator. Medical Education, 35(6), 555–564.

Kember, D., & Ginns, P. (2012). Evaluating teaching and learning: A practical handbook for colleges, universities and the scholarship of teaching. Routledge.

Kihlberg, P., Perzon, M., Gedeborg, R., Blomqvist, P., & Johansson, J. (2011). Uniform Evaluation of Clinical Teaching - an Instrument for Specific Feedback and Cross Comparison between Departments, 1(2), 139–150.

Lewisson, N., Hellgren, L., & Johansson, J. (2013). Quality improvement in clinical teaching through student evaluations of rotations and feedback to departments. Medical Teacher, 35(10), 820–5.

Nation, J. G., Carmichael, E., Fidler, H., & Violato, C. (2011). The development of an instrument to assess clinical teaching with linkage to CanMEDS roles: A psychometric analysis. Medical Teacher, 33(6), e290–e296.

Roff, S., McAleer, S., Harden, R. M., Al-Qahtani, M., Ahmed, A. U., Deza, H. … Primparyon, P. (1997). Development and validation of the Dundee ready education environment measure (DREEM). Medical Teacher, 19(4), 295–299.

Roff, S., McAleer, S., & Skinner, A. (2005). Development and validation of an instrument to measure the postgraduate clinical learning and teaching educational environment for hospital-based junior doctors in the UK. Medical Teacher, 27(4), 326–331.

Schönrock-Adema, J., Heijne-Penninga, M., Van Hell, E. a, & Cohen-Schotanus, J. (2009). Necessary steps in factor analysis: enhancing validation studies of educational instruments. The PHEEM applied to clerks as an example. Medical Teacher, 31(6), e226–e232.

Snell, L., Tallett, S., Haist, S., Hays, R., Norcini, J., Prince, K. … Rowe, R. (2000). A review of the evaluation of clinical teaching: new perspectives and challenges. Medical Education, 34(10), 862–870.

Stalmeijer, R. E., Dolmans, D. H. J. M., Wolfhagen, I. H. a P., Muijtjens, A. M. M., & Scherpbier, A. J. J. a. (2008). The development of an instrument for evaluating clinical teachers: involving stakeholders to determine content validity. Medical Teacher, 30(8), e272-7.

Swanwick, T. (2014). Understanding medical education: Evidence, theory and practice. John Wiley & Sons.

Van der Hem-Stokroos, H. H., Van der Vleuten, C. P. M., Daelmans, H. E. M., Haarman, H. J. T. M., & Scherpbier, A. (2005). Reliability of the Clinical Teaching Effectiveness Instrument for appraisal of individual and group teaching quality. Medical Education, 39(9), 57.


Table 1: A comparison of the original questionnaire, as well as the original proposed “behavioural anchors”, with the adapted questionnaire which was circulated to the participants (column 3).

Proposed Domain/Construct/”Behavioural Anchor” for Original Questions

Original Question (from the Uppsala Questionnaire)

Adapted Question


New questions. No assigned domain.


Which rotation are you currently on or regarding which rotation would you like to provide feedback?

Demographic information was gathered to facilitate analysis.

What is/was the location of this rotation?

Eg. Tygerberg, Worcester, Paarl, etc.

What year did your rotation take place?

During which month(s) did your rotation take place?

Learning Climate

What were the introduction and the welcoming at the department like?

What was the introduction to the rotation like?

The original was split into two questions to better suit the setting

What was the reception at the ward like?

Communication of Goals

To what extent did you know what you were expected to learn during your rotation (both theoretical and

practical skills)?

To what extent did you know what you were supposed to learn during this rotation (both theoretical knowledge and practical skills)?

Wording altered for clarity

Communication of Goals

Were your supervisors familiar with the course curriculum, concerning theory and practical skills?

Did you feel your tutors know the course curriculum concerning theoretical knowledge?

"Tutors" refers to those who have supervised your learning.

Question split into two clearer questions and wording altered for clarity

Did you feel your tutors knew the course curriculum concerning practical/clinical skills?

Control of Session

To what extent was your rotation well planned (tasks during the day, schedule, framing, etc.)?

Did you get the impression that your rotation was well thought out and planned?

Eg. Tasks during the day, schedule, etc.

Wording altered for clarity

New question. No assigned domain.


On an average day, how long did you spend waiting?

For ward rounds, tutorials, etc

New question, at the suggestion of one expert, with the agreement of another

Learning Climate, Evaluation

Did your supervisor/s take an interest in you, assessing your knowledge and skills?

In your opinion, did your supervisors/s assess your theoretical knowledge?

Question split into two clearer questions

In your opinion, did your supervisors/s assess your practical/clinical skills?

New question. No assigned domain.


Did you receive feedback with regards to the above instances of assessment?

Question added for more information

Self-directed Learning

Did the rotation stimulate you to further learning in the field concerned?

Has the rotation stimulated further learning in the field concerned?

Wording altered for clarity

Control of Session, Understanding and Retention

Did you handle patients on your own (e.g. by leading rounds, suggesting further investigations/diagnoses/


Did you have the opportunity to evaluate patients and formulate management plans before discussion them with your supervisors?

(eg. By taking responsibility for history and examination and suggesting further


Question split in order to better suit the context

How many patients did you see on a typical day as part of your clinical duties?

(In clinics, ward rounds, tutorials, admissions)

Question added for more information

How many patients did you see on a typical day on your own initiative?

Question added for more information

Understanding and Retention

What was the teaching of practical skills like?

What was the teaching of practical skills like in the clinical area?

Wording altered for clarity


How often did you get feedback during your rotation (e.g. patient record notes, practical skills, etc.)?

How often did you get feedback concerning tasks during this rotation?

Patient record notes, practical skills, etc.

Wording altered for clarity

Learning Climate, Understanding and Retention

Did staff members, other than doctors, take an interest in your training?

Did you feel that staff members, other than doctors, took an interest in your training?

Wording altered for clarity

New Questions. No original question or proposed anchor


Overall, how would you rate your enjoyment of the rotation?

Question added for more information

New Questions. No original question or proposed anchor


Overall, how would you rate this rotation as a learning experience?

Question added for more information

New Questions. No original question or proposed anchor


Any other comments?

With regard to any of the above questions, on another topic, or with regards to the questionnaire?

Question added for more information


Table 2: Eigenvalues and total variance explained of the different components


Initial Eigenvalues

Extraction Sums of Squared Loadings


% of Variance

Cumulative %


% of Variance















































91 .606


























Table 3: Factor loadings for components 1 and 2 prior to Varimax rotation

Component Matrix"



















































. 821


Extraction method: Principle component analysis.

2 components extracted.


Table 4: Factor loadings after varimax rotation with Kaiser extraction, multiple items now loaded on both components. Given the state of factor loadings prior to rotation, it was decided that rotation did not help to achieve simple structure, and that the second component was not interpretable.

Rotated Component Matrixa






















































Extraction Method: Principal Component Analysis.

Rotation Method: Varimax with Kaiser Normalization.

Rotation conveged in 3 iterations


There are no conflicts of interest.
This has been published under Creative Commons "CC BY-SA 4.0" (


Please Login or Register an Account before submitting a Review

Anthony David.M - (18/06/2018)
A useful paper indeed. The adaptation of a validated questionnaire gives a lot of contextually to the instrument. Similar such adaptations help in making instruments relevant to particular contexts. This adapted instrument can be used to measure Clinical Learning Environments in other contexts too.
Trevor Gibbs - (04/08/2017) Panel Member Icon
I enjoyed reading this paper and although it reflected much of the work found in previous papers on the same subject, this does not mean that the results are not useful. The authors should be congratulated on their attempt to modify a validated instrument to suit a different educational environment, context and culture.
I suppose to me this paper further illustrates whether we stick with validated instruments or we move to a much more qualitative approach that may give a truer meaning to the value of the learning environment. As the authors state, a Delphi approach to enquiry might have been better in showing validity
I look forward to reading more from this group and think that the paper is a useful read to all those involved in the subject.
Peter Dieter - (04/08/2017) Panel Member Icon
Thanks for the comment that convinced me that I misinterpreted your article. I change my first comment after reading your comment and re-reading the article as follows: The article is very interesting for readers who work in a similar field and can be very useful for further research to determine the perceptions of the clinical learning environment. (3 stars).
Peter Dieter - (10/06/2017) Panel Member Icon
Unfortunately, the manuscript does not provide any new insights how to determine perceptions of the clinical environment. Moreover, the method used is doubtful, since no Delphi method was applied. In their take home message the authors themselves write: "This calls into question whether meaningful study of various Domains is possible using this or similar instruments."