New education method or tool
Open Access

Med Stud Games as a Serious Game for Medical Education in Indonesia: a Validation Study

Azhar Rafiq[1], Yoga Pamungkas Susani[1], Pujiarohman[1]

Institution: 1. Mataram University, Faculty of Medicine
Corresponding Author: Mr. Azhar Rafiq ([email protected])
Categories: Educational Strategies, Learning Outcomes/Competency, Teaching and Learning, Simulation and Virtual Reality
Published Date: 30/06/2021


Introduction: Recently, learning methods have advanced into the development of serious games. There are many potential benefits to the implementation of serious games in medical schools. Med Stud Games (MSG), a serious game for medical students in Indonesia, requires validation on its pedagogic quality according to the 4 pillars of learning, which are attention, active learning, feedback, and consolidation.


Methods: To determine the validity of MSG as a serious game, we described the purpose and potential of MSG as a learning method using the Graafland framework, followed by a focus group discussion (FGD). We used a purposive sampling approach to obtain participants consisted of content experts, medical education experts, and medical students. The participants were asked to experience and evaluate MSG for three days, and the results were discussed in an FGD. Data were analyzed using in vivo coding, which was categorized into four themes, including attention, active learning, feedback, and consolidation.


Results: The transcript resulted in 491 lines, which consisted of 100 quotes and 21 codes. The attention theme was categorized into bite-sized content delivery, animation, audio, and visual. The active learning theme was categorized into learning-game interaction, gamification, and content management. The feedback theme was categorized into answers, scores, points, and punishment, which helped users to identify their learning needs. Finally, the consolidation theme involved the identification of learning needs that can help users to perform learning repetitions.


Conclusion: MSG satisfied the four pillars of learning. This study assessed face validity. Further studies are needed to assess content, concurrent, and predictive validities. 


Keywords: Medical Education; Serious Games; Med Stud Games; eLearning; Pedagogic Quality


Learning methods in medical education continues to develop, from conventional lectures, problem-based learning, to the use of technologies such as serious games (Wood, 2003; Swanwick, 2010). A serious game is a digital game aiming to increase the knowledge, skill, or behavior of its users in real-life (Graafland et al., 2014). It has two main objectives of motivating and education, packed into a game (Drummond, Hadchouel and Tesnière, 2017). This new learning medium may become the next level of education that may enhance learning in medical school.


For it to be used in medical school, a serious game needs to be assessed to prove its learning quality. Currently, there is no study on serious games for medical education in Indonesia. One assessment that can be used to measure pedagogy quality of serious games for medical education is the four pillars of learning by Drummond et al., which consists of attention, active learning, feedback, and consolidation (Drummond, Hadchouel and Tesnière, 2017). A serious game can achieve a high quality of learning effectiveness if it can satisfy the four pillars of learning. The Graafland framework was made to assess the quality of serious games in health care education (Graafland et al., 2014). It contains five categories of items to assess, including game description, rationale, functionality, validity, and data protection. The results of the Graafland framework and the four pillars of learning were used to create an FGD guideline.


Med Stud Games (MSG) is the only serious game available for medical students in Indonesia. Several medical students used it to study and exercise before exams. This game can be used by the students to learn and exercise immunology. However, the game requires an assessment to confirm whether it has the capabilities to be considered as a serious game or not.


This research aims to assess Med Stud Games from the experience of using Med Stud Games from the medical lecturer and medical students’ perspectives. We use FGD to explore what the respondent sees and feel when using MSG for three days. We use purposeful sampling approach, a total of 11 respondents consisting of medical school’s lecturer in immunology and medical education, also medical students that haven’t taken immunology module and the randomly choose them according to their last module score divided into group high, middle, and low. Only 9 respondents participate until the FGD. FGD facilitated by researcher. The result of an experience in learning and evaluating MSG for three days discussed in FGD. Data collected from FGD recordings.


Data analyzed by the researcher using in vivo coding and then divide categories according to four pillars of learning. We use debriefing methods to increase validity and data interpretation. While analyzing and interpreting the data, the researcher asks another professional researcher in the field of medical education to analyze and give advice from the researcher analysis.


The research passed the ethical clearance from the Faculty of Medicine, Universitas Mataram.


Group of medical students consist of 6 people and only 3 that join the research until FGD, consist of each person from the group with a high, middle, and low score. Group of medical lecturers consist of 5 people and only 3 that join the research until FGD, consist of 1 immunology lecturer/medical doctor and 2 medical education/medical doctor.


Total of transcript data is 491 lines. FGD result analyzed using Atlast.ti trial version 8.4.2. The analyze result in 100 quotes and 21 codes. The codes grouped again into 4 big themes according to 4 pillars of learning (Tables 1-4).


Table 1: Comparison between attention pillar of learning as pedagogic quality and analysis results of MSG.

Attention (Baldi and Bucherelli, 2005; Ivory and Kalyanaraman, 2007; Sitzmann, 2011; Drummond, Hadchouel and Tesnière, 2017)



a. Alerting

Alertness needs to be set to a medium level. If the users understand the content, their alertness will increase.

Lay language

Easier for the users to understand the meaning of the content

The first component of alertness is graphic. Visual inconsistency confuses the player, hence lowering the attention level.

Visual Consistency

Participants suggested the need for visual consistency

The second component of alertness is the sound. The sound in this game was still in the lower or basic level.


Sound or voice

The user interface graphic was still in a low or basic level of alertness.


User interface and its correlation with attention and learning

The animation increases physiological arousal, therefore causes higher alertness.


Animated objects make it easier for the user to understand the learning contents

The easier the user uses the app, the more alertness they will have.

Light application

The application size was small, so it was easy to download and install

b. Orienting

A learner needs to be able to filter relevant information. There is no content stressing.

Content Stressing

Participants stated the need for content stressing to differentiate between the “must-know” and the “nice to know” contents

c. Executive

It allows the player to concentrate on the task without being distracted. Bite-sized content allows players to consume the content bit by bit that increases motivation and higher immersion, which are in line with higher focus.

Bite-sized content

Small-sized content enabled the user to learn step by step



Table 2: Comparison between active learning pillar of learning as pedagogic quality and analysis results of MSG.

Active Learning (Baldi and Bucherelli, 2005; Ivory and Kalyanaraman, 2007; Sitzmann, 2011; Drummond, Hadchouel and Tesnière, 2017)



a. Learning-Game Interaction

The integration of serious games into campus activity can help to achieve maximum learning effectiveness.


Participants suggested the use of MSG as a part of a lecture session on campus

Active learning is more effective compared to passive lectures.


Interaction actively involves users in a learning session. There is a need for various methods to improve interaction.

b. Gamification

The timed challenge increases the user’s alertness to select available actions before the time runs out.


There is a limited time to do the quiz. If the time runs out, the quiz will end even though the user has not finished yet.

Reward as a part of gamification’s feedback increases motivation for players to do more interaction and activities.

Score & point system

Participants suggested improving the gamification system to increase user motivation.

Players will get bored easily with low-level difficulty. In a learning environment, players should be given a system that allows them to find out their current level of understanding/skills and try to advance to harder topics in line with increasing levels.

Difficulty level

participants suggested a difficulty level system according to the advancement of learning content.

c. Content Management

User-generated contents are also one of the ways to increase active learning. However, it needs to be moderated to prevent non-pedagogic or low-level learning contents.

Content from users

Respondents suggested a feature that allows the user to contribute by creating their learning content to MSG.



Table 3: Comparison between feedback pillar of learning as pedagogic quality and analysis results of MSG.

Feedback (Baldi and Bucherelli, 2005; Ivory and Kalyanaraman, 2007; Sitzmann, 2011; Drummond, Hadchouel and Tesnière, 2017)



The score allows user to reflects on their activities and find out their progress. This will help to learn according to temporal difference learning theory.


The score is given to the user after quiz interaction.

Comparison of the player's scores with the best one gives user motivation therefore higher alertness.

High Score Board

A set of highest score from all.

The point gives motivation and currency that they can use to advance the topics. However, the player still does not know how to use it.


The point was given to the user after completing learning or quiz content.

Punishment has a role in increasing alertness and feedback for the user to reflect their activities. Further development should be able to tell the user the scale of the relevance of information. So higher punishment should be given to the more important topics/concepts.


When completing the quiz user only have 4 wrong chances, after that the quiz will be finished even though the user hasn’t finished yet.

An explanation is a basic way to give the correct information about the problems they meet in the serous games. So, they can reflect their action and act accordingly.

Answer & Explanation

Notification is given to user about right or wrong of their answer including the explanation why.



Table 4: Comparison between consolidation pillar of learning as pedagogic quality and analysis results of MSG.

Consolidation (Baldi and Bucherelli, 2005; Ivory and Kalyanaraman, 2007; Sitzmann, 2011; Drummond, Hadchouel and Tesnière, 2017)



Consolidation is trying to make players repeat the activities to allow them to shift from conscious to unconscious brain networks. This will help them to master subjects easily. The research found out that this happens mainly because of motivation the players get from all other factors in the serious games.


Encourage user to repeat the learning content.

Learning needs will be identified after players receive feedback and reflect on their actions in serious games.

Identification of learning needs

Help user to identify their weakness in learning subjects and trigger the user to relearn the learning content.


Attention theme is finding the ability of MSG in making user keep focus inside the game. In this theme, there are two groups of codes: content delivery and focus. Content delivery is the ability of MSG to show or deliver the content. Focus is about things that make user focus using the app and not thinking or doing anything outside of the application. 6/8 of attention codes already meet the desirable pedagogic quality.

Active learning theme is finding the ability of MSG in involving users actively in learning activity or games in the app. There are codes: learning-game relation with the user, gamification, and content management. Learning-game relation with the user is a relation shape that found in suggested for MSG and can affect the learning activity. Gamification is a design technique and a game mechanism to influence the user. Content management is how MSG in managing user-based content. 4/6 of active learning codes already meet the desirable pedagogic quality.


Feedback theme is finding the ability of MSG in giving things to users. 4/5 of feedback codes already meet the desirable pedagogic quality. Consolidation theme is finding the ability of MSG in allowing users to repeat the activity or mastering the important concept of learning content. 2/2 of consolidation codes already meet the desirable pedagogic quality. Total 16/21 (76%) of the codes already meet the desirable pedagogic quality.


Reading content from MSG is different from reading an e-book. By using bite-sized content, MSG is easier to be consumed. Current medical student prefers the use of technology and bite-sized teaching while studying medicine (Eckleberry-Hunt and Tucciarone, 2011; Schwartz et al., 2019). Better content delivery, by reducing the size of learning content will improve memory retention and focus of the students. This shows that MSG suit for the learning for current medical students.


Neurology scientist said that there is three attention networks, awareness, orientation, and executive (Petersen and Posner, 2012). Learners should stay alert during the learning process. The goal is to reach a medium level of arousal associated with the highest gains in knowledge. MSG support awareness network by using animation, audio, feedback, and motivation. Animation has a benefit in a medical study because it can deliver complex concept like immunology into a simple representation that easier to understand and have good long memory retention (Stith, 2004; Klymkowsky and Reiness, 2007). Even though MSG animation is helping the user to understand the concept, MSG still does not have a visual consistency that confuses the user when reading from one part of the content to another content. A confused player may be lowering the alertness needed to learn effectively. Another attention aspect of simplification in MSG is the use of lay language. In line with current research that said simplification of medical learning content liked by medical students (Kazzazi and Bartlett, 2017).


Audio as feedback for learning activity can improve awareness and immersion of the player. Respondent specifically said that the sound in MSG become an element of surprise. This excitement is happened because of the attractiveness of the media content that gives a good emotional response (Ivory and Kalyanaraman, 2007). Another example of the use of excitement in MSG is paper sound that played when user navigating throughout the content.


The second part of the attention network is an orientation to filter important information. MSG is not yet helping relevant information. There is not enough content stressing to help the student to differ from must to know and nice to know the content. In the Indonesian standard of medical doctor competency, a medical student has a list of diseases and practical clinical skill ranging from level 1 to 4 (Konsil Kedokteran Indonesia, 2012). Stressing content in serous games will be necessary to help a student while studying to become a medical doctor. This way student may choose which content they need to master therefore may need to repeat a lot.


The third attention network is an executive that allows the user to concentrate on the task given without thinking about any distraction from their environment. In serious games, this concentration can be called as immersion. Immersion is a sensation of taking all the attention of player in serious games (Jennett et al., 2008). The immersion level Is equal to the learning effectiveness in serious games (Drummond, Hadchouel and Tesnière, 2017). MSG immersion comes from motivation. Motivation in MSG is created when the user knows the answer to the problem and see the high scoreboard. This way user has emotional involvement that higher the immersion therefore better learning effectiveness (Jennett et al., 2008). This could happen because the connection in amygdala proved to do interaction between emotion and cognition, which means the emotion involvements can trigger the creation and recovery of episodic memory (Murray, Holland and Kensinger, 2013).


Active learning as the second pillars of learning in MSG is quite good. Respondent praise the MSG is more interactive compared to online courses. As part of active learning, gamification like score and point system play a big role in improving interactivity. This system can improve motivation to do more interaction. However, there is a lot of room to be improved (Yunyongying, 2014; Drummond, Hadchouel and Tesnière, 2017). Respondent suggests new features like difficulty level, the ability for the user to create their own content, and the implementation of MSG in lectures or classroom activity. This integration of serious games in learning activity can enhance and achieve the maximum potential of serious games (Sitzmann, 2011). There is also the need for the clarity of what and how to use the benefit of having points in MSG. One important aspect of gamification is the difficulty level that allows the user to choose to play from the start or the advance topic according to their needs. The challenges in level will help them to achieve better understanding and skills when solving various problems in the future (Cheng, She and Annetta, 2015). Lastly, the content stressing can also be applied in a gamification system. For example, when user solving important clinical cases, which is the highest needed competency, they will get a higher point and score compared to lower-level competency cases.


Serious games can have various feedbacks like process bar, scoring, achievements, experience point, and virtual currency (Drummond, Hadchouel and Tesnière, 2017). MSG has answer and explanation, score, high score board, punishment, and experience point for doing activities. MSG also has a process bar that related to time when do quiz. Answer and explanation of the problems can improve students confidence from the knowledge that they received (Yunyongying, 2014). Scoring in MSG is given at the end of the games. This is the better option compared to the score that given while playing that may distract the learner from their main job to study (Katz et al., 2014). Lastly, the respondent still does not know how to use the point feedbacks.


Consolidation is the enhancement of using aware memories at the first time learning to become using unaware memories. If the repetition of this switch is continuously trained, it can be a long term memory that is faster when doing switches (Wouters et al., 2013). Repetition will be more effective if using various learning methods. Research shows that serious games are better to be used as a supplement for a lecture, discussion, tutorial, and any other method that need an instructor to maximize its learning potential (Sitzmann, 2011). MSG does not have any features that remind the user to do repetition explicitly. However, MSG does give users motivation to reread the learning content and repeat interactions. In theory, repetition that influenced by motivation has the same effect to create long term memories (Wouters et al., 2013). Therefore, it has a good learning quality. From all of the four themes, we create connections between them that results in a diagram of codes relationship (Figure 1). The line shows the code belongs to. The arrow shows its effect.


Figure 1. Diagram of codes relationship from FGD.



In summary, the attention, active learning, and feedbacks in MSG linked together to create motivation that will improve consolidation. User learning in MSG from the attention aspect and then do active learning. The MSG then give feedback and allow users to identify learning needs that create motivation. At the end of the process, the motivation gives the user the urge to repeat the learning contents and activities.


MSG satisfied the four pillars of learning. It delivered bite-sized content, animation, audio, and visuals, which increased users’ attention. Various interaction methods and gamification systems improved active learning. Feedback using correct answers, scores, points, and punishment helped users to identify their learning needs, which encouraged users to repeat the learning contents. MSG still needs improvements to achieve maximum learning effectiveness.

Take Home Messages

  • Medical schools recommended to implement serious games as an alternative learning method.
  • Serious games developer that focuses on medical education needs to implement and emphasize the features from the four pillars of learning.
  • Further studies are needed to validate more aspects of the four pillars of learning and to evaluate the effectiveness of implementing serious games in medical schools.

Notes On Contributors

Azhar Rafiq, Bachelor of Medicine. Currently, a medical student in clinical clerkship phase at Hospital of Province Nusa Tenggara Barat. Interested in developing the best medical education experiences using technology. ORCiD:

Yoga Pamungkas Susani, M.D., also has Master and Doctor degree in medical education. A medical lecturer and medical education researcher at Mataram University, Faculty of Medicine, Indonesia.

Pujiarohman, Master degree in psychology. A psychology lecturer at Mataram University, Faculty of Medicine, Indonesia.


Figure 1. Source: the authors.

We thank all the medical education lecturers in the Faculty of Medicine, University of Mataram, for all the support, suggestion, and critics of this project.


Baldi, E. and Bucherelli, C. (2005) ‘The inverted “u-shaped” dose-effect relationships in learning and memory: modulation of arousal and consolidation’, Nonlinearity in Biology, Toxicology, Medicine. SAGE Publications, 3(1), pp. 9–21.


Cheng, M. T., She, H. C. and Annetta, L. A. (2015) ‘Game immersion experience: Its hierarchical structure and impact on game-based science learning’, Journal of Computer Assisted Learning, 31(3), pp. 232–253.


Drummond, D., Hadchouel, A. and Tesnière, A. (2017) ‘Serious games for health: three steps forwards’, Advances in Simulation, 2(1), p. 3.


Eckleberry-Hunt, J. and Tucciarone, J. (2011) ‘The Challenges and Opportunities of Teaching “Generation Y”’, Journal of Graduate Medical Education, 3 (4): 458–461.


Graafland, M., Dankbaar, M., Mert, A., Lagro, J., et al. (2014) ‘How to systematically assess serious games applied to health care’, JMIR Serious Games. JMIR Publications Inc., 2(2), p. e11.


Ivory, J. D. and Kalyanaraman, S. (2007) ‘The effects of technological advancement and violent content in video games on players’ feelings of presence, involvement, physiological arousal, and aggression’, Journal of Communication, 57(3), pp. 532–555.


Jennett, C., Cox, A., Cairns, P., Dhoparee, S., et al. (2008) ‘Measuring and Defining the Experience of Immersion in Games’, International Journal of Human-Computer Studies. 66(9), pp 641-661.


Katz, B., Jaeggi, S., Buschkuehl, M., Stegman, A., et al. (2014) ‘Differential effect of motivational features on training improvements in school-based cognitive training’, Frontiers in Human Neuroscience. Frontiers Media SA, 8, p. 242.


Kazzazi, F. and Bartlett, J. (2017) ‘Condensing embryology teaching for medical students: can it be taught in 2 hours?’, Advances in Medical Education and Practice, (8), pp. 797–806.


Klymkowsky, M. W. and Reiness, G. (2007) ‘Point of View: Textbooks - Essential or Superfluous? Teaching without a Textbook: Strategies to Focus Learning on Fundamental Concepts and Scientific Process’, CBE--Life Sciences Education, 6, pp. 190–193.


Konsil Kedokteran Indonesia (2012) Standar Kompetensi Dokter Indonesia. Jakarta Pusat: Konsil Kedokteran Indonesia.


Murray, B. D., Holland, A. C. and Kensinger, E. A. (2013) ‘Episodic Memory and Emotion’, in Handbook of Cognition and Emotion. London: Guilford Press, pp. 156–175.


Petersen, S. E. and Posner, M. I. (2012) ‘The attention system of the human brain: 20 years after’, Annual Review of Neuroscience. NIH Public Access, 35, pp. 73–89.


Schwartz, A. C. et al. (2019) ‘Bite-Sized Teaching: Engaging the Modern Learner in Psychiatry’, Academic Psychiatry, 43(3), pp. 315–318.


Sitzmann, T. (2011) ‘A Meta-Analytic Examination Of The Instructional Effectiveness Of Computer-Based Simulation Games’, Personnel Psychology. John Wiley & Sons, Ltd (10.1111), 64(2), pp. 489–528.


Stith, B. J. (2004) ‘Use of animation in teaching cell biology’, Cell Biology Education, 3(3), pp. 181–188.


Swanwick, T. (2010) Understanding Medical Education. 1st edn. Wiley-Blackwell.


Wood, D. F. (2003) ‘ABC of Learning and Teaching in Medicine’, BMJ, 326(1), pp. 328–330. Available at: (Accessed: 14 October 2019).


Wouters, P., van Nimwegen, C., van Oostendorp, H., van Der Spek, E., et al. (2013) ‘A meta-analysis of the cognitive and motivational effects of serious games’, Journal of Educational Psychology, 105(2), pp. 249–265.


Yunyongying, P. (2014) ‘Gamification: Implications for Curricular Design’, Journal of Graduate Medical Education, 6(3), pp. 410–412.




There are no conflicts of interest.
This has been published under Creative Commons "CC BY-SA 4.0" (

Ethics Statement

The research passed the ethical clearance from the Faculty of Medicine, Universitas Mataram, Ethics Commission of Health Research: No. 35/UN18.F7/ETIK/2019.

External Funding

This article has not had any External Funding


Please Login or Register an Account before submitting a Review

Hui Meng Er - (31/07/2021) Panel Member Icon
This is an important area of study in view of the rise of online learning as well as the need to diversify technology enhanced learning resources to meet the interest of generation z learners. Gamification learning has evolved at a fast pace in the recent years, and it is crucial to ensure that the tools are designed according to sound education principles. The assessment instrument for MSG developed by the authors from the qualitative study is very useful, and the fact that the aspects being evaluated are aligned with the four pillars of learning support its validity and credibility. It would be useful for the authors to explain the basis for selecting MSG to support teaching and learning of immunology, and give a description of the learning outcomes and features of MSG. In thematic analysis, quotes from the FGD should be included as evidence for the identified themes. There are several flaws in the sentence structures and grammar throughout the manuscript that affect its quality. Nevertheless, the article is an important contribution to the field of study. I look forward to the revised version of the manuscript.
Mary Higgins - (19/07/2021) Panel Member Icon
Thank you for this interesting paper. I enjoyed reading most of it and any comments here are intended to help improve your paper.
-Firstly, in the title, is "Med Stud Games" the title of a specific set of games? If so suggest having it in inverted commas and giving more context in the methods. If it refers to a general group of Medical Student games then suggest full words for the title of a paper.
-In your introduction it would be useful to have more of a background (one paragraph) on the evidence to date and what this study adds
-Suggest qualifying what FGD means before using it in the text (needs to be qualified once in both the abstract and the paper before using the acronym).
-This is essentially a qualitative study. Having read the results a couple of times, and the accompanying tables, I still do not see any evidence for the focus group discussions - there are no obvious quotes. I also do not completely understand what your results are!
-Suggest if this is indeed a qualitative study that you review some of the reporting guidelines which may help with a revision.
Your passion and interest in this subject shines through. That is so commendable. I look forward to a revision where you make your results more explicit.