Submitted:
22 May 2024
Posted:
23 May 2024
You are already at the latest version
Abstract
Keywords:
1. Introduction
2. Materials and Methods
2.1. Conceptual Approach to Construct Validity Evidence
2.2. Study Design and Sample
2.3. Procedures
2.4. Scale Adaptation
2.5. Construct Validity – Content Analysis
2.6. Construct Validity – Response Process Analysis
2.7. Construct Validity – Internal Structure Analysis
2.8. Construct Validity – Relationship to Other Variables Analysis
2.9. Construct Validity – Relationship to Other Variables Analysis
2.10. Ethical Approval
3. Results
3.1. Participants
3.2. Construct Validity – Content Evidence
3.3. Construct Validity – Response Process Evidence
3.4. Construct Validity – Internal Structure Evidence
3.5. Construct Validity – Relationship to Other Variables Evidence
3.6. Construct Validity – Consequences Evidence
4. Discussion
4.1. Limitations and Threats to Validity
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
Appendix A
Appendix B
Appendix C
Appendix D
References
- Oxford English and Spanish Dictionary. “motivation, n.” (2021). Retrieved November 1, 2023, from https://www.lexico.com/en/definition/motivation.
- Deci, E.; Ryan, R.M. Intrinsic Motivation and Self-Determination in Human Behavior; Springer Science & Business Media: Berlin/Heidelberg, Germany, 1985; ISBN 0306420228. [Google Scholar]
- Ryan, R.M.; Deci, E.L. Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions. Contemp. Educ. Psychol. 2000, 25, 54–67. [Google Scholar] [CrossRef] [PubMed]
- Deci, E.L.; Ryan, R.M. The “What” and “Why” of Goal Pursuits: Human Needs and the Self-Determination of Behavior. Psychol. Inq. 2000, 11, 227–268. [Google Scholar] [CrossRef]
- Mayer, R. E. (2008). Learning and instruction. Prentice Hall.
- Dyrbye, L.N.; Thomas, M.R.; Shanafelt, T.D. Systematic Review of Depression, Anxiety, and Other Indicators of Psychological Distress Among U.S. and Canadian Medical Students. Acad. Med. 2006, 81, 354–373. [Google Scholar] [CrossRef] [PubMed]
- Brazeau, C.M.; Shanafelt, T.; Durning, S.J.; Massie, F.S.; Eacker, A.; Moutier, C.; Satele, D.V.; Sloan, J.A.; Dyrbye, L.N. Distress Among Matriculating Medical Students Relative to the General Population. Acad. Med. 2014, 89, 1520–1525. [Google Scholar] [CrossRef] [PubMed]
- Mata, D.A.; Ramos, M.A.; Bansal, N.; Khan, R.; Guille, C.; Di Angelantonio, E.; Sen, S. Prevalence of Depression and Depressive Symptoms Among Resident Physicians. JAMA 2015, 314, 2373–2383. [Google Scholar] [CrossRef] [PubMed]
- Cooke, M.; Irby, D.M.; Sullivan, W.; Ludmerer, K.M. American Medical Education 100 Years after the Flexner Report. New Engl. J. Med. 2006, 355, 1339–1344. [Google Scholar] [CrossRef] [PubMed]
- Drolet, B.C.; Rodgers, S. A Comprehensive Medical Student Wellness Program—Design and Implementation at Vanderbilt School of Medicine. Acad. Med. 2010, 85, 103–110. [Google Scholar] [CrossRef]
- Slavin, S.J.; Schindler, D.L.; Chibnall, J.T. Medical Student Mental Health 3.0. Acad. Med. 2014, 89, 573–577. [Google Scholar] [CrossRef] [PubMed]
- Dyrbye, L.N.M.; West, C.P.; Satele, D.; Boone, S.; Tan, L.; Sloan, J.; Shanafelt, T.D. Burnout Among U.S. Medical Students, Residents, and Early Career Physicians Relative to the General U.S. Population. Acad. Med. 2014, 89, 443–451. [Google Scholar] [CrossRef]
- Binning, J. (2016). “Construct”. Encyclopedia Britannica. Retrieved November 1, 2023, from https://www.britannica.com/science/construct.
- DeVellis, R.F. Scale Development: Theory and Applications, 3rd ed.; Sage Publications: Saunders Oaks, CA, USA, 2016; pp. 50–100. [Google Scholar]
- Guay, F.; Vallerand, R.J.; Blanchard, C. On the Assessment of Situational Intrinsic and Extrinsic Motivation: The Situational Motivation Scale (SIMS). Motiv. Emot. 2000, 24, 175–213. [Google Scholar] [CrossRef]
- Jeno, L.M.; Raaheim, A.; Kristensen, S.M.; Kristensen, K.D.; Hole, T.N.; Haugland, M.J.; Mæland, S. The Relative Effect of Team-Based Learning on Motivation and Learning: A Self-Determination Theory Perspective. CBE—Life Sci. Educ. 2017, 16, ar59. [Google Scholar] [CrossRef] [PubMed]
- Zimmerman, B.J. Investigating Self-Regulation and Motivation: Historical Background, Methodological Developments, and Future Prospects. Am. Educ. Res. J. 2008, 45, 166–183. [Google Scholar] [CrossRef]
- Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). New York: American Council on Education.
- Downing, S.M. Validity: on the meaningful interpretation of assessment data. Med Educ. 2003, 37, 830–837. [Google Scholar] [CrossRef] [PubMed]
- Cook, D.A.; Beckman, T.J. Current Concepts in Validity and Reliability for Psychometric Instruments: Theory and Application. Am. J. Med. 2006, 119, 166–e7. [Google Scholar] [CrossRef] [PubMed]
- Goodwin, L.D.; Leech, N.L. The Meaning of Validity in the New Standards for Educational and Psychological Testing:. Meas. Evaluation Couns. Dev. 2003, 36, 181–191. [Google Scholar] [CrossRef]
- Vallerand, R. J. , & Fortier, M. N. (1998). Measures of Intrinsic and Extrinsic Motivation in Sport and Physical Activity: A Review and Critique. In J. L. Duda (Ed.), Advances in Sport and Exercise Psychology Measurement (pp. 81-101). Morgantown, WV: Fitness Information Technology.
- Schaufeli, W.B.; Martínez, I.M.; Pinto, A.M.; Salanova, M.; Bakker, A.B. Burnout and Engagement in University Students. J. Cross-Cultural Psychol. 2002, 33, 464–481. [Google Scholar] [CrossRef]
- Posch, L. , Bleier, A., Lechner, C. M., Danner, D., Flöck, F., & Strohmaier, M. (2017). Measuring Motivations of Crowdworkers: The Multidimensional Crowdworker Motivation Scale. arXiv (Cornell University). http://export.arxiv.org/pdf/1702. 0166. [Google Scholar]
- Muthen, L. K. , & Muthen, B. O. (2017). MPlus Version 8 User’s Guide.
- Nunnally, J. , & Bernstein, I. H. (1994). Psychometric theory. McGraw-Hill Humanities/Social Sciences/Languages.
- Bland, J.M.; Altman, D.G. Statistics notes: Cronbach’s alpha. BMJ 1997, 314, 572. [Google Scholar] [CrossRef]
- DeVellis, R.F. Scale Development: Theory and Applications, 3rd ed.; Sage Publications: Saunders Oaks, CA, USA, 2016; pp. 50–100. [Google Scholar]
- Using and interpreting Cronbach’s Alpha | UVA Library. (n.d.). Retrieved , 2023 from https://data.library.virginia. 1 November.
- Browne, M.W. An Overview of Analytic Rotation in Exploratory Factor Analysis. Multivar. Behav. Res. 2001, 36, 111–150. [Google Scholar] [CrossRef]
- Hooper, D.; Coughlan, J.; Mullen, M.R. Equation modelling: Guidelines for determining model fit. Electron. J. Bus. Res. Methods 2008, 6, 53–60. [Google Scholar] [CrossRef]
- Kline, R.B. Principles and Practice of Structural Equation Modeling, 2nd ed. The Guilford Press: New York, NY, USA, 2005.
- Hu, L.T.; Bentler, P.M. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Struct. Equ. Model. Multidiscip. J. 1999, 6, 1–55. [Google Scholar] [CrossRef]
- Modification indices | Just Enough R. (n.d.-b). Retrieved November 1, 2023 from https://benwhalley.github.io/just-enough-r/modification-indices.html#:~:text=T%20he%20modification%20index%20is,was%20added%20or%20constraint%20freed.
- Deci, E.L.; Vallerand, R.J.; Pelletier, L.G.; Ryan, R.M. Motivation and Education: The Self-Determination Perspective. Educ. Psychol. 1991, 26, 325–346. [Google Scholar] [CrossRef]
- Ten Cate, O. , Snell, L., & Carraccio, C. (2010). Medical education: Past, present and future. Perspectives on Medical Education, 29(6), 320-327. [CrossRef]
- Kusurkar, R.A.; Croiset, G.; Ten Cate, O.T.J. Twelve tips to stimulate intrinsic motivation in students through autonomy-supportive classroom teaching derived from Self-Determination Theory. Med. Teach. 2011, 33, 978–982. [Google Scholar] [CrossRef] [PubMed]
- Ellaway, R.H.; O’gorman, L.; Strasser, R.; Marsh, D.C.; Graves, L.; Fink, P.; Cervin, C. A critical hybrid realist-outcomes systematic review of relationships between medical education programmes and communities: BEME Guide No. 35. Med Teach. 2015, 38, 229–245. [Google Scholar] [CrossRef] [PubMed]
- Mann, K.V. The role of educational theory in continuing medical education: Has it helped us? J. Contin. Educ. Heal. Prof. 2004, 24, S22–S30. [Google Scholar] [CrossRef] [PubMed]
- Umer Shafique, Aamir Ijaz, Muhammad Salman, Waheed uz Zaman, Nadia Jamil, Rabia Rehman, Amna Javaid; Removal of arsenic from water using pine leaves, Journal of the Taiwan Institute of Chemical Engineers 43 ,256–263(2012). [CrossRef]
- Stipek, D. J. (2002). Motivation to learn: Integrating Theory and Practice. Allyn & Bacon.
- Blumberg, P. (2000). Evaluating the Evidence That Problem-Based Learners Are Self-Directed Learners: A Review of the Literature. In D. H. Evensen & C. E. Hmelo (Eds.), Problem-Based Learning: A Research Perspective on Learning Interactions (pp. 199-226). Lawrence Erlbaum Associates.
- Murad, M.H.; Varkey, P. Self-directed Learning in Health Professions Education. Ann. Acad. Med. Singap. 2008, 37, 580–590. [Google Scholar] [CrossRef] [PubMed]
- Cronbach, L.J.; Meehl, P.E. Construct validity in psychological tests. Psychol. Bull. 1955, 52, 281–302. [Google Scholar] [CrossRef]
- Kane, M.T. Validating the Interpretations and Uses of Test Scores. J. Educ. Meas. 2013, 50, 1–73. [Google Scholar] [CrossRef]
- American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.

| Validity Evidence Sources | Validity Evidence Definitions |
|---|---|
| Content | Content validity focuses on the construct’s dimensions and how well they are represented. |
| Response Process | Response process validity is the extent to which the actions and thought processes of respondents demonstrate that they understand the instrument questions the same way as the researchers do. |
| Internal Structure | Internal structure evidence asks whether a group of questions designed to assess the same dimension exhibit higher correlations with one another compared to groups of questions aimed at measuring different dimensions. |
| Relationship to Other Variables | Relationship to other variables evidence asks if external correlations (i.e., correlations to entirely different instruments) are confirmatory or counter-confirmatory. |
| Consequences | Consequences evidence asks what is the impact on respondents of being appraised by the instrument? |
| Item | Present Tense SIMS Version | Past Tense SIMS Version |
|---|---|---|
| 1 | Because I think that the activity is interesting | Because I thought that the activity was interesting |
| 2 | Because I am doing it for my own good | Because I was doing it for my own good |
| 3 | Because I am supposed to do it | Because I was supposed to do it |
| 4 | There may be good reasons to do this activity, but personally I don’t see any | There may have been good reasons to do this activity, but personally I don’t see any |
| 5 | Because I think that this activity is pleasant | Because I thought that this activity was pleasant |
| 6 | Because I think that this activity is good for me | Because I thought that this activity was good for me |
| 7 | Because it is something that I have to do | Because it was something that I had to do |
| 8 | I do this activity but I am not sure if it is worth it | I did this activity but I am not sure if it was worth it |
| 9 | Because this activity is fun | Because this activity was fun |
| 10 | By personal decision | By personal decision |
| 11 | Because I don’t have any choice | Because I didn’t have any choice |
| 12 | I don’t know; I don’t see what this activity brings me | I don’t know; I don’t see what this activity brought me |
| 13 | Because I feel good when doing this activity | Because I felt good when doing this activity |
| 14 | Because I believe that this activity is important for me | Because I believe that this activity was important for me |
| 15 | Because I feel that I have to do it | Because I feel that I had to do it |
| 16 | I do this activity, but I am not sure it is a good thing to pursue it | I did this activity, but I am not sure it was a good thing to pursue it |
| SIMS Context |
Intrinsic Motivation (IM) Qs 1, 5, 9, 13 |
Identified Regulation (IR) Qs 2, 6, 10, 14 |
External Regulation (ER) Qs 3, 7, 11, 15 |
Amotivation (AM) Qs 4, 8, 12, 16 |
|---|---|---|---|---|
| Post-situational Cronbach’s alpha |
0.98 | 0.98 | 0.95 | 0.93 |
| Short-term Cronbach’s alpha |
0.98 | 0.82 | 0.99 | 0.91 |
| SIMS Post-situational Context | ||||
|---|---|---|---|---|
| Intrinsic Motivation (IM) | Identified Regulation (IR) | External Regulation (ER) | Amotivation (AM) | |
| Intrinsic Motivation (IM) | 1.0 | 0.63 | -0.25 | -0.3 |
| Identified Regulation (IR) | 0.63 | 1.0 | -0.16 | -0.43 |
| External Regulation (ER) | -0.25 | -0.16 | 1.0 | 0.3 |
| Amotivation (AM) | -0.3 | -0.43 | 0.3 | 1.0 |
| SIMS Short-term Context | ||||
|---|---|---|---|---|
| Intrinsic Motivation (IM) | Identified Regulation (IR) | External Regulation (ER) | Amotivation (AM) | |
| Intrinsic Motivation (IM) | 1.0 | 0.27 | -0.19 | -0.23 |
| Identified Regulation (IR) | 0.27 | 1.0 | 0.18 | -0.29 |
| External Regulation (ER) | -0.19 | 0.18 | 1.0 | 0.16 |
| Amotivation (AM) | -0.23 | -0.29 | 0.16 | 1.0 |
| SIMS Post-situational Context | ||||
|---|---|---|---|---|
| Item | Intrinsic Motivation (IM) | Identified Regulation (IR) | External Regulation (ER) | Amotivation (AM) |
| 1 | 0.76* | |||
| 5 | 0.87* | |||
| 9 | 0.85* | |||
| 13 | 0.77* | |||
| 2 | 0.83* | |||
| 6 | 0.85* | |||
| 10 | 0.35* | |||
| 14 | 0.77* | |||
| 3 | 0.76* | |||
| 7 | 0.84* | |||
| 11 | 0.68* | |||
| 15 | 0.85* | |||
| 4 | 0.83* | |||
| 8 | 0.78* | |||
| 12 | 0.86* | |||
| 16 | 0.66* | |||
|
Factor Correlation |
IM | IR | ER | |
| IR | 0.69* | |||
| ER | -0.25* | -0.07 | ||
| AM | -0.30* | -0.48* | 0.28* | |
| SIMS Short-term Context | ||||
|---|---|---|---|---|
| Item | Intrinsic Motivation (IM) | Identified Regulation (IR) | External Regulation (ER) | Amotivation (AM) |
| 1 | 0.56* | |||
| 5 | 0.83* | |||
| 9 | 0.98* | |||
| 13 | 0.68* | |||
| 2 | 0.60* | |||
| 6 | 0.79* | |||
| 10 | 0.47* | |||
| 14 | 0.79* | |||
| 3 | 0.71* | |||
| 7 | 0.90* | |||
| 11 | 0.50* | |||
| 15 | 0.77* | |||
| 4 | 0.77* | |||
| 8 | 0.75* | |||
| 12 | 0.75* | |||
| 16 | 0.81* | |||
| FactorCorrelation | IM | IR | ER | |
| IR | 0.25* | |||
| ER | -0.18† | 0.44* | ||
| AM | -0.12 | -0.36* | 0.12 | |
| SIMS Post-situational Context | ||
|---|---|---|
| Model Fit Indices | BIC | 8235.9 |
| χ²/df | 247.2/98 = 2.52 | |
| CFI | 0.894 | |
| RMSEA (95% CI) | 0.099 (0.084-0.114) | |
| SRMR | 0.079 | |
| SIMS Short-term Context | ||
|---|---|---|
| Model Fit Indices | BIC | 8025.6 |
| χ²/df | 146.2/98 = 1.49 | |
| CFI | 0.954 | |
| RMSEA (95% CI) | 0.056 (0.036-0.074) | |
| SRMR | 0.082 | |
| Post-situational | SIMS | IM | IR | ER | AM | Cronbach’s alpha |
|---|---|---|---|---|---|---|
| MCMS | IM | 0.85 | 0.65 | -0.21 | -0.29 | 0.93 |
| IR | 0.17 | 0.23 | 0.06 | -0.08 | 0.98 | |
| ER | -0.09 | -0.01 | 0.30 | 0.23 | 0.89 | |
| AM | -0.40 | -0.55 | 0.32 | 0.76 | 0.98 | |
| AMS | IM | 0.64 | 0.58 | -0.21 | -0.16 | 0.96 |
| IR | 0.44 | 0.51 | -0.11 | -0.20 | 0.96 | |
| ER | 0.14 | 0.08 | 0.10 | 0.21 | 0.95 | |
| AM | -0.34 | -0.49 | 0.29 | 0.81 | 0.31 | |
| MBI-SSE | -0.40 | -0.33 | 0.23 | 0.37 | 0.998 | |
| Short-term | SIMS | IM | IR | ER | AM | Cronbach’s alpha |
|---|---|---|---|---|---|---|
| MCMS | IM | 0.86 | 0.30 | -0.22 | -0.18 | 0.98 |
| IR | -0.09 | 0.30 | 0.24 | -0.15 | 0.96 | |
| ER | -0.07 | 0.01 | 0.32 | 0.05 | 0.88 | |
| AM | -0.32 | -0.29 | 0.21 | 0.59 | 0.99 | |
| AMS | IM | 0.64 | 0.31 | -0.12 | 0.02 | 0.91 |
| IR | 0.18 | 0.39 | 0.25 | -0.18 | 0.97 | |
| ER | -0.14 | 0.11 | 0.35 | 0.12 | 0.96 | |
| AM | -0.1 | -0.31 | 0.13 | 0.65 | 0.84 | |
| MBI-SSEE | -0.45 | -0.10 | 0.23 | 0.31 | 0.93 | |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).