MISMATCH BETWEEN STUDENT AND TUTOR EVALUATION OF TRAINING NEEDS : AN EXPLORATORY STUDY OF TRAUMATOLOGY ROTATIONS

Clinical training in medical schools in Spain is performed by rotations in university hospitals. During these internships, students are expected to acquire and master basic procedural skills. However, the assessment tools available rarely check whether these skills are completely acquired by the students. We have used an e-portfolio to determine the optimal number of times the students need to repeat a procedure to be able to perform it independently. The results were compared with the actual performance during the internships. An e-portfolio collected qualitative information about the internships. Quantitative information was also requested about the number of times each clinical skill was performed. Later, a survey asked these students and their teachers the optimal number of times each skill should be repeated before it could be considered fully acquired. The questionnaire was answered by 98.6% of the students and 70.3% of their teachers. Out of the 21 clinical skills and procedures selected, both students and their tutors agreed in a similar optimal value in 16 of them; only in five of them, teachers thought that students needed a greater number of times than that selected by the students. When these optimal values were compared with the actual values recorded in the portfolio during the internships, it was found that about half of the clinical skills were carried out less frequently than expected, thus providing an important feedback about the internships. Quantitative information collected in portfolios reveals a moderate mismatch between students and tutors perceptions of their training needs.


Introduction
The idea that doctors are competent to work with patients immediately after graduation has been luckily abandoned during the last decades (1).In fact, a very low positive correlation between outcomes in written tests and examinations and the actual clinical practice has been demonstrated (2)(3)(4)(5).In Spain, the entrance to the postgraduate training (specialization) is granted based on the scores obtained in a multiple choice test (MIR examination) that mainly assess the theoretical knowledge level.Some clinical skills like reading X-ray images or EKGs are also included, as well as clinical cases, but a comprehensive OSCE-type assessment is not among the priorities of the organizers, the Health Ministry, the authority that controls the MIR examination (6)(7).Thus, it is not without surprise that the training of medical students in the medical schools has traditionally lacked a serious clinical training, in which even a list of minimum skills and clinical maneuvers is missing and the student is treated frequently as a hospital tourist (8).
Things are slowly changing since the adoption of a system based on the training in clinical competences (9) and the adoption of assessment tools such as OSCEs in medical schools (10)(11).While some of these competences are related to medical knowledge, that is the theoretical background, many of them, specially those clinical competences relative to clinical skills and procedures (12) require a different mechanism to teach and learn them and, more importantly, to consider them sufficiently acquired.These clinical competences, therefore, need time to be learned and also need repetition before deciding if a student is able to perform those tasks with autonomy and, therefore, to consider that he/she has acquired the corresponding learning oucome (13)(14).However, determining what is the optimal number of times that the students need to repeat a procedure to be competent enough to perform it independently remains a challenge in undergraduate medical education (15).But, some studies has been carried out to determine these numbers in postgraduate medical education, for internal medicine or surgery residents (16)(17)(18)(19)(20).
In a previous paper, we presented evidence that a reflexive portfolio is an important tool to analyze the acquisition of traumatology competences of medical students (21).In that work, the teachers assessed the level of acquisition of clinical competences by reading critically all the portfolios, but this was done qualitatively, without taking into account the number of times the different skills were carried out.With this in mind, we have modified our portfolio in several aspects.First, we have made an electronic version, which makes easier the process of portfolio assessment.Secondly, we have implemented measures to determine the number of times that the student performs each skill and compared it to the values obtained by surveying these own students and also their teachers.

Material and Methods
We have used an updated version of our portfolio previously described (21), and that it is used to assess the competences and learning outcomes assigned to Traumatology in the curriculum of the Medical Degree at the University of Murcia.Basically, the portfolio is now an electronic portfolio (e-portfolio), designed with two different sections.The first is a qualitative part where the students describe the activity carried out, highlight what they have learned and make reflections about the most remarkable aspects of their internship.The second section is the quantitative part, where they mark in a series of boxes the number of times the different learning tasks are carried out.These learning tasks are 21, related to the five competences that are evaluated with the e-portfolio (5 competencies).Students fill up the eportfolio while they assist to its Traumatology internship, consisting of a mandatory 10-day rotation (four days in outpatient appointments, four days in the operating room and two days in the emergency room).This activity is necessary in order to pass the discipline and it accounts for 10% of the final result.The number of students registered was 215 and all of them had personal and private access to the e-portfolio, located in a server of the Universidad de Murcia (http://practicum-med.inf.um.es/portfolio/).
Close to the end of the academic year, once the clinical skills exam was carried out, we administered a questionaire to all these students and ask them about the number of times they consider each skill should be repeated to consider that they have have acquired the corresponding outcome.The questionnaire was answered by 212 of the 215 students enrolled (98.6%).This questionnaire was also administered to the tutors of the internship, being answered by 52 out of 74 (70.3%).

Statistical analysis.
Data analysis was performed with the Statistical Package for the Social Sciences (SPSS), version 19.Means and standard deviations were obtained to describe the results and the statistical significance between means was performed by means of an ANOVA.A p value lower than 0.05 was considered to be statistically significant.

Results
Table 1 shows the comparison between the estimated number of times that the students and their tutors think it is necessary to repeat a procedure or skill in order to consider effectively acquired each learning task.Most of the values (16 out of 21) considered by students and tutors were very similar in both cases, except for five of them, where tutors statistically consider that students should repeat them many more times.Regarding the performance of our students (table 2) in relation to these optimal numbers, we found that almost half of the skills or procedures (12 from the perception of students and 11 from the perspective of tutors, out of the 21 considered), were performed a lower number of times than the optimal value.The rest of the skills or procedures were performed more often and exceeded the optimal value.Only one procedure from the point of view of students and two from the point of view of tutors were performed the number of times required and no statistical differences were observed.
Of the skills or procedures less performed, 5 of them belong to the competence related to the diagnosis of injuries of the locomotor system.This can be due to the insufficient number of cases that are attended in the hospital.Also during the clerkship period, there may be few cases of low back pain, scoliosis, kyphosis and/or lordosis (competence 2: identify, by physical examination, locomotor injuries).

Discussion
Several authors have shown that qualitative portfolios, regardless of their content validity, have a moderate reliability.One of the measures to improve them is the addition of quantitative aspects that provide information regarding the number of times each of the actions or clinical tasks are performed (21)(22)(23).It is known that practice improves the learning of procedures and clinical skills (13)(14).
The e-portfolio used in this research allows this improved assessment, the classical qualitative area, and the new quantitative area.This new area adds to the qualitative part, where students indicate the activities carried out and his reflections, by marking the number of times that each one of the tasks that correspond with the learning results are performed.This quantitative area allows to easily ascertain if the students acquire the proposed competences or if we have to modify the distribution of the 10 days during which they rotate in consultation, emergency, and operating room or to use other strategies.The main problem with this strategy is to determine the optimal number of times that each one of the proposed skills or procedures should be repeated.We have solved this issue by asking the tutors of the students, that work closely with them during their clerkship.Also, the students have given their opinion.Nobody knows better than oneself if a medical maneuver or procedure may be considered as controlled and then, it could be done autonomously in any moment or, on the contrary, it must be practiced more.
Interestingly, the present study shows that out of the 21 clinical skills and procedures selected, both students and their tutors agreed in a similar optimal value in 16 of them; only in five of them, tutors thought that students needed a greater number of times than that selected by the students.This result shows a moderate mismatch between perceptions of training needs in both groups and there are multiple reasons that are interesting to analyze.
Regarding the achievement of competencies proposed for the internship, the results indicate that competence 1 (diagnose of locomotor system injuries) has been performed a lower number of times than the optimal value would indicate.Clearly, this would indicate that students did not acquire these learning outcomes, but it is important to know that many of these concepts were also taught in the theoretical classes and in the seminars.In any case, it is a sign of that the internship should be modified in order to allow a better achievement of this competence.But, some diseases such as muscular or tendon injuries have a low frequency of presentation, thus we could only improve them by increasing the number of days of the clerkship, a task not always easy.In other cases, such as the severity of injuries and ligamentous injuries, is surprising that in spite of their remarkable frequency of presentation, many students do not acquire them.This could be explained by the organization of the Traumatology Services since in many hospitals, emergencies are seen first by family physicians who request the presence of the traumatologist only for serious cases that may require reductions or surgical treatments.
Regarding competence 2 (identifying by physical examination, the lesions of the locomotor system) and 3 (recognition of lesions by imaging), we can see that they have been highly attained, with some of them showing numbers much greater than the optimal values.This is the case of the imaging techniques, radiographic and magnetic resonances images that the students have seen in a high number.This contrast with the low number of optimal images these same students think they should see.It is clear that reading only 8 to 10 images is clearly insufficient for traumatology, unless that number is applied to each pathology.It is also possible that they see images very frequently in lectures and seminars, hundreds of radiographs and magnetic resonances, so they feel they do not need to observe more in the clerkship.
Competences 4 (treat orthopedically, without surgery and minimally invasive treatment, injuries) has two learning outcomes with values well below the optimal values since they are procedures they should master, at least for the immobilizations and participation in arthrocentesis.the tutors to encourage the student's participation.On the other hand, they see and know how to manage wounds very well.Similarly, competence 5, related to the operating room, has values lower than the optimal and we should work on that, helping them to increase the number of surgical washings and suturing, although these tasks are also seen in General Surgery.We can conclude that they go to the operating room more often than needed, but they are not helping or they are not allowed to help the surgical team.
From the results obtained in this study, it can be deduced that the e-portfolio is a tool that provides information that is difficult to obtain by more traditional assessment instruments (24)(25).However, their use does not substitute those instruments, but complements them (26), helping to improve the quality evaluation of the whole process and of the institution in which the tool is implemented.For this reason, it is necessary to act to optimize the way students are tested (27).
Although the portfolio is currently one of the tools for assessment and learning preferred by students (28), the addition of quantitative elements provides a new level in the assessment of medical students.We also agree with Chertoff et al. (29) and think that there is still room for improvement of the e-portfolio, for instance, analyzing the reliability of these mixed portfolios or the analysis of competencies according to the hospital in which students perform their rotations, and many others.

Table 1 .
Optimal number of times that students and teachers think it is necessary to repeat a procedure or skill to be able to achieve the learning outcome.*,p<0,05

Table 2 .
Actual number of times that the skill or procedure are performed by students during the hospital internship.*, p<0.05 vs optimal mean of students (