Development and Validation of Instruments to Assess Competencies and Knowledge in Sustainability in Nutrition Students

1. Introduction

The world we live in needs professionals who contribute to social transformation, which is why it is essential for universities, as the final step in education, to actively engage in providing sustainability education to students [1,2,3] ensuring that they have the necessary skills to thrive in a sustainable future [2].

In 2017, UNESCO recognized the role of education at all levels in the development of sustainability competencies and in achieving the United Nations’ Sustainable Development Goals [2,4], Education for Sustainable Development, a term used to encompass efforts in sustainability education, is considered key to achieving the goals [5,6]. In response to this need, many universities are integrating components of sustainability into their programs [1], even when sustainability is not a primary component of the courses [7].

Currently, there is a wealth of literature demonstrating the implementation of sustainability strategies and the assessment of sustainability competencies in the fields of engineering, environment, and education. However, many healthcare professions have lagged in defining their role in sustainability [3].

The current food system is faced with the challenge of producing accessible and high-quality nutritious food without overexploiting natural resources. However, at this moment, both are in danger due to irresponsible and excessive use of resources and a lack of regulation in production processes. This is why sustainability is key in strategies to transform the current food production model [8,9]. The training of nutritionists is calling for a paradigm shift in the science of nutrition and food systems, integrating physiological aspects of nutrition and health with environmental health [10]. Nutrition students must understand and apply the concepts that define sustainable nutrition. Therefore, universities must develop, implement, and evaluate practices to strengthen the acquisition of sustainability competencies [9].

The present research aims to validate that the use of an intervention based on the systemic approach to nutrition can contribute to the development of sustainability competencies in nutrition students. It also seeks to validate the assessment of such competencies through an instrument.

1.1. Definition and Framework Model of Competencies in Sustainability

Competencies are a way to describe educational goals and include cognitive, functional, ethical, and personal dimensions. They are relevant because they link the knowledge, skills, and attitudes of students [11,12].

In the last 10 years, there has been a growing interest in defining competencies for sustainable development. This is because the culmination of education in sustainable development is reflected in the development of sustainability competencies, which go beyond the acquisition of knowledge and should encompass all cognitive capabilities and skills, as well as the social and motivational preparation to address the transdisciplinary challenges posed by sustainability [5].

By promoting sustainability competencies, the aim is to empower individuals not only to understand the fundamental principles of sustainability but also to develop the capacity to apply that knowledge in the effective resolution of environmental, social, and economic problems.

Despite advances, it is currently rare for there to be a plan for teaching sustainability competencies, leading to teaching sustainability topics without considering students’ prior knowledge. That is why it is vital that teachers are prepared to assess their students’ competencies before starting a course; otherwise, students may develop fragile learning around change and sustainability [7].

There are many models of competencies for sustainability developed by different authors, among which the following stand out:

Wiek, Withycombe, and Redman propose [13] 5 groups of competencies: systemic thinking, anticipatory thinking, normative thinking, strategic thinking, and interpersonal thinking. Rieckmann [14] suggests the following twelve competencies: systemic thinking and complexity management, anticipatory thinking, critical thinking, ecological action, cooperation in heterogeneous groups, participation, empathy and a perspective of change, interdisciplinary work, communication and media use, planning and implementation of innovative projects, evaluation, tolerance for frustration and ambiguity. Lozano in 2017 proposes 12 competencies that are divided into 3 groups: Universal, Community-based, and Social Justice and Environmental Education [15] (Bianchi, 2020 C.E.).

Due to these differences in definitions, UNESCO released a report in 2017 incorporating a series of key competencies for sustainability that could help students work in alignment with the goals of the 2030 Agenda for Sustainable Development [4], at this moment, the following competencies are considered key for education in sustainable development [7]: Systemic thinking, anticipatory thinking (future-oriented thinking), values thinking, strategic thinking, interpersonal thinking, intrapersonal thinking, implementation thinking, and comprehensive problem-solving are considered key competencies for education in sustainable development.

1.2. Assessment of Competencies and Knowledge in Sustainability

There are empirical studies that describe methodologies for assessing the acquisition of competencies in sustainability [1], however, there is no consensus on how to do it effectively. Studies in education have indicated that traditional assessment methods are not suitable for evaluating competencies in sustainability [17] due to their systemic and multidisciplinary origin and their link to performance [1,3,18].

For the development of items in the instruments, 30 empirical studies on the assessment of competencies in sustainability were consulted, and the competency framework model presented by Annelin & Boström et al. was consulted. In order to design a valid and reliable instrument, a triangulation of instruments was conducted among the instruments presented by Yüksel and Yilmaz Önaly et al. and Irazusta-Garmenia et al. that assess knowledge in sustainability in health area students, and the instrument presented by Savage et al. that assesses the development of sustainability competencies in university students (See Figure 1). Additionally, the results obtained from semantic networks were used to adapt the instruments to the Mexican context. Triangulation of instruments is a research strategy that involves comparing information obtained from different sources or tools in order to understand a phenomenon in a more reliable and comprehensive manner. It involves using multiple methods, techniques, or data collection sources to address the same research question from different perspectives, and is considered a powerful technique to improve the reliability and validity of a research instrument.

2. Materials and Methods

2.1. Design and Triangulation of Instruments to Assess Competencies and Knowledge in Sustainability

In order to adapt the constructs used in the instruments found in the literature to the Mexican population, 43 nutrition students and professors from a university in Mexico were asked to write down 5 words related to sustainable nutrition, and they were also asked to list them in order of importance. A value of 5 was assigned to the term mentioned as the most important for sustainable nutrition, and a value of 1 was assigned to the term mentioned as the least important. This technique was used to identify the most relevant concepts in sustainable nutrition among a population of nutrition faculty and students in Mexico. These results were used to ensure that the constructs within the instruments were well conceptualized, and they also helped to identify relationships between the concepts in order to assign attributes or properties to them. The internal consistency of the semantic network table was measured, and a Cronbach’s Alpha of 0.71 was obtained, indicating reliability in the matrix.

2.2. Instrument to Assess Sustainability Knowledge

Using the semantic network technique, the instruments presented by Yüksel and Yilmaz Önal were adapted [19] and by Irazusta-Garmenia et al. (Irazusta-Garmendia et al., 2023) For the construction of an instrument that could be used to assess knowledge about sustainable nutrition in students in Mexico, a questionnaire was developed. The questionnaire includes 38 questions on knowledge of basic concepts in sustainability, priority of concepts in food sustainability, and impact and production of food. It includes true and false questions, priority ranking using a Likert scale, and rating of environmental impact of production. Responses regarding knowledge in sustainable nutrition are assessed using a rubric where correct answers receive 1 point and incorrect answers receive 0 points. Knowledge in sustainability is evaluated by summing the correct answers and calculating a percentage (See Table 1).

2.3. Instrument to Asses Sustainability Competencies

Thirty instruments were reviewed to evaluate the development of the 8 key competencies in sustainability. Most studies have not applied simultaneous measures to all key competencies, so it was necessary to adapt an instrument to evaluate the 8 key competencies in nutrition students in Mexico [2,21]. The instrument developed by Savage et al. was reviewed [22], it is based on a self-perception questionnaire of the participants about their level of confidence in acquiring 5 key competencies after taking a sustainability leadership course. The instrument includes 3 questions for each key competency, and each question uses descriptors used by Wiek et al. [13] additionally, the instrument uses a Likert scale (1-4) (1-Disagree) to (4-Agree) to evaluate the confidence and competence to perform each of the actions described by the instrument [22,23].

2.4. Study Design

A quasi-experimental, cross-sectional, and quantitative research was conducted at the end of the semester August-December 2023, where a non-probabilistic sample of students from the Bachelor’s Degree in Nutrition and Integral Well-being received a systemic intervention on sustainable nutrition. At the end of the workshop, self-assessment instruments of competencies and knowledge in sustainable nutrition were administered and compared with the results obtained by the control group. Descriptive statistics were used to measure the development of competencies and knowledge in sustainability, the non-parametric Mann-Whitney U test to measure differences between the groups, and the Spearman correlation test to measure the strength of correlation between knowledge in sustainability and development of competencies in sustainability.

2.5. Sample

A total of 45 3rd year nutrition students between 18-24 years old answered the instruments.

3. Results

3.1. Analysis of Tools for Assessing Sustainability Competencies

During the systematic review of articles, a variety of measurement tools were identified. Some studies used more than one tool; however, the most commonly used tool with 15 mentions (46.8%) was the use of self-assessments, where students respond based on a specific scale, rating their competencies, knowledge, and attitudes in sustainability (See Figure 3). The tools found were categorized after reading the descriptions and reaching a standard terminology for each of them.

Of the studies analyzed, 66.6% used a quantitative methodology for competency assessment, 20% used a qualitative methodology, and 13.3% used a mixed methodology incorporating tools such as interviews, focus groups, and mind maps (See Table 2).

3.2. Reliability of Scales for Measuring Competencies and Knowledge in Sustainability

To test the reliability of the instruments, Cronbach’s alpha was calculated, which is the most common measure to assess the internal consistency of the instruments (See Table 3).

The reliability of the scales used in all competency dimensions was very satisfactory (Cronbach’s α ≥ 0.89); however, this value should be interpreted with caution due to the difficulty of measuring complex constructs. As indicated by Schmitt et al., when a complex construct has other important characteristics, such as the incorporation of new content, moderate reliability, as in the case of the scale used to assess sustainability knowledge (Cronbach’s α = 0.59), should not be a hindrance to using an instrument. Nevertheless, for future applications, it is suggested to consult with experts to increase the internal consistency of this instrument [24].

3.3. Validity of the Scales for Measuring Competencies and Knowledge in Sustainability

Another measure to determine the quality of the instruments is validity, which is understood as the ability of the instrument to measure what it intends to measure [5]. There are 3 types of validity: convergent, content, and external.

Convergent validity compares the instrument with other instruments. Since there are few instruments that evaluate competencies and knowledge in food sustainability, it is difficult to determine this type of validity with this instrument [25]. Content validity refers to the extent to which the instrument incorporates the different facets of food sustainability. Since the instrument was adapted from previously used instruments based on the literature, it could be considered that content validity was satisfactory [25]. External validity refers to the relationship between the results of the instrument and other assessments. It is expected that students with greater knowledge in sustainability will achieve higher scores on the competency and knowledge scales [25].

The average scores in the level of acquisition of competencies and knowledge in sustainability are higher in all dimensions for the students who underwent the intervention (see Table 4 and Table 5), which strengthens the external validation of the instruments.

Using the Mann-Whitney U test, the results obtained in the intervention group were compared against the results in the control group. The students who received the intervention achieved significantly higher scores in all sustainability sub-competencies (W=100, p=0.019) (see Figure 6), as well as in the total percentage of sustainability knowledge (W=53, p=0.01) than the students who did not receive the intervention, which aligns with expectations (see Table 6).

The level of development of sustainability competencies should be correlated with sustainability knowledge. By measuring the strength and direction of the correlation, it can be observed that there is indeed a weak correlation (0.25) between the two (See Table 7). However, in social sciences, Senthilnathan points out that a correlation between 0.2 and 0.3 is weak, but it should still be considered for analysis [26].

4. Discussion

The development of sustainability competencies in higher education students is critical for them to become agents of change [18], there are many tools that have been used for research and academic purposes; however, several of these tools have not been applied correctly to the context due to a lack of available information. [1], Therefore, it can be concluded that empirical studies are needed to help implement and evaluate the development of competencies in higher education students in Mexico.

Regarding the instruments used, it can be concluded that the questionnaire used to assess the development of sustainability competencies shows high internal consistency. The dimensions used in the instrument (Systemic Thinking, Anticipatory Thinking, Values-based Thinking, Strategic Thinking, Interpersonal Thinking, Intrapersonal Thinking, Implementation-focused Thinking, and Comprehensive Problem-solving) are suitable for evaluating competency development in nutrition students, suggesting that it is a robust instrument with the potential to be applied. However, the tool used to assess sustainability knowledge would benefit from being divided into dimensions to increase the internal consistency of the scale and measure the impact of each dimension on the acquisition of sustainability knowledge. Nevertheless, it can be seen that the use of an intervention based on a systemic approach to nutrition has a positive impact on the development of competencies and the acquisition of sustainability knowledge, as the average score obtained in the evaluation of sustainability sub-competencies was higher in the intervention group than in the control group in all cases. Additionally, it can be concluded that there is a significant difference in the level of acquisition of competencies and knowledge between the intervention group and the control group.

The findings in the literature review align with the systematic reviews published by Cebrian et al. [18] and by Redman et al. [27] Self-assessment questionnaires on the acquisition of knowledge, competencies, and attitudes in sustainability were the most commonly used assessment tools. The reasons cited by the authors are aligned with the fact that self-assessment questionnaires are quick to complete, easy to distribute and administer to a large population, and provide a wealth of information about the teaching-learning process [1,28], additionally, self-assessment questionnaires represent a practical way to assess competencies because externally evaluating them is very complicated and laborious for instructors [29], however, regardless of being efficient tools for practical reasons, they represent good pedagogical practices [29]. There are studies that support the educational validity of using self-assessment questionnaires to evaluate sustainability competencies due to the formative nature of sustainability competencias [29], in theory, students have complete access to the necessary data to respond to self-assessment because they can reflect on how confident they feel in carrying out certain actions [29]. In pedagogical terms, self-assessment is a competency and therefore should be developed through practice, as they are educational activities that yield a lot of data to help understand students’ learning experience [29,30].

Self-assessment questionnaires on the level of competency acquisition are widely used to measure acquired skills and professionalism. Although they have received some criticism, many authors support the validity of self-assessments for measuring the level of competency acquisition as they capture students’ self-perception regarding their own skills [31,32]. Some authors point out that accurate self-assessment of competencies is necessary for lifelong learning (Yeo & Chang, 2019)

5. Conclusions

The article describes the development of two instruments, one focused on assessing the development of competencies and the other on evaluating the acquisition of sustainability knowledge in nutrition students at a university in Mexico. While there are many publications that have studied the evaluation of sustainability competencies, this study demonstrates that the educational tool of self-assessment can be incorporated into the instruments used in the assessment of competency development and knowledge acquisition in nutrition students. Self-assessment tools facilitate the application of instruments, provide a wealth of information, and pedagogically empower students in their own learning. Additionally, they empower teachers to guide and lead activities aligned with competency fulfillment, center learning on the student, and above all, can help measure the level of learning and mastery of the subject after implementing an educational strategy.

Furthermore, based on the results of reliability and validity, it can be concluded that both instruments can be used to assess the development of competencies and acquisition of sustainability knowledge in nutrition students. However, the instrument used to assess knowledge would benefit from being subdivided into dimensions to evaluate internal consistency more accurately. Additionally, the results reveal that the application of an intervention based on the systemic approach to nutrition has a positive impact on the development of competencies and the acquisition of sustainability knowledge in nutrition students in Mexico. Further research can evaluate the international as well as interdiciplinary applicability of these findings.