GreenComp-Based Questionnaire (GCQuest): Questionnaire Development and Validation

1. Introduction

It is universally recognised that the urgency of sustainable practices is paramount and requires a collective commitment by humanity, underpinned by ongoing research [1]. The application of scientific knowledge must involve the global community and promote initiatives that are both globally relevant and contextually specific. Consequently, the development of sustainability competencies should begin with a thorough assessment of local realities and be supported by tailor-made collecting data tools to evaluate the effectiveness of implemented actions [2,3,4].

Previous work included a literature review of data collection tools focusing on the evolution of the concept of sustainability and its competencies, and of its assessment and resulted in an initial mapping of existing explicit statements on data collection tools, like questionnaires [5]. Despite the high number of results listed, the lack of explicit statements of data collection tools makes their use impractical, and as stated, most of the existing questionnaires are focused on Higher Education students or in different subdomains of Sustainability, like Business or Tourism.

Afterwords, a reflection paper, which resulted in the definition of the GreenComp framework [6] as a source of statements for the questionnaire [7]. The third stage, which is the subject of this paper, seeks to present the methodological processes and dynamics that culminated in the questionnaire that will be presented below. Concerning that, the purpose of this paper is two-fold: initially, to elucidate the evolution process and methodology employed in the creation of a questionnaire; and subsequently, to expand upon and situate its application. The questionnaire development itself was based on a mixed methods study design concerning three phases being the last one, based on statistical data, namely the reliability and validity of the data collection tool.

The paper follows an integrated structure concerning the i) introduction, ii) the theoretical context for sustainability competencies questionnaire development, iii) the instrument development methodology, and iv) Final considerations. Also. Are presented several appendixes. Its inclusion is essential to understand all the undertaken processes.

2. Theoretical Context for Sustainability Competencies Questionnaire Development

The data collection tool development required to consider previous research on competencies for sustainability, namely the work by Wiek and colleagues, as explained below.

2.1. Research on Competencies for Sustainability

“Sustainability” was introduced in the academic and public discourse in 1987. The Brundtland Report [8] presented to a broader audience the concept of sustainable development and brought about a significant increase in the visibility and impact of the sustainability debate. Since this date, the amount of research on this subject has been considerable and it is expected to continue to increase, making available a considerable amount of information concerning competencies for sustainability [9]. Yet, as Annelin & Boström [2] point out, the literature reveals ambiguities in the way it assesses competencies for sustainability, even though these are well defined [9,10].

Cébrian, Junyent and Mulà [4] point out that, despite the large number of initiatives and contributions in this field, there is a need to step up research, not only into the operationalization of competences for sustainability, but also into the design and testing of assessment tools to determine the development of these competences in a wide range of educational levels and contexts. Several authors also stated that sustainability research must be contextualised to the reality of each country or context [2,9,11,12]. Regarding this, previous work revealed that sustainability competencies assessment tools and its explicit statements are lacking in the literature [5].

Based on the need of more research on sustainability, The research and development project (https://) is being developed. This project combines the potential of mobile augmented reality games in an urban context with the promotion of sustainability competencies. The also demanded a collection data tool aimed to respond to its research question. “How does an intelligent urban learning environment that integrates a user-friendly application with co-created augmented reality games and resources with pathways and challenges, designed to involve educational actors and the community in general, promote changes in citizens’ knowledge, skills, values and attitudes, empowering them for sustainable development?” [13]. This understanding of the concept of competence(s) is consistent and informed by the ones that set forth by the European Council regarding the concept of key-competences for lifelong learning [14].

In addition to this challenge, there was another one that connects with the very nature and dynamics of the project, the initial motto for this research. In light of the fact that the project is based on the exploration of game-based learning, through mobile devices with augmented reality content, which guide players through pre-established paths during the creation of various games, it was necessary to analyse the theoretical framework in order to consider the specificities, mobilization and scope of the games themselves [15,16,17].

Therefore, it was essential to have a questionnaire wider enough to respond to own needs, and to be used in different research or contexts.

The development of a data instrument requires a comprehensive understanding of the relevant research domain, which should guide all stages of instrument formulation [18,19]. Specifically, the development of a questionnaire to assess sustainability competencies is based on the contextual activities designed to foster these competencies within the project. Furthermore, the instrument must be robust enough to be applied in analogous contexts and accessible to speakers, which requires a new validation process for the integrated items derived from previously validated questionnaires, to reduce potential misinterpretations [20].

2.2. GreenComp as an Inspiration for Sustainability Competencies Assessment

The “GreenComp-based Questionnaire” (GCQuest) is presented in detail.

2.2.1. Why Use GreenComp to Assess Sustainability Competencies?

The rationale behind the use of GreenComp [6] as a source of statements to be included in the questionnaire is based on: a) It is a recent theoretical document; b) It is part of a broad context of references and guidelines that originated from the European Green Deal [21]; c) It offers an integrative and multidimensional perspective on sustainability; d) It is a reference for educational and training initiatives; e) It provides a detailed account of its design process; f) It was developed and validated by a large team of European researchers; and g) It presents the statements explicitly. The direct use of the 169 GreenComp statements was not considered adequate, due to the large extension of such data collection tool and they would require adaptation to the typical formulation of questionnaire items (interrogative vs declarative) [22].

GreenComp as framework is organised to stem comprehensive knowledge, skills, and attitudes (KSA) that are essential for fostering sustainability-oriented behaviours. It comprises twelve competencies, which are grouped into four interdependent areas. The four interdependent areas are as follows: “Embodying sustainability values”; “Embracing complexity in sustainability”; “Envisioning sustainable futures”; and, “Acting for sustainability”. These four interdependent areas comprise, in total 12 competences. These are organized 3 by 3 in each area, prevailing a total of 169 statements. Collectively this organization form a framework that guides individuals from an ethical grounding to impactful action. This structure reflects GreenComp’s commitment to a multi-dimensional understanding of sustainability. It ensures that values, analysis, foresight and action are seamlessly integrated to support sustainable decision-making and practices.

The first area, “Embodying sustainability values,” encompasses competencies pertaining to the valuation of sustainability, the promotion of fairness, and the advancement of nature. Collectively, these competencies establish a fundamental ethical framework for sustainable action. The emphasis on valuing sustainability encourages individuals to perceive sustainability as a guiding principle, prioritising environmental and social well-being. The concept of supporting fairness emphasises the necessity of equity and inclusivity, addressing the social dimensions of sustainability and underscoring the importance of just and ethical practices. The promotion of nature encourages individuals to understand and through it, to respect ecological systems, thereby fostering a sense of interconnectedness with the natural world. Together, these competencies intend to establish awareness towards sustainability, wherein individuals perceive it not merely as an objective to be attained, but as a deeply held value that informs both personal and collective actions.

The second area addresses the analytical competencies necessary for navigating sustainability’s interconnected challenges through systems thinking, critical thinking, and problem framing. This is achieved by embracing the inherent complexity of sustainability. Systems thinking promote individuals to understand the interconnections between ecological and social systems, aiming to ensure a holistic perspective that aims to solve complex environmental issues. The application of critical thinking enables learners to subject information to rigorous evaluation and analysis of sustainability challenges in an objective manner. Problem framing, in turn, is concerned with the identification and definition of the fundamental issues within complex scenarios, which represents a crucial stage in the development of effective and well-informed strategies. This area fosters the development of critical competencies that equip individuals with the ability to analyse and navigate complex sustainability challenges, thereby preparing them for informed, context-sensitive decision-making.

The third area, entitled “Envisioning Sustainable Futures,” encourages the development of forward-looking competencies, including futures literacy, adaptability, and exploratory thinking. These are crucial for envisioning and shaping sustainable pathways. Futures literacy enables individuals to conceptualise and prepare for a range of potential sustainability scenarios, fostering a proactive mindset. The capacity for adaptability allows for flexibility and responsiveness in the face of environmental and social change. Furthermore, exploratory thinking encourages creativity in problem-solving and innovation for sustainable solutions. Such competencies foster a mindset that not only anticipates future challenges but also engages in the creation of resilient and sustainable outcomes. In conclusion, the competence of acting for sustainability encompasses the abilities to engage in collective action, to take initiative, and to act in accordance with one’s awareness and intentions. Political agencies provide individuals with the capacity to engage with policies and advocate for sustainability at institutional levels. Collective action underscores the significance of collaboration in attaining sustainable objectives, whereas individual initiative underscores the personal accountability for propelling sustainability in everyday actions. The structured pathway from values to action, as outlined by GreenComp, provides a balanced framework that fosters sustainability as both an ethical commitment and a practical imperative.

These areas are structured through three competencies that are of equal value and are interconnected, thereby ensuring a holistic approach to sustainability education.

Each competence is presented through knowledge, skills, and attitudes (KSA) statements, which provide a multi-dimensional perspective on the expectations within each area. Knowledge statements define the foundational understanding necessary for each competence, skills statements specify actionable abilities, and attitudes statements highlight the mindsets that support sustainable behaviours. This tripartite structure ensures that learners are equipped with not only the conceptual understanding and practical abilities but also the ethical perspectives needed to meaningfully engage with sustainability challenges.

This last consideration is in perfect alignment with the terminology employed in the development of the research question for the research and development project (https://), which serves as the prima facie basis for the development of this research work.

Another challenge that emerged was linked to the number of statements to be included in the questionnaire that was to be developed. This led to the necessity of reducing the number of statements in the questionnaire itself. To address this issue, an in-depth analysis of the theoretical-base framework was undertaken to establish cutting principles that were well-founded and duly integrated with a complex, multidimensional vision of Sustainability [23].

2.2.2. Reading GreenComp and Understanding Its Purpose

GreenComp [6] has the objective of supporting educational and training systems in cultivating individuals towards systemic and critical thinking skills and demonstrating concern for the present and future well-being of our planet. Bianchi et al. [6] stated that the competencies delineated in the framework are considered suitable for all students, regardless of their age or academic achievement. Also, the framework is applicable to a range of educational settings, including formal, non-formal and informal contexts, and is poised to enhance global efforts to cultivate sustainability literacy. GreenComp serves as a flexible and non-prescriptive guide that promotes educational initiatives focused on sustainability, influencing both individual and societal levels while addressing pressing environmental issues. To promote a sustainable future, it is imperative to integrate equity, justice and a deep connection to nature within a lifelong learning framework that empowers individuals to engage holistically in sustainability practices.

2.2.3. Merging Games’ Features and GreenComp: The Potentialities of Mobile Augmented Reality Games and the Assessment of Competencies for Sustainability Based on the Framework

To gain a deeper understanding of the potential of games, it was essential to identify their limitations. One notable limitation is that, in this research context, players are playing games only once. Literature suggests that the limited time allotted for playing may inhibit the full development of players’ attitudes, skills and knowledge. This limitation may also hinder the identification of stronger links between game elements and educational outcomes [16,17]. All game dynamics and its multimedia and AR resources must be designed with this time limitation and its potential effects in mind. Furthermore, the team, based on literature discussed the potential of games, not to alter fundamental policies, but to alter mentalities and raise awareness of sustainability issues and the importance of effective sustainable development. This is stated by Fabricatore and López [24], referring “that [sustainability] games are not fully leveraged to develop mindsets and skills required to engage with sustainability” (p. 10). More recently, Boncu, et al. [25] also referred that while games may not directly affect policies, their capacity to enhance individual perspectives and conduct is essential for fostering efficient sustainable development and advocating enduring pro-environmental behaviours. In parallel with this discussion, an analysis of GreenComp framework revealed that the “Embodying sustainability values” [6] competence area was accorded the highest priority. This priority is largely justified by the time constraints of the activities and games. This approach ensures that the internal dynamics of the games facilitate the raising of awareness of sustainability issues and the importance of effective sustainable development [13].

It was therefore necessary to develop a new questionnaire that would address the above issues by adopting or adapting the GreenComp Framework competence area “Valuing Sustainability” [6]. To this end, the development of this tool followed several methodological principles and stages.

3. Instrument Development Methodology

The questionnaire was developed and validated using a mixed-methods study design [26]. Its development was divided into three phases: Phases 1 and 2 (qualitative phases) for the development of the questionnaire, and Phase 3 (quantitative phase) for the validation of the questionnaire. The procedures related to these phases are described in the following sub-points. Approval under the General Data Protection Regulation was obtained prior to implementation (Appendix C). Informed consent was always given, in both digital and paper versions.

3.1. Adopting or Adapting the GreenComp Framework area “Embodying Sustainability Values”?

As mentioned above, the question of adopting or adapting the GreenComp framework does not cover all the competence areas identified in the framework, as the development of this questionnaire is based on the competence area ‘Valuing Sustainability’ only. The analysis of this area of competences identified 42 statements. Given the specificities of the activities and games, its outdoor context, and the application of the questionnaire at the end of the activities, the team, based on relevant literature concerning this, that was considered necessary to reduce the number of statements to be included in the questionnaire, knowing that the longer the instrument, the greater the threat to validity [27].

The process of cutting statements was conducted in accordance with two criteria, namely, i) the collaboration of three national experts and ii) the implementation of several pilot-tests [28]. Additionally, the involvement of two international experts was sought, who agreed to participate in this research. This process involved content analysis of the statements made by the experts and face validity analysis based on different target groups. Subsequently, the questionnaire integrates 25 statements, each measured using a 6-point Likert scale [29], where 1 corresponds to “I disagree” and 6 corresponds to “I agree”[30,31]. Traditionally an odd scale is used in the literature, but in this questionnaire, an even scale was used to avoid neutral responses from respondents [27]. Given the context of application, the language version of the GreenComp framework was used (https://op.europa.eu/en/publication-detail/).

3.1.1. Cutting Items Process

The process of selecting statements from GreenComp to create the questionnaire was based on several criteria, with the aim of ensuring its validity and usability [32].

3.1.1.1. Expert’s Analysis

Three national experts were invited to analyse a draft version of a hypothetical questionnaire. The experts were selected based on their published work and contacted by professional their email address. Upon acceptance of the invitation, the draft version was sent, requesting that the analysis be as explicit and open as possible. The draft version of the questionnaire is attached as Appendix A.

The outcomes of the three experts’ analyses are presented in the Appendix B and had several points of convergence. These included: a) the modification of certain words to enhance comprehension of the statements in question; b) the necessity to elucidate certain concepts in some of the statements; c) the implementation of grammatical corrections with the objective of enhancing comprehension of the statements in question; and d) the identification of several redundant statements. Care was taken to ensure that the analyses carried out by the experts were valued but did not detract from the statements made in the theoretical framework, which had already been developed and validated by various experts within the European academic context.

3.1.1.2. Pilot-Testing

At the same time as the three experts were invited to analyse the draft version of the questionnaire, two pilot-tests of the original questionnaire were carried out among adult population. Pilot testing is useful for evaluating the effectiveness of statements, although a pilot study is not usually carried out before the instrument is submitted for external evaluation. A pilot test involves administering the questionnaire to a sample drawn from the targeted population, or not.

The first pilot test was carried out with the draft questionnaire in an online version, and the link to fill it in was published in various posts in groups on a well-known social network. Adults were asked to complete the questionnaire. This pilot-test had a total of 47 respondents. The second pilot-test was carried out also with the draft questionnaire in an online version. The filling link was shared to a group of 30 teachers of Basic and Secondary Education on an training course.

Before filling in the questionnaire, it was approved by the University’s General Data Protection Regulation Office (Appendix C). Participation was voluntary and involved informed consent.

3.1.2. The Pilot-Tests Results’ Analysis: The Alphas and Omegas of the Questionnaire

Researchers can examine the pilot-test data to identify items that have been answered significantly differently from the others. These items do not fulfil the purpose of distinguishing between respondents or to assess if the answers are correct or incorrect. A lack of correlation between items indicates a specific problem, which may endanger the intended construct. Conversely, highly correlated items may reveal another problem, suggesting minimal differentiation between items, and thus limited or absent discrimination. In the case of questionnaires containing filter or distractor items, these can be evaluated through a pilot test. Confirming the expected functionality of these items in a pilot study strengthens the argument for the validity of the instrument [27]. To ensure validity and reliability of the questionnaire, the collected data was statistically analysed using JASP software version 0.18.3 [33]. The use of this software facilitated the analysis of the data and its correlations [34,35]. This analysis process was complex, and its results were, along with other combined data, essential for the final questionnaire.

The analysis of this data allowed us to understand the selection of items to be cut to realise this new questionnaire on sustainability competences based on the GreenComp framework. Also important is the structure and internal organisation of the questionnaire [27]. These aspects were also considered during this phase.

Concerning these, the initial statistical analysis that led to the subsequent cutting of the statements is shown below, in Table 1 and Table 2, which refer to the response data collected in the two pilot tests, adult Public and Teachers, labelled Pilot Test 1 and Pilot Test 2 respectively.

Overall, the statistical analysis shows, on the one hand, very high overall Cronbach’s Alpha (α) and McDonald’s Omega (ω) values, but when the same variables are analysed according to the different sustainability domains, more acceptable values are found. According to Taber [36] and Kalkbrenner [37] Cronbach’s α can be interpreted as presented on Table 3.

Concerning McDonald’s ω, and according to Green & Yang [38], higher values on ω signifies greater confidence in the result analysis. This is also stated by Kalkbrenner [37], indicating that ≥.65 to .80 values are acceptable, and >.80 are strong values.

Considering the high values of both Cronbach’s α and McDonald’s ω along all items, the experts’ analysis were essential to determine the statements to be dropped, originating the first version of the questionnaire.

3.2. The first Version of the Questionnaire

The process described in section 3.1 resulted in a questionnaire comprising 25 positive statements. Its use enables the mitigation of acquiescence bias, which is especially pronounced when negative items are employed, resulting in confusion and inconsistency in responses [39]. Furthermore, the utilisation of exclusively positive items enhances internal consistency, as evidenced by the elevated values of Cronbach’s alpha. These factors suggest that the responses may yield more reliable data [39].

Based on Cobern and Adams [27], the random organisation of the statements was undertaken with the following considerations in mind: a) The grouping of similar items (in this case by competencies and domains) may present a potential risk in that the initial items in the group may affect responses to subsequent items; b) Dispersing similar items throughout a survey is considered appropriate, usually achieved by randomly assigning positions; c) It is advisable to position critical items at the beginning of the survey, since that respondents’ focus tends to wane on lengthy questionnaires; and d) A potential criterion for the ordering of items is the level of reflection required by each content item [27].

A free random number generator software tool (https://pickerwheel.com/tools/random-number-generator/) was used, which precluded the possibility of c) and d) being completed, while simultaneously reducing the potential for bias on the part of the authors.

Two direct response questions were included at the beginning of the questionnaire to assess both attention during the activity and the respondent’s understanding of key concepts, such as sustainability. This filter is designed to reinforce the argument for the questionnaire’s validity [27].

This questionnaire is available at https://doi.org/.

4. Final Considerations

The process of developing this questionnaire has been both challenging and rewarding. At the same time, we have sought to create a versatile tool that will enable the development of further data collection and research in this area. As mentioned, this article is the culmination of a series of three articles which together demonstrate the task that has been undertaken. The creation of a questionnaire that can be used in a variety of contexts reaffirms the need for further analysis and restructuring, as several authors, including Armin Wiek and many others, have noted. This demonstrates the continuing need for further research into sustainability issues. It is possible that the interpretation of the static analysis may be biased in some dimension, but we have endeavoured to use the literature to provide an informed and enlightened analysis of the data with a view to developing valid work on this matter. Another key issue is the appropriateness of items for different audiences. In this study, the items were kept as faithful as possible to the original GreenComp [6] text wording to avoid distorting the original meaning of each item. However, sentences on sustainability issues tend to be complex and difficult for younger respondents to understand. Therefore, further work may include adapting the questionnaire for school-age learners. It is also stated in GreenComp [6] and other academic literature that sustainability should be understood as a complex concept. In this sense, it could be considered reductive to have chosen only one area of competence. However, this is due to the number and complexity of the statements and the links between them in the framework.

This justifies further research and the development of questionnaires specific to each area of sustainability competencies, or even an integrative questionnaire comprising all the statements. Given the pressing need for effective and sustainable development, it is imperative to recognise the need for further research in this area. Consequently, although the first version of this questionnaire has been completed, as mentioned above, the development of a more complete questionnaire based on GreenComp [6] remains to be undertaken. The process will be lengthy and comprehensive, demonstrating the value of collaborative and multidimensional work. The different perspectives, understandings and specificities of the various contributions will culminate in revised questionnaires that will be extensive and complex, given the complexity of sustainability concept and its field or research.

The questionnaire is being applied to the users/players of the app games and activities. Following the data collection, the two foreign researchers have yet to analyse the data obtained, which will be done as soon as there is enough data for a broader, valid and binding analysis. Also, in future research, it would be essential to use this questionnaire over an extended period with the similar population to establish its reliability. Furthermore, it is important to test the validity and reliability of this questionnaire in different application contexts, namely in-speaking countries, aiming to identify areas for improvement and to establish it as a relevant tool in education for sustainability.