Students as Experts-In-Training for Quality Assessment of Computer Science OERs

1. Introduction

The use of Information and Communication Technologies (ICT) in education has increased significantly in recent decades, leading to the coexistence of traditional and digital educational resources. Digital educational resources [1] are digital elements designed to support learning processes and can vary in their structural and functional granularity. Basic resources may be combined into learning objects, which possess specific didactic functionality and structure [2]. These learning objects are considered smaller but complete educational units and usually integrate theoretical content, concept maps, learning activities, and assessment elements [3]. The availability of digital educational materials has grown exponentially, allowing access through the Internet anytime and anywhere, thus improving accessibility for students in remote areas or with disabilities. However, this easy access contrasts with the difficulty of efficient faceted search, especially for teachers, who often spend excessive time reviewing unsuitable recommended resources. Moreover, simple learning resources frequently lack explicit functionality. For example, a multimedia resource such as a single image may support learning but cannot always be considered functionally complete for educational purposes.

A good solution would be to provide educational resources through dedicated portals integrating recommender systems [4,5], so that digital resources are dispensed, suitable for learning needs (even for users with disabilities), efficient, adequate in their techno-pedagogical characteristics and with an assured level of quality. Three specific situations arise from this fact [6]. First, the need to establish standardized mechanisms for labeling, classifying, cataloging and describing all available resources, so that they present a sufficiently clear structure to allow efficient searches on the portals that contain them. Metadata helps to solve this problem as they contain structured information about the information represented by the educational resource itself, many of them having to do with the teaching-learning process itself, such as difficulty, type of interaction, age range, context, end users, etc. The use of metadata in recommender system algorithms facilitates the identification, description, classification and efficient search, and therefore retrieval, of digital educational resources by all users (students, teachers) and even other software systems that can provide value-added services to the learning process [7]. In this sense, the LOM (Learning Object Metadata) or IEEE 1484.12.1 2002 standard is internationally recognized for use in the description of learning objects [8]. In this standard, learning objects are defined as “entities, digital or non-digital, that can be used for learning, education or training [9].

The second issue refers to the need to store and classify resources in such a way as to facilitate their maintenance, location and sharing. This is achieved thanks to the so-called Repositories of Learning Objects (ROA) and they represent a key infrastructure in any e-learning process [10]. Likewise, there are standards for the development of e-learning that set the standard in the creation of systems that integrate applications for online teaching and learning processes [11,12], in which content can be reused and shared between users and between computer systems or harvesting services [13]. Potential users of materials need to be able to locate relevant material and to assess it with respect to a number of factors (such as suitability for purpose and license requirements). Discoverability is an issue for web resources in general, for those seeking reusable learning materials, some of the most basic requirements are still the most elusive [14].

The third issue is the need to evaluate the quality of digital educational materials. There are several facts that hinder the evaluation of digital educational resources [15]. The first aspect refers to the possibilities offered by new technologies so that anyone can autonomously generate and self-publish all kinds of resources, including educational ones. On the one hand, this situation favors the dynamization and democratization of the production of educational resources, but it also has as a counterpart in the generation of a huge number of materials whose quality cannot be assured and whose viability for use in formal educational environments is unknown. Therefore, the process of filtering digital educational materials is a key step in their evaluation [16]. The second fact is more related to the process of adequacy and integration of the evaluation of digital materials to the currently applied editorial quality models, as well as to the need or not to adapt already existing quality criteria to new materials [17] identifying specific quality indicators that might be used to provide information on which materials to recommend to users [18]. The third issue is the proliferation of generative artificial intelligence in educational content creation, which poses new challenges for evaluating authenticity and originality of resources [19].

The best way to evaluate the quality of digital educational resources is the use of evaluation standards for digital educational resources. In this article, we have selected the UNE71362:2020 standard, a suitable tool for conducting a holistic assessment [20,21,22,23] and in whose development some of the authors have actively participated. Recent advances in the field of Open Educational Resources (OER) evaluation include the use of machine learning techniques for automatic quality assessment [24], the development of recommendation systems based on semantic metadata [25], and the integration of user-centered usability metrics [26]. These approaches complement traditional manual evaluation methods, offering scalability and consistency in evaluating large volumes of resources. Furthermore, standards such as QTI 3.0 (Question and Test Interoperability) and xAPI (Experience API) are redefining how the effectiveness of educational resources is measured and tracked in terms of real learning outcomes [27]. The integration of learning analytics allows correlating the perceived quality of resources with their measurable impact on student performance [28].

Considering all the context above, this article poses three research questions:

RQ 1. After measuring and comparing the quality of educational resources produced in different ways, for example, those generated by users and those produced by professionals, are there any significant differences in terms of quality criteria?

RQ 2. Can the UNE71362 standard be an appropriate quality framework for such evaluation?

RQ 3. How do evaluators’ prior experience with educational technologies influence the consistency and accuracy of their quality assessments?

To answer the above questions, an evaluation experiment on open educational resources (OER) was designed and implemented with students from a master’s program in teacher training in the field of computer science. The resources used in the experiment were hosted in a Spanish public digital open access repository called Procomun (https://procomun.intef.es/), which has over 111,259 registered users and over 79,508 developed learning resources (May 25th, 2026) available for use and adaptation. Users can be professionals from the publishing world or individual ones (teachers or higher education students enrolled in educational studies). Regarding the resources used in this experience, some of them were created by the students themselves (with different levels of educational experience) and by professionals. Each student evaluated their own resources (as a self-assessment task) and a peer assessment was also carried out by other students. The specific evaluation process was implemented using a teacher’s profile from the UNE 71362:2020 standard, detailing the evaluation of both the quality level of the resources and the metadata associated with them.

This article describes in detail the evaluation experimentation, the qualitative and quantitative results obtained, and the conclusions that allow us to answer the research questions. In this regard, the work is structured as follows. First, the materials used in the experience are presented followed by the description of the methodology used for the evaluation. Next, the results obtained are fully described, followed by discussion, conclusions and guidelines for future work.

2. Materials

Online Learning Object Repositories (LORs) are a type of online databases [29], specially designed to host collections of digital educational materials that use metadata standards for structuring the content, thus allowing their systematization and reuseness, improving efficiency and facilitating learning design [30]. In this sense, they allow the creation, access, and sharing of learning resources through the Internet.

2.1. Online Learning Object Repository: Procomun

One of the most important LOR in Spain is Procomun, created in 2014 at the initiative of the National Institute of Educational Technologies and Teacher Training (INTEF), supported by the Spanish Ministry of Education and Vocational Training. Procomun integrates a social network that facilitates interactions between different users that make up the Spanish educational community. It promotes the use and sharing of digital educational resources such as other prior experiences like the open learning object repository and collaborative authoring platform LeMill [31] aiming to create an open educational resource ecosystem. Although general-purpose web searching Google-like is powerful, searching mechanisms for specific purposes may rely on metadata in a federated repository.

The resources are stored and classified in Procomun with an application profile of the LOM standard, called LOM-ES [32], easily located and retrieved through an integrated faceted semantic search engine. Anyone can generate and upload their own developed educational resources by simply registering in the portal, so that a great set of educational resources can be found, with very diverse quality and learning features [33]. Among all the collections of digital educational materials stored by Procomun, the educational resources of the Gauss Project [34] are worth highlighting and have been used in this work as an example of professionally produced materials to evaluate. This project was promoted by the Spanish Ministry of Education to create a collection of high-quality digital educational materials for mathematics taught in Primary and Secondary Education [35].

2.2. Resources and Evaluators

The OER selection process was carried out by ensuring a sufficiently diverse sample to obtain different quality profiles. To this end, two groups of resources were selected. On the one hand, resources associated with the Gauss project, with high psycho-pedagogical and technical quality, developed by professionals, and comprehensive in terms of metadata tagging. The expected result of the quality assessment is that they will have higher scores than resources freely developed by individual users. On the other hand, a set of resources developed by teachers belonging to different spanish autonomous communities and registered as regular users in the Procomun repository. Even though they are developed by teachers themselves (from secondary education, high school, or vocational training), the initial expectation is that they can present a different level of quality.

The 17 OERs finally chosen are summarized in Table 1 using the “description” field available for each resource (if this field is empty, it is indicated in the table with the text “Does not exist”). The Origin column indicates the origin of the selected resources: 5 belong to the Gauss Project and 12 other items have been produced by anonymous users (8 produced by individual users and 4 by educational organizations).

As can be seen in Table 1, the 5 materials associated with the Gauss project have been uploaded and stored very carefully, with a clear distinction between the descriptions of their metadata, very specific and representative. On the contrary, from the other set of 12 resources, 5 have no description at all, 3 of them are stored with a very brief description, just a descriptive name (“Introduction to programming”, “Introduction to database design” and “Advanced use of classes”), and only 4 have a proper extensive and representative description. Interestingly, all those with a precise description have been elaborated by individual users, while 4 resources elaborated by educational organizations have either no description or a very brief description quite similar to each OER title.

Regarding the evaluators, a total of 19 people participated in the evaluation of the set of digital educational materials. These evaluators were students of the master’s degree in Teacher Training (Computer Science track). Engaging students in creating learning resources has demonstrated pedagogical benefits. In this experience, a common and scalable approach is to use a peer-review process where students are asked to assess the quality of resources authored by other peers or institutions. Since judgments about students as experts in training do not have the same value as that issued by experts, a redundancy-based method is used where the same evaluation task is assigned to several students as has already been done in other experiences [36].

Table 1 also shows the total number of evaluators who participated in its evaluation (last column). Table 2 shows the total number of resources evaluated by each evaluator, how the resource evaluations have been distributed among the evaluators, and how many times each resource has been evaluated (regarding the peer evaluation). Evaluators are presented with an anonymous identifier in the LOEP online tool [37] to preserve their identity.

From the outset, an attempt was made to achieve an equitable distribution of resources and evaluators, but since this was a voluntary activity, it was only partially achieved. On average, most peer evaluators have evaluated the same number of resources (3 or 4). Only one evaluator evaluated one resource, and one evaluator evaluated two resources. Most of the resources have been evaluated at least 3 times, and several resources have been evaluated more than four times. Specifically, and as mentioned above, the resources belonging to the Gauss Project have obtained a greater number of evaluations (5.4 times on average), compared to those resources not belonging to the project (2.83 times on average).

Research manuscripts reporting large datasets that are deposited in a publicly available database should specify where the data have been deposited and provide the relevant accession numbers. If the accession numbers have not yet been obtained at the time of submission, please state that they will be provided during review. They must be provided prior to publication.

Interventionary studies involving animals or humans, and other studies that require ethical approval, must list the authority that provided approval and the corresponding ethical approval code.

In this section, where applicable, authors are required to disclose details of how generative artificial intelligence (GenAI) has been used in this paper (e.g., to generate text, data, or graphics, or to assist in study design, data collection, analysis, or interpretation). The use of GenAI for superficial text editing (e.g., grammar, spelling, punctuation, and formatting) does not need to be declared.

3. Methods

The methodology used to develop the OER quality assessment is based on three elements:

• Assessment of the quality of LOM metadata.
• Content assessment using the UNE 71362 standard.
• Use of the online LOEP tool to collect assessment data from a web form that implements the standard teaching profile.

3.1. Quality Measurement of LOM Metadata

To perform the detailed analysis of the quality of the educational resources contained in the Procomun repository proposed in this research work, the first step was to measure the quality of the semantic tagging (metadata) accompanying the educational resources in the repository. Since the standard used in Procomun is the LOM-ES application profile [38], the quality metric LOM Metadata Quality Evaluation Model was selected [39], based on the model presented by [40] and derived from the theoretical contents of [41], five parameters are defined that are rated on a 0-10 scale to measure metadata quality. More detailed information, including the formulas used to rate each parameter, can be found in [42]. They are briefly described below (Romero-Pelaez et al., 2018; Zawacki-Richter et al., 2022).

• Completeness: Measures the amount of information included in the metadata for a comprehensive representation of the learning object, considering non-empty metadata fields and assigning different weights to them.
• Conformance: Evaluates how well the metadata meets the requirements for the learning object to be easily found, identified, and selected. It considers both the quantity and usefulness of information, including predefined and free-text metadata fields.
• Consistency: Assesses whether the metadata complies with the specifications of the LOM standard, including the correct use of attributes, fields, and vocabularies.
• Coherence: Indicates the degree to which all metadata describe the learning object consistently, based on semantic similarity between descriptive metadata fields.
• Findability: Measures how easily a learning object can be located in a repository by analysing connections between objects through shared keywords stored in metadata fields

3.2. Quality Standard UNE 71362:2020

For evaluation of the intrinsic quality of educational resources, the authors chose the set of criteria and indicators established in the UNE 71362:2020 standard, due to their active participation in the task force for its development [43]. It is a quality framework focused ex-profeso on digital educational materials and contributes to the quantitative and qualitative evaluation of digital educational resources, helping producers, users and evaluators in their objective assessment to select the best digital educational resources for their educational purposes [44]. The standard is valid for the following categories of users:

• Author/Creator: Any user creator of digital educational materials (teachers, students, teams) can use the standard as a guide for creating high-quality materials.
• Consumer/User: Users who consume digital educational materials can apply the standard select the highest quality materials that meet their educational objectives.
• Reviewer/Evaluator: The standard allows users to evaluate the quality of digital educational material. This role is the one selected for the development of this work.
• Provider/Distributor: Entities such as educational institutions, state administration or companies that create educational digital content will be able to guarantee the quality of their resources by following the standard.

The UNE 71362 Standard classifies quality indicators into three large blocks, for each of which it defines a series of criteria. Table 3 lists the top-level criteria belonging to each block. As can be seen in the table, there are a total of 15 general criteria for quality assessment under the three blocks mentioned above. However, some of these criteria are interrelated through some of their secondary indicators as will be shown lately [45].

Finally, it is important to highlight that within the definition of the UNE 71362:2020 standard, there are two specific application profiles also described, adapted to specific cases to facilitate and optimize the application of the standard. These profiles are as follows:

• Teacher’s profile: this is a profile adapted to teachers who want to produce or evaluate digital educational materials in a specific and specialized domain, but who do not have advanced knowledge of educational technologies or accessibility. In the evaluations carried out in this work, this profile of the standard will be used to facilitate its use.
• Learner’a profile: This is a profile adapted to a non-specialized user.

For the experimental development presented in this work, the profile of the teacher was finally chosen since the students who have acted as evaluators in the experimental part are enrolled in a master’s degree for Teacher’s Training, therefore a minimum level of experience, digital competences and academic level is assumed.

3.3. The LOEP Tool

The LOEP tool (Learning Objects Evaluation Platform) was developed in the framework of the doctoral thesis, offering various metrics through a web platform for the systematic evaluation of learning objects, each metric associated with a different standard [42]. The platform was chosen for the present work, offering diverse scenarios and educational contexts in a very flexible system that supported adding new learning objects and quality metrics, adaptable to add new features. Among the various quality assessment metrics offered by the platform, two of them were chosen:

• The LOM Metadata Quality Evaluation Model (LOM Metadata Quality Evaluation Model).
• The UNE 71362 Standard itself with its “Teacher” application profile.

Figure 1 shows part of the LOEP form to perform the evaluation of a specific resource with the UNE 71362:2020 (teacher’s profile). As can be seen, each evaluation criterion can be scored from 0 to 10 points, as well as not scored by selecting the “N/A” option.

The LOEP tool allows the assessor to score the resources easily. Each of the sub-elements of the 15 quality criteria has a scale from 0 to 10, and the option “Not Applicable” (N/A) can be selected if any of the checks does not make sense in the context of the resource being assessed (for instance, it has no sense to rate indicator 14.7, referring to visual or audible alerts, if no alerts, either visual or audible, appear in the resource that’s being assessed).

In addition, the LOEP tool allowed three free-text fields to be filled in, so that the evaluator could answer the following open-ended questions in an optional way:

• Is there anything else you would like to comment on your experience using this material?
• Is there any way that this material could be improved? If yes, please indicate the number of the indicator that you would improve, how and why.
• Any additional comments.

3.4. Evaluator’s Training Protocol, Ethical Considerations and Bias Mitigation

To ensure consistency in all the evaluations, a three-phase training protocol was implemented for student evaluators:

• Familiarization Phase: 2-hour session on educational quality principles and navigation in the UNE 71362 standard.
• Calibration Phase: Joint evaluation of 2 pilot resources with group discussion of criteria and resolution of interpretive discrepancies.
• Validation Phase: Individual evaluation of a reference resource with personalized feedback before starting formal evaluations.
• This protocol was designed to minimize inter-evaluator variability and ensure uniform understanding of quality criteria (Inter-rater Reliability Protocol, adapted from [48]. Moreover, several measures were implemented to mitigate potential biases:
• Anonymization: Resources were presented without information about their origin (Gauss Project, institutional or individual) during initial evaluation.
• Randomization: The presentation order of resources was randomized for each evaluator.
• Conflict declaration: Evaluators declared any prior familiarity with the resources.
• Blind evaluation: A 30% sample of resources was evaluated by evaluators who were unaware of the specific study objectives.

4. Results

This section presents the results obtained after applying the LOM metadata quality assessment model and the UNE 71362:2020 standard to the selected resources. The names of the items have been shortened to improve readability.

4.1. Quality of Metadata

As indicated above, a first evaluation is automatically performed using the LOM Metadata Quality Evaluation Model, which measures the quality of the metadata contained in each resource analyzed. As can be seen in Table 4, the evaluation of metadata quality using this model offers rather low scores for most of the criteria considered. In particular, the Findability criterion always offers a score of 0, probably because there is no information available about the connections between the resources in the Procomun repository at the time of this evaluation. However, the Consistency criterion always offers the highest score, which indicates that all the analyzed resources comply with the LOM metadata standard. The Completeness and Conformance criteria do not offer very high scores overall, illustrating the fact that in general, developers have not completed many of possible metadata offered by the LOM standard. Finally, the Coherence criterion offers quite variable scores, indicating those resources that, while presenting little metadata, present it in a coherent manner in terms of the similarity between the descriptions used. The OERs with the highest scores for each criterion are highlighted in bold. The resource with the best score, “Advanced Class Usage”, is a resource outside the Gauss Project, developed by an individual user.

4.2. Results of the Quality Assessment

This section presents the results obtained after applying the LOM metadata quality assessment model and the UNE 71362:2020 standard to the selected resources. The names of the items have been shortened to improve readability.

The best way we have found to illustrate the evaluations and be able to compare them is the spider diagram, showing the scores obtained in each of the 15 global criteria of the standard. Figure 2 to 7 show some of these charts for the 17 resources analyzed in total. By looking closely at the profile of the diagram, one can quickly infer certain qualitative and quantitative aspects about the quality according to the different criteria applied.

First, the resources with the highest quality will be those in which the diagram is shown with the most spherical shape and closest to the external threshold, which indicates a score close to 10 in all criteria (see Figure 2).

There are other cases where the quality is high overall, although there is a criterion for which the score is significantly lower, resulting in a peak towards the inside of the circumference (Figure 3).

In other cases, such as Figure 4, the quality is more uneven, finding internal (low quality) and external (high quality) peaks in the diagram. In this case, the resource presents a high quality in terms of portability, robustness and technical stability and accessibility of the audiovisual content, but a particularly poor quality in the ability to generate learning, adaptability and didactic description.

Finally, we can also find resources with poor or very poor overall quality, which is reflected in a diagram in which the line is very close to the center of the circumference in all criteria (Figure 5).

It is also interesting to analyze which UNE 71362:2020 criteria offer the lowest and highest average scores. Table 5 shows the average scores for each criterion in the standard, ordered from highest to lowest values. Clearly, the criteria belonging to the first block (didactic criteria) obtain the worst scores when applying the standard. This may be due to the greater specialization of the evaluators in this type of criteria. Besides, these evaluators have less knowledge about the technological and the accessibility criteria. This may lead to the evaluators being particularly demanding with the didactic criteria, thus making more relaxed evaluations of the technological and accessibility criteria. The exception to this is the score for criterion 2 (Quality of content), which is the highest of all. This is probably due to the fact that the developer focuses most of the effort on the quality of the content, rather than over other didactic criteria, perhaps more complex a priori, that may be forgotten.

Finally, Table 6 presents the final scores obtained according to the UNE 71362:2020 standard, ordered from the highest to the lowest value, also indicating the origin of the resources. As can be seen, the resources that come from the Gauss Project obtain higher scores, which indicates that quality has been in the center of the development of these materials. Gauss is a specific co-financed project managed directly by the INTEF, which implies greater experience and knowledge in the project developers and resources available. In the same way, although not all, those items that were developed within educational organizations also obtain quite satisfactory scores. However, we can also find resources developed by individual users that obtain high scores (e.g. “Building my computer”), although it is true that the three worst resources in terms of quality according to the UNE 71362:2020 have been developed by individual users.

5. Discussion

Authors should discuss the results and how they can be interpreted from the perspective of previous studies and of the working hypotheses. The findings and their implications should be discussed in the broadest context possible. Future research directions may also be highlighted.

Once the individual results of each digital educational material have been analyzed, our objective is to compare the quality of the three types of resources we have selected, those belonging to the Gauss Project, those produced by educational organizations and finally those developed by individual users.

Figure 6 shows a first comparison in the form of a network of spider diagrams, comparing the averages obtained for each criterion of the UNE 71362:2020 standard for the materials belonging to the resources belonging to the Gauss Project (blue line), with those developed by educational organizations (yellow line), and those developed by individual users (red line). As expected, materials developed within the Gauss Project score higher on all 15 criteria. The same occurs when comparing resources developed by educational organizations (yellow line) with resources developed by individual users. Specifically, the criteria belonging to the first block (didactic criteria) offer the greatest difference between scores, while those belonging to the technological and accessibility block show smaller differences. Once again, the fact that the evaluators use the “Teacher” profile and that they are not specialists in educational technologies or accessibility may lead them to judge more accurately the criteria of the first block, and to detect more easily the differences between the resources that belong or not to funded projects.

In some criteria, however, the differences are reduced or even reversed between those developed in the Gauss Project and the other institutional projects, specifically “Motivation”, “Reusability”, “Robustness, technical stability” and “Accessibility of the audiovisual content”. This may depend on the intrinsic objectives of each project [49] and the alignment of the criteria chosen in the UNE71362 standard with these objectives. For example, there are points of coincidence between the groups of resources for the criteria of “Accessibility of audiovisual content” and “Reusability”. In the first case, this is a quality criterion that is usually quite well known among educational content developers, while the accessibility of textual content is not taken care of in such detail when there is no specific knowledge (there is a greater difference between the two groups of resources). Regarding reusability, the very nature of the learning objects available in Procomun means that practically all developers have this feature in mind when developing their own materials.

On the other hand, Table 7 illustrates the specific scores for each block of criteria (didactic, technological and accessibility), as well as the final score according to the UNE 71362:2020, for each group of resources, according to their origin. Data confirm what was shown in the spider diagrams in Figure 6, that is in all the blocks, as well as in the final score of the standard, the resources developed within the Gauss Project obtain higher scores, although the block in which these differences are more pronounced is the didactic criteria, probably for the reason indicated above. The differences are greater between Gauss Project resources and resources developed by individual users. Finally, the differences are smaller between Gauss Project resources and those developed by other educational organizations, being especially small in blocks 2 and 3 (technological and accessibility), where the scores are practically the same. However, there is still quite a difference in the final score, mainly due to the influence of the didactic criteria block.

A final study that has been considered interesting is the one that analyzes the correlation coefficient between each of the criteria of the UNE 71362:2020 and the final score of the standard, as well as between the score of each block and the final score, and between the score of the quality of the metadata and the final score of the standard. This coefficient indicates the degree of correlation between two variables and varies between 1 (strong and direct correlation) and -1 (strong and inverse correlation). The closer its value is to 0, the less correlation there is between the two variables. Table 8 shows all the correlation coefficients calculated: one of the variables is always the final score of the UNE 71362:2020, while the other variable is indicated in the “Criterion” column. Correlations have been calculated for the 15 criteria, which are ordered from highest to lowest coefficient. The three blocks of criteria (didactic, technological and accessibility) are also ordered from highest to lowest coefficient.

Results show that there is a high correlation between practically all the criteria belonging to the UNE 71362:2020 and the final score of the standard, which makes sense since the final score is calculated from these criteria. What is interesting in this case is to see which criteria have the highest and lowest correlation. It can be clearly seen how the criteria in the third block have the highest correlation (accessibility), while those in the first block have the lowest one (didactic). The first criterion (didactic description) has a particularly low correlation, considering that it influences the final score of the standard. This tendency is also observed when looking at the correlations by blocks (last rows of the table). The explanation for this behavior may come from the order in which the evaluators complete the evaluation according to the standard: at the beginning of the evaluation, the evaluator may not be so clear about the overall level of quality of the resource he/she is scoring, and therefore the first criteria may vary more from the final score. However, as the evaluation progresses, their scores converge towards the final score that the resource will obtain, which is reflected in the correlations of the technological block and especially the accessibility block. A possible solution to alleviate this effect would be for the evaluation tool to alter the order in which each criterion is evaluated, so that the order would be randomized and each time a resource is evaluated it would be done in a different way.

6. Conclusions

This section is not mandatory but can be added to the manuscript if the discussion is unusually long or complex.

The main conclusions regarding the research questions posed before conducting this experimental study are the following:

• RQ 1. The origin of an educational resource, although it does not represent an excluding aspect when determining its relevance for a specific educational need, is a very important source of information in this decision. It has been found that those resources that come from educational organizations, specialized in the development of digital educational materials, generally have a higher quality level than other resources developed by individual users. Therefore, although it is possible to find high quality resources developed by such users, the fact that a learning object is developed by an educational organization or within a specific project provides some assurance as to the quality of that resource. At least, clearly, regarding the quality of the metadata that helps the recommender systems to find the resources that better fits teacher’s needs and learning objectives. The results of the study clearly show that it is possible to measure and compare the quality of educational resources created under different conditions and by professionals with varying levels of expertise. Structured assessments using models like the LOM Metadata Quality Evaluation Model and the UNE 71362 Standard enabled meaningful comparisons between resources. Notably, materials from the Gauss Project consistently outperformed those developed by individuals, reinforcing the value of institutional experience and coordination in the production of high-quality learning objects.
• RQ 2. The study of the correlation coefficients has confirmed the importance of all the criteria considered in the final score obtained with the UNE 71362 Standard, in such a way that no criteria considered less important or of less consideration have been found. Moreover, the standard itself has proven to be a robust and comprehensive framework for this kind of evaluation. Its structure, divided into didactic, technological, and accessibility blocks, supports both detailed assessments and the identification of strengths and weaknesses across diverse resources. The study also demonstrated high internal consistency among the criteria, especially in the technological and accessibility dimensions, thus validating the appropriateness of UNE 71362 for educational quality evaluation [50].
• RQ 3. Finally, it is worth noting that the evaluators’ prior experience, primarily in teaching rather than educational technology, influenced the scoring process. Evaluators tended to assess didactic aspects with greater precision and variability, whereas technological and accessibility criteria were often rated more uniformly, perhaps due to a lack of specialized knowledge. The analysis of correlation coefficients showed that early-evaluated criteria (particularly didactic ones) had weaker correlations with the final score, suggesting some initial uncertainty. As the evaluation progressed, later criteria showed higher alignment, reflecting a growing confidence in scoring. This finding suggests the need to randomize evaluation order in future applications of the standard, to enhance scoring reliability across all dimensions.

Another set of conclusions regarding the use of standardized application profiles and metadata in open repositories are that the study carried out on metadata quality using the LOM Metadata Quality Evaluation Model did not offer particularly interesting results, since the scores obtained by the resources analyzed varied greatly, and there were even criteria in the model for which the score was always 0. Also, the digital educational materials available in repositories such as Procomun present a high variability within their quality, both globally and in relation to the various criteria analyzed with the UNE 71362 Standard (didactic, technological and accessibility). This requires a great capacity for critical analysis, as well as an important ability to adapt to the available resources, in order to be able to select those learning objects that best suit teaching needs. Finally, the UNE 71362 Standard can be an indispensable tool for the development and analysis of digital educational materials, in terms of its great usefulness in the process of evaluating the quality of these resources, considering aspects as diverse, but as necessary as educational, technological and accessibility.

For future work we intend to extend the analysis carried out in the present work with broader systematic study. At present, we already hold a larger number of learning objects, with specific cataloging (e.g., attending to various fields of study or subjects), with a larger number of evaluators and reviewers developing evaluations in new five consecutive years 2020-2025, to analyze more exhaustively the usefulness of the standard. We want to use AI ML algorithms to make resources’ clusterization based on different quality criteria addressing user’s needs, i.e. accessibility. We also want to further study the quality of resources developed by students that might include an intermediate moderation process to separate high-quality resources from low-quality ones, or to map certain characteristics in clusters and better explore how learner sourcing can be used for evaluating the quality of learning resources [51]. Also, some improvements in the application of the standard itself must be tested to reduce the effect of the order in which the criteria of the standard are analyzed. In this way, the criteria could be analyzed in a random order that favors a more stable correlation between the criteria scores and the final score of the standard between criteria. And finally, we would like to try the use of other quality assessment measures for comparative studies.