The Impact of COVID-19 on Learning Loss in Elementary School Students: A Comparative Study of Academic Performance Across Grades

1. Introduction

Global pandemic outbreaks in the 21st century have had far-reaching consequences, affecting numerous regions worldwide [1,2,3]. In response to the rapid spread of the COVID-19 virus, drastic measures were implemented worldwide, chiefly among them school closures [4,5]. This sudden disruption posed a significant challenge in meeting the educational needs of students [6] and severely impacted students’ mental [7,8] and physical health and learning [9,10].

Early studies examining students’ academic performance during the COVID-19 pandemic reported a decline in learning outcomes, commonly referred to as learning loss [11,12,13]. For example, Engzell et al., (2021) sought to investigate the extent to which school closures in the 2020 impacted primary school performance (math, spelling and reading) in a wide sample of Dutch students (aged 8 to 11 years). The authors found a learning loss of about 3 percentile points. The authors posited the effect was equivalent to one-fifth of a school year.

Notably, the COVID-19 pandemic has been associated with a substantial increase in learning losses [15,16,17,18]. The suspension of face-to-face education in countries affected by the pandemic aimed at mitigating the spread of COVID-19; however, it has resulted in short-term learning losses and potential long-term consequences for students’ educational journeys [19]. The magnitude of the impact is staggering, with an estimated 1.6 billion students worldwide affected by school closures (UNESCO, 2020). Educational researchers have been actively investigating the deficits, disruptions, and learning losses experienced by students [9,14,20,21], underscoring the urgent need for interventions to mitigate learning loss during distance learning [16,17,18].

Extensive global studies conducted during the COVID-19 pandemic consistently highlighted the prevalence of learning loss among students [22]. Learning loss is a pervasive issue affecting students across all levels of education [23]. Infants and toddlers exposed to previous pandemics and epidemics have demonstrated enduring adverse effects, such as impaired cognitive development and reduced educational attainment [24,25]. Cognitive development is intricately interconnected with social, emotional, and psychomotor development, forming the basis for knowledge acquisition and sustaining a holistic learning process.

The early years of childhood are particularly critical, as failure to acquire fundamental skills during this period can lead to a decline in developmental levels. Even when schools eventually reopen, the effects of learning loss may persist, negatively impacting children’s educational attainment and potentially affecting future generations [26].

A recent systematic review [10] compiled the findings of 24 studies that analyzed potential learning loss during various phases of lockdown, highlighting several negative consequences of lockdown and the transition to virtual learning schemes on children’s academic performance in various learning areas. It is worth noting that 35% of the included studies did not report any learning loss; therefore, the negative impact on academic performance during lockdown should be further examined to understand the conditions that are most conducive to learning loss.

There is evidence that learning loss, associated with COVID-19 restrictive measures does not show a uniform pattern across all students. For example, both Maldonado and De Witte (2022) and Kuhfeld et al. (2020) found learning losses in some subjects but insignificant impacts in others. On the other hand, Tomasik et al. (2021) observed an impact on elementary school students, but they did not find any impact on secondary school students. This is consistent with the literature indicating that students in the early grades may be more vulnerable than secondary school students due to their inability to seek learning independently, owing to differences in developmental and cognitive abilities [29,30].

Other discriminating factors are certainly related to the characteristics of the students’ family backgrounds. Engzell et al. (2021) determined that losses were up to 60% greater among students from non-educated families.

On the other hand, some specificity in terms of learning domains has been observed. In a study conducted in the United States, more than 1.5 million students exhibited specifically worse performance in mathematics and reading scores compared to the previous academic year [31].

Therefore, several contributing factors could be identified, including: (i) socioeconomic status (type of household and family income); (ii) access to technology, (iii) learning domain; academic year affected by remote learning.

The literature representing the impact of COVID-19 on students’ learning progress is limited in terms of the number of available studies, geographical regions analyzed, and the number of participating students. Given the innovative nature of this topic, it is understandable that educational researchers have only just begun to analyze the learning losses that students have experienced.

The present study aims at shedding light on the variables that make students more susceptible to learning loss, with a particular attention for academic domains affected by remote learning.

We analyzed the differences in learning loss among elementary school children based on their grade level (from II to V) during COVID-19, assuming that the effects of the pandemic on learning may vary depending on the developmental and educational stage affected. Specifically, in the present study we advanced the hypothesis that distance learning had a higher impact when it occurred in the early stage of the learning process, namely at the I grade level of elementary school. Accordingly, the study compares the performance results and outcomes in different learning areas, including spelling, reading comprehension, reading speed and accuracy, as well as mathematical skills.

2. Materials and Methods

2.1. Participants

The study was carried out at four different primary schools in southern Italy during the last semester of the academic year 2022/2023 and enrolled 332 students. Participants were recruited from the second (N= 139), the third (N=56), the fourth (N= 103) and the fifth (N= 34) grades of primary schools. Children attending the first-grade school were excluded due to: 1) the exiguous number of children (N= 11); 2) the test provided for the assessment of learning skills in this sample (PRCR-2, (Cornoldi et al., 2009) differs in its structure, hampering the possibility to compare the performance in this test with those of the other ones.

For the present study, it is worthy to note that second-grade students did not experience remote learning. Conversely, the third-grade primary school children were those who began their schooling during the first wave of COVID-19 and, as a result, they commenced their education journey through distance learning. On the other hand, in children of the fourth- and fifth-grade school learning of academic skills occurs in the traditional school context.

Children were all monolingual Italian native speakers, and they were included in the study after parents were informed about the activities and provided the informed consent. The study was approved by the ethics committee of the Department of Humanistic Studies, University of Naples “Federico II”, Naples, Italy (N° prot. 3/2019) and in agreement with the Declaration of Helsinki (2013) and by the school authorities.

2.2. Materials

The assessment of learning skills (i.e., mathematical, spelling, reading and comprehension skills) was carried out through different psychometric tests (below a brief description has been provided for each of them). Tests were consistent with the grade level of each child.

Reading and comprehension. In order to assess the reading and comprehension skills, participants were provided the “MT Reading test” [33]. The MT battery is the most psychometrically valid Italian instrument to measure the main relevant outcomes about reading skill: speed and accuracy. For the assessment of reading skills, participants were provided a passage which they were instructed to read aloud as fast and accurately as possible within a 4-min time limit. The story was appropriate to the grade level of each participant regarding cognitive and linguistic demands with an increasing number of syllables. Children’s reading skills were assessed through two main parameters:

-

Speed: is computed by dividing the number of syllables in the passage and the time spent (in seconds) reading them. In the literature there is a particular consent in considering this measure as a more sensible index of reading skills as compared to the accuracy, especially in transparent/regular orthographies as the Italian (Consensus conference, Specific Learning Deficits 2010; Landerl et al., 1997).

-

Accuracy: is computed by counting the number of errors (i.e., misreaded word) adjusted for the amount of text read.

Raw scores (speed and accuracy) were compared with the normative data and assigned to a performance category (see below):

-

CAC: Completely Accomplished Criteria (values above the 70° percentile)

-

SP: Sufficient Performance (values between the 15° and the 70° percentile)

-

AR: Attention Required (values between the 5° and the 15° percentile)

-

IIR: Immediate Intervention Required (values below the 5° percentile)

The comprehension task consisted of two stories participants must read (silently or loudly) and answer to 10/12 multiple-answer questions. The story was appropriate to the grade level of each participant. The task was not time constraint: the assessment of the comprehension skills relies uniquely on the accuracy (number of correct responses).

Raw scores about the accuracy were compared with the normative data and assigned to a performance category. Performance categories were identical to those of the reading test.

Writing. In order to assess the writing skills participants were provided the spelling-to-dictation task of the test “BVSCO-2” [36]. In this task, the examinator read aloud a text at students’ pace. Participants were instructed to spell the text as fast and accurately as possible. In this test, stories were adapted to the grade level and the complexity modulated regarding the syntax and the frequency of the words. Importantly, the structure of the test was adapted for testing a single or group of students (respectively individual or collective administration).

The main score in this task is the accuracy (the total number of errors made by the student). Errors can be classified as: 1) Phonological Errors (PE), namely errors which are not respected in the grapheme-phoneme relation (omission, insertion, transposition, substitution); 2) Non-Phonological Errors (NPE), namely a wrong graphic (visual) representation of the words without errors in the relation between grapheme and phoneme (e.g., “squola” instead of “scuola”- “school”). In other words, the miswritten word is phonologically plausible, but orthographically wrong; 3) Accent and Double Letters Errors (ADLE). The former refers to errors in placing the accent (e.g., “mangio”- “I eat” instead of “mangiò”- “He ate”); the latter refers to the omission or the addition of a double consonant (e.g., “pala”- “shovel” instead of “palla”- “ball”).

Accuracy (total number of errors) or each type of error (i.e., PE, NPE, ADLE) were compared with the normative data and assigned to a performance category. Performance categories were identical to those of the previous two tests (CAC, SP, AR, IIR).

Mathematical skills. In order to assess the mathematical skills, students were provided with the collective tasks of the AC-MT battery [37]. This battery consists of four subtests:

-

Written arithmetic calculations subtest: participants performed two calculations for each of the four basic number operations (addition, subtraction, division and multiplication), based on two numbers. This subtest aims to assess the student’s ability to apply the normal steps underlying the calculation and whether they are proceduralized

-

Quantity discrimination subtest: participants were provided six pairs of numbers, and they had to circle, for each pair, the highest one. This subtest assesses the semantic knowledge about numbers and requires the right reading of the numbers (lexical level)

-

Number transformation subtest: participants were provided a set of six digits, each with their own positional category. Students had to write the correct structure of the number (e.g., 2 tens and 5 ones= 25). The task aims to assess the ability to elaborate the syntactic structure of the number which regulates the relationship among the digits that compose them

-

Number ranking subtest: the participants order number sequences from the smaller to the higher and vice versa. This subtest assesses the semantic representation of the numbers

For each subtest, the accuracy is computed in terms of the number of right answers without constraints in terms of time. The subtests allow the assessment of 2 main mathematical skills: 1) written arithmetic calculations; 2) knowledge about numbers. The latter is computed by summing the participant’s accuracy in the quantity discrimination, number transformation and number ranking subtests.

The written arithmetic calculation and the knowledge about the numbers scores were compared with the normative data and assigned to a performance category (see below):

-

OT: Optimal performance (values above the 70° percentile)

-

S: Sufficient performance (values between the 15° and the 70° percentile)

-

AR: Attention required (values between the 5° and the 15° percentile)

-

IIR: Immediate Intervention Required (values below the 5° percentile)

2.3. Procedure

Children were tested both collectively and individually in a quiet room in their school. The collective and individual assessments were provided in two consecutive days. The collective assessment included the comprehension task, the spelling-to-dictation task and the AC-MT subtests. The individual assessment included the MT Reading test

3. Statistical analysis

Firstly, a chi-square analysis was carried out in order to test the equal distribution of males and females in each grade. Then, a set of univariate ANOVAs were carried out in order to highlight differences between grades (second, third, fourth and fifth) in terms of learning skills (reading, comprehension, written arithmetic calculation and knowledge about the numbers). For each ANOVA, Gender was conceived as a factor in order to test its main effect and the potential interaction Grade x Gender. For the significant results, post-hoc comparisons were Bonferroni corrected. Secondly, in order to test whether the performance categories for each learning skill were equally distributed in each class a set of Chi-square tests was carried out with performance categories (e.g., for the reading task: CAC, SP, AR, IIR) and grades (from II to V) as variables. Multiple pairwise comparison tests were performed to determine which pairwise comparisons among the individual joint conditions were significant. Multiple comparisons were carried out according to Cox and Key (1993): “the individual cells must be recognized as being independent with independent chi-square cell contributions. Thus, the absolute value of the difference between the two cells contributions, Δ(X²), will be distributed as Chi-square with one degree of freedom. Conceptually this is comparable to applying Chi-square to a (1xk) frequency table where k=2. To determine whether the difference between the two Chi-square cell values was significant, the difference was compared with Δ(X²)*=3.841, the theoretical distribution value for Ⲁ= 0.05 and one degree of freedom.”

All the analyses were carried out with “Statistical Package for Social Sciences” (IBM Corp. Released 2021. IBM SPSS Statistics for Windows, Version 28.0. Armonk, NY: IBM Corp).

4. Results

4.1. Gender Distribution Among Grades

The first Chi-square analysis highlighted a significantly different distribution of males and females in each grade (X²(3)= 15.55, p< .001; Table 1).

4.2. Continuous Performance in Each Task Among Grades

Spelling-to-dictation task. The ANOVA reported the 1) significant main effect of gender (F_(1,310)= 9.28, p< .001, n²_p= .03) with females performed better (i.e., they were more accurate) as compared to males; 2) significant main effect of grade (F_(3,310)= 11.23, p< .001, n²_p= .10). The analysis failed to report a significant interaction effect grade x gender (F_(3,310)= .88, p= .45, n²_p< .001). In fact, females committed less errors as compared to males, but both genders presented a similar trend of the data. The post hoc comparisons highlighted that children of the third grade made more errors (mean of errors= 16.21; sd= 11.35) as compared to those of the second (mean of errors = 8.75, sd= 6.69; p< .001), fourth (mean of errors= 7.94, sd= 7.99, p< .001) and fifth (means of errors= 6.79, sd= 6.22, p< .001; Figure 1). No further comparisons reached the significance. Results suggested that third-grade children had lower writing skills as compared to those of the second, fourth and fifth grade.

Comprehension task. The ANOVA reported only the main effect of grade (F_(3,310)= 7.17, p< .001, n²_p = .06). The analysis did not highlighted neither a main effect of the gender (F_(1,310)= 3.41, p= .07, n²_p = .01) nor a grade x gender interaction (F_(3,310)= 1.08, p= .36, n²_p = .01). Post-hoc comparisons highlighted that children of the third grade were less accurate (mean of accuracy= 5.29, sd= 2.11) as compared to those of the second (accuracy mean= 6.25, sd= 2.10, p= .04), fourth (accuracy mean= 7.03, sd= 2.39, p< .001) and fifth grade (accuracy mean= 6.88, sd= 2.36, p= .006; Figure 2). Comparisons have also highlighted a low accuracy in the comprehension task in children of the second grade as compared to those of the fourth (p=.04). Consistently with the previous results, the analyses carried out on the comprehension task revealed a greater difficulty among third-grade children in deducing or inferring information from written stories.

Written arithmetic calculation. The ANOVA reported: 1) the main effect of gender (F_(1,310)= 8.51, p< .001, n²_p = .03); 2) the main effect of grade (F_(3,310)= 119.11, p< .001, n²_p = .54); 3) the interaction effect of grade x gender (F_(3,310)= 2.72, p= .04, n²_p = .03). Post-hoc comparisons highlighted heterogeneous results with children of the third grade which were less accurate (accuracy mean= 5.32, sd= 1.99) as compared with children of the fourth grade (mean off accuracy= 6.50, sd= 1.57, p< .001). On the other hand, the accuracy of children of the third grade were higher than those of the children of the second class (accuracy mean= 2.86, sd= 1.21, p< .001). No further differences in terms of accuracy were highlighted between third and fifth grade (accuracy mean= 5.91, sd= 1.78, p= .47; Figure 3). Finally, children of the second grade were less accurate as compared with those of the other grades (p values < .001).

Knowledge about numbers. The ANOVA reported only the main effect of grade (F_(3,310)= 3.72, p= .01, n²_p = .03). The analysis did not highlight neither a main effect of the gender (F_(1,310)= 3.06, p= .08, n²_p < .001) nor a grade x gender interaction (F_(3,310)= 2.55, p= .06, n²_p = .02). Post-hoc analyses reported one significant comparison, between children of the third and the fourth grade (p= .03). Specifically, students of the third class were more accurate (third grade: accuracy mean = 19.10, sd= 3.86; fourth grade: accuracy mean= 17.33, sd= 3.05; Figure 4).

Reading speed. The ANOVA reported only the main effect of grade (F_(3,310)= 68.83, p< .001, n²_p = .40). The analysis did not highlight neither a main effect of the gender (F_(1,310)= 1.02, p= .31, n²_p < .001) nor a grade x gender interaction (F_(3,310)= .45, p= .72, n²_p< .001). Post-hoc comparisons reported a linear trend about the increase of reading skills as reported in Figure 5. Thus, the syllables per second linearly increased as a function of the class (second: mean of sill./sec= 2.10, sd=.74 < third: mean of sill./sec= 2.69, sd= .81 < fourth: mean of sill./sec= 3.57, sd= .86 < fifth: mean of sill./sec= 3.67,sd=.89) All the pairwise comparisons were significant (p values <.001).

4.2. Distribution of the Performance Categories Among Grades

The Chi-square analysis reported a different distribution of the performance categories among grades for the spelling-to-dictation task (X²(9)= 43.69, p< .001), written arithmetic calculation (X² (9)= 25.64, p= .01) and knowledge about numbers (X² (9)= 64.48, p< .001). Results about the comprehension task just approached the significance (X² (9)= 16.31, p= .06). Chi-square analyses, failed to report any statistical significance in reading tasks in terms of both speed and accuracy (speed: X² (9)= 13.47, p= 0.14; accuracy: X² (9)= 11.07, p= .27).

Spelling-to-dictation task. Table 2 shows, for the spelling task, which classification of proportions has a large disparity between the actual observed frequencies (oij) and the theoretically expected frequencies (eij).

Specifically, the performance category IIR for the third grade has the largest chi-square contribution, with a higher observed frequency of this category performance as compared to the expected frequency. The IIR category for the third grade contributes 39% of the total chi-square value (16.93/43.96). Post-hoc analyses highlighted that the observed frequency of the IIR performance category in children of the third grade was significantly higher as compared to the expected frequency. This difference was significantly higher as compared to the same difference in both the other categories (CAC, SP, AR; all p values < .05) and grades (second, fourth and fifth; all p values < .05). In other words, the IIR performance category was more likely to occur in children of the third grade and this distribution was significantly different from the chance.

Comprehension task.Table 3 shows, for the comprehension task, which classification of proportions has a large disparity between the actual observed frequencies and the theoretically expected frequencies.

Specifically, the performance category IIR for the third grade has the largest chi-square contribution, with a higher observed frequency of this category performance as compared to the expected frequency. The IIR category for the third grade contributes 41% of the total chi-square value (6.43/16.31). Post-hoc analyses highlighted that the observed frequency of the IIR performance category in children of the third grade was significantly higher as compared to the expected frequency. This difference was significantly higher as compared to the same difference in both the other categories (CAC, SP, AR; all p values < .05) and grades (second, fourth and fifth; all p values < .05). In other words, the IIR performance category was more likely to occur in children of the third grade and this distribution was significantly different from the chance.

Written arithmetic calculation. Multiple comparisons for the performance categories and grades in terms of written arithmetic calculation did not report significant comparisons. In other words, none of the relevant chi-square cell differences was at least the value of the critical difference to reach the significance (see Table 4 for the disparity between the actual observed and the theoretically expected frequencies).

Knowledge about the numbers. For the knowledge about the numbers, the classification of proportions between the actual observed frequencies and the theoretically expected frequencies is reported in Table 5.

Results revealed that the observed frequency of the OT performance category for children attending the second and third grades was significantly higher than the expected frequencies. Furthermore, the performance category OT for the fourth grade has the largest Chi-square contribution, with a lower observed frequency of this category performance as compared to the expected frequency. The OT category for the fourth grade contributes 24% of the total chi-square value (15.28/64.48). Post-hoc analyses highlighted that the observed frequency of the OT performance category in children of the fourth grade was significantly lower as compared to the expected frequency. This difference was significantly higher as compared to the same difference in both the other categories (CAC, SP, AR; all p values < .05) and grades (second, fourth and fifth; all p values < .05). In other words, the OT performance category was less likely to occur in children of the fourth grade and this distribution was significantly different from the chance. However, the children of the fourth grade were also more likely to have a sufficient performance as compared to the chance.