An Analysis of the Online Final Examination Items for Ninth-graders in the Mathematics Course Using the Rasch Measurement Model

Indonesia's National Examination has been abolished since 2020. Hence, the Indonesian junior high schools make their final examination items for the 9th-grade, and from the results, the school determines students' graduation. Therefore, this study is aimed to evaluate the Online Final Examination items in one of the public junior high schools in Bandung, Indonesia. The sample was 234 students in grade 9 using their mathematics examination tests, comprising 20 multiple-choice items with 4 options, while the data processing used Winsteps software with the Rasch modeling technique. Subsequently, the Rasch model results showed an acceptable person separation statistic of 1.54 and sufficient person reliability at 0.74. The item separation statistics were in a good category at 4.59, while the reliability at 0.95 was excellent. Although four online final examination items were in the fit category, 16 were good and capable of dividing students according to their abilities. The result also provided very detailed data about the quality of the items and the ability of each grade 9 student. Since each test item is included in the fit category, this study contributes information on teachers' preparing and evaluating the Online Final Examination.


Introduction
Indonesia is one of the countries in Southeast Asia with a high number of Covid-19 cases, and as of May 18, 2020, there were 1.7 million cases and 48,305 deaths. This has caused many challenges and problems in education, as students have not attended school since May 2020, and teaching and learning activities are taking place online [1]. Several problems arise when implementing online learning, as the teachers do not have good pedagogical and technological knowledge and technological knowledge to teach on these platforms [2]. Also, students are not ready to take online lessons [1].
A national examination is a measuring tool for sustainable national growth in the education sector, which aims to see if students can think critically, be innovative, and provide solutions for the world. Meanwhile, National examinations are usually held in many countries to determine students' graduation and future when entering high school or university [3], [4]. Similar to other countries, Indonesia student's examination is held every year to determine the students' graduation and analyze their quality in each region. Due to Covid-19, the Indonesian Ministry of Education decided to abolish the national examinations from elementary to high school level. Therefore, each school was allowed to hold a final examination and determine the students' graduation independently. This development can be beneficial depending on the angle from which experts and teachers analyzed it. Consequently, the government cannot determine the quality of education in each region anymore because every school has different graduation examinations and standards. The worst aspect is that schools may assess a child's graduation subjectively and pass a child with a poor achievement. Since this problem is yet to be solved, ana-lyzing and researching the quality of final examination items for grade 9 junior high school students is an important to be performed.
The challenges of final graduation examination are even greater because they are held online, using Google Classroom, Zoom, Google Meeting, and other applications. Furthermore, the examination is in the multiple-choice and description form, inputted into the Google platform to allow students to work online. In the end, the teacher does not have to check the multiple-choice answers, as students can immediately see the final results. However, Indonesia does not have a stable internet network and an effective platform to ensure students do not cheat during the examination. They may use calculators while solving the mathematics questions or cellphones to ask friends for answers, and even people around can help them get perfect scores. Hence, the analysis of these results is very important to evaluate the students' abilities and ensure cheating does not occur.
During the Covid-19 pandemic, Indonesia implemented a 75% work-from-home rule, which prevented proper communication and meetings for producing and evaluating the final examination questions. The mathematics teachers experienced difficulties asking for help from other teachers to conduct the construct validation analysis while making their test. Furthermore, this was the first time the school was to determine the junior high school students' graduation based on the final examination scores because of the Covid-19 pandemic. Therefore, the school lacked experience making adequate items to determine the junior high school student's graduation.
This research focuses on analyzing the online mathematics examination to replace the national examinations in determining student graduation. The novelty is the analysis of two important points, the items' quality and students' abilities using Rasch Analysis with WINSTEPS software. Furthermore the research was performed at a public school in Bandung, Indonesia, using the Rasch measurement model to evaluate the difficulty, reliability, and quality of items, alongside analyze the probability curve and students' abilities. Consequently, the results are expected to contribute to education as an evaluation tool when the national examination is abolished. They can also serve as a reminder for teachers when preparing items for final examinations, such as midterms, semester finals, or the online mathematics finals.
By processing the final examination data using the Rasch measurement model, the results can answer the following questions: 1. How are the validity values in the online mathematics final examination with aspect of person fit, item fit, and unidimensionalty? 2. How are the reliability values of online mathematics final examination items based on i) KR-20 Cronbach's coefficient alpha; and ii) Person and item reliability indices?
3. How are the students' ability levels and item difficulty levels of online mathematics final examination items? 4. Are there any indications of cheating when students work on the online mathematics final examination?

National Examination
Every year, many countries use the national examination to determine the graduation of grade 9 students in junior high schools. These countries have their designations for this examination, for instance, it is called Scholastic Aptitude Tests (SAT) or American College Testing (ACT) in America [5]. In China, it is known as the Chinese National College Entrance Examination (CEE; gaokao) [3] and in Indonesia, it is called the Ujian Nasional (UN) [6].
The national examinations are conducted once yearly around May and are important for most students to determine their future and further education. In Indonesia, students who fail can retake it in the following year. The national examination is used to evaluate students, schools, and provinces to help the government know the quality of education. From the results, the government can focus on areas that require special attention and improvement to ensure education in the country is fair and equitable.
Several studies on national examinations were previously performed, such as Virginia LoCastro, compared the national examination on the English subject in Japan using sociocultural analysis [7]. In another study, Bai et al. (2014) examined the National College Entrance Examination and analyzed students' abilities at two leading universities in China, with implications for university admission policies and practices. This study suggested that admissions should consider the National College Entrance Examination results, alongside the high school achievement over the past three years, count the prizes and awards received in high school, and consider homeroom teacher recommendations. Although the National College Entrance Examination score is just a number, this study perceives that many factors should be reconsidered, for example, general and language skills and attitudes. Sanz and Pavón (2015) explained that the national foreign language examination system in Spain uses an online system. The illustration of the platform, management tools, security, and user interaction are well explained, and the study concluded that the use of online systems for national examinations existed before the coronavirus pandemic. Meanwhile, no specific research discussing the item analysis of the national examination using the online system was discovered. Therefore, this research contributes to discussing the national examination item analysis using an online system during Covid-19. The analysis can also be used as a reference source, benchmark, and comparison material to assess the quality of national examination items in Indonesia during Covid-19. Consequently, a public school was used as the research sample.

The Rasch Model
The Rasch model is a mathematical model and measurement tool [10]. It focuses on the approach to construct measurement in the social sciences field, which usually uses the more familiar WINSTEPS software and can show the response structure of the assessment [11]- [13]. Moreover, it can enter the class, student work, questionnaire items, or final examinations and provide specific measurement data [14], [15]. Although the Rasch model shows the measurement criteria and test fitting responses, the analysis of insufficiently fit data should be continued by carefully evaluating the reliability and validity [16]. The model is also a method for analyzing examination results to investigate the correlation between item difficulty and students' abilities [17], [18].
Meanwhile, Karlin and Karlin (2018) stated the importance of validating the test items. They affirmed that the validity test can determine the accuracy of test items in measuring students' abilities, for instance, in mathematics. Validation can also ascertain whether the difficulty level of the test items is based on the student's abilities. Can the test items separate the students into three or four levels based on their abilities? Are the test items self-explanatory? Do test items confuse students? The research concluded that ensuring test items given are appropriate and good in measuring students' abilities is important.
Subsequently, the number of items with high, medium, and low difficulty should be planned properly, hence the student's ability level can be measured specifically. The test items that are too difficult will achieve mostly incorrect answers, while students will correctly answer too easy questions. These conditions cannot measure which students have low, medium, or very good abilities. Therefore, test items should be created with 50% medium items, 30% easy, and 20 percent with high difficulty.
The item analysis using the Rasch model has been performed in many previous studies in various fields such as medicine, pharmacy, physics, chemistry, social science, and so on to validate and evaluate the quality of the questionnaire items [20]- [23]. However, only a few studies have used the Rasch model to analyze items in the mathematics field, while none that analyzed final school examination items using the model was discovered.
A study on the use of the Rasch model to validate and analyze items in 2011 was found. Mohsen Tavakol and Reg Dennick used the model to improve assessment in medical education by analyzing 355 medical students using 24 final clinical knowledge items [24]. The analysis results concluded that Rasch analysis supports the diagnosis of quality, alongside provides feedback for each test item and students' ability to inform lecturers on methods to improve the quality of examination items. In 2011 again, Abdullah et al. (2012) analyzed examination items on Microelectronic material using the Rasch model and concluded on its ability to analyze students' abilities when answering examination items.
Also, Jennings and team in 2016, analyzed multiple-choice examination items conducted on 101 students at the Arizona University and found that they were not very difficult, hence the classification of students' abilities was not measured properly [25]. They advised that the item-making should focus on measuring the quality performance rather than just looking at student grades and rankings. Furthermore, research by Nopiah et al.
(2011) on Engineering Mathematics Courses on the code paper KKKQ2114 used the Rasch model to validate and analyze items' quality and found that 10% were in the misfit category Therefore, the items should be corrected or removed, as the bad items cannot measure students' abilities according to the teacher's wishes.
Based on these previous studies, this research concluded that the Rasch model is very important and has many benefits for evaluating and improving the quality of items as well as measuring and analyzing students' abilities. Furthermore, the results from the model can be used as evaluation material to modify the teaching method or the item formed to fit the guidelines for good test items.

Research Methodology
This study used the Rasch model to analyze test items from the Online Final Examination data. The measurement model can show summary statistics such as mean, Standard Deviation, maximum value, amount of data, etc., and classify students' abilities as low, medium, or high by evaluating separation and logit. It can also determine whether the items for obtaining a measurement were met, the level of difficulty and whether students are careful while answering or making guesses.

Research Population
The research data comprised mathematics examination items for the 9th-grade graduation class held in May 2021. Meanwhile, the sample was the 9th-grade students in one of the public junior high schools in Bandung, Indonesia. The purposive sampling technique was used in the selection, and the general information about the study sample is shown in Table 1. This public school was chosen because it has good national accreditation and quality. The final examination items were made by the 9th-grade mathematics teachers and validated by 2 curriculum experts, then signed by the principal and used to determine the students' graduation in mathematics. Subsequently, the research sample was 234 students from 6 classes (9A, 9B, 9C, 9D, 9E, 9F), comprising 155 female and 79 male students, which made up 66.24% and 33.76%, respectively.

Data collection
A teacher with the initials IM made 20 graduation examination items for mathematics in the multiple-choice form, consisting of 6 items on numbers, algebra, and geometry each, and 2 items on probability and statistics each. Table 2 shows more specific information. The Final Examination Items were designed for use with ordinal data that are scored in two categories (0 or 1) namely dichotomous data. Each test item was assigned a 5-point score, and the student's total scores will range from 5 to 100. Based on the results, this study used the Rasch measurement model to classify the students' achievement when working on the Online Final Examination items.

Process
This research analyzed the teaching and learning process by interviewing 2 mathematics teachers at the school. According to the interview results, these activities were still occurring online, and the school used Google Meeting and Google Classroom, and occasionally Zoom to interact with students. Meanwhile, homework and exercises were given through Google Classroom. From the interview results, the teachers stated that no math software or videos were used for mathematics lessons during the Covid-19 pandemic.
The 9th-grade final examination information was obtained through Google Meeting and lasted for 90 minutes, with the students using laptops to work on the items and a handphone placed behind them for supervision. However, not all the students have these gadgets, and some only used handphones to take the final examination. Another problem was the slow and disconnected internet, which did not allow the teacher to monitor the students properly. Finally, the instructions were given to students to log in at 9 P.M and finish the test items at 10.30 P.M.

Statistical analysis
The Online Final Examination Items was analyzed using WINSTEPS version 3.73. This research used the Person fit statistics to evaluate the consistency of the students' answers in the online examination. An inconsistent Person fit is caused by cheating or guessing while answering the items. Table 3 shows the criteria for assessing the items and the students' ability levels. A too-small MNSQ value (< 0.5) means that students with a low ability answered correctly a 'difficult' item but incorrectly for the rest of items A too-small Outfit ZSTD value (<-2.0) indicates that a student with a low ability answered correctly a 'difficult' item but incorrectly for the rest of items.
Meanwhile, the reliability of the Online Final Examination items can be analyzed using the Rasch model based on the KR-20 analysis, item and person reliability and item and person separation. The references table for the interpretation of data reliability can be seen in Table 4.  [28]. Rasch analysis is able to determine the students' ability levels and difficulty of online final mathematics examination items. Wright map was used to visualize students' ability and difficulty levels on the same logit scale. On the wright map, item difficulty is on the right and students' ability levels are on the left. The easiest questions are at the bottom and the hardest questions are at the top of the wright map and for low ability students are at the bottom and high students are at the top.
Lastly, Guttman Scalogram was used to assess students' responses to each item. The careless, guessing and cheating students when students work on the online mathematics final examination can be analysed from the guttman scalogram.

Unidimensionality
Unidimensionality is frequently analyzed using the Principal Component Analysis of Rasch Residual (PCAR). The analysis of unidimensionality was performed with the Rasch model, and the results obtained were organized in the output table as shown in Figure 1. The construct validity results of the Raw variance explained by empirical measures produced a score of 32.8%, while the Rasch model predicted 32.9%. Since the empirical construct validation is almost the same as the value predicted by the Rasch model, the minimum unidimensionality requirement of 20% was met (> 40% means good, and > 60% means excellent). Meanwhile, the unexplained variance for the 1st until 5th contrast is less than 10%, which falls in the ideal range value of less than 15% (Sumintono & Widhiarso, 2015). This findings indicate that the online mathematics final examination has a strong evidence of unidimensionality, that is, the items undoubtedly measured the construct of 9 th -grade mathematics.

Item Fit
Construct validation can be performed by involving several validators to obtain better results. Similar items are first tested in small groups, then the results are re-evaluated per item. The teachers can also ensure that the sentences per item are unambiguous and easy to understand. However, although the construct validation was not performed, it can be predicted by the Rasch model. This model was very effectively applied, as it can obtain reliable validity analysis results and is easy to use because it can be directly analyzed by computer applications. Table 5 shows the output table of item fit, which represents the content validity analysis results that can be seen from the level of items' suitability. Table 5 shows the results of the item fit analysis. There were two item statistics of misfit orders, namely the infit and outfit. The outfit statistics were used more frequently due to their higher sensitivity to data with extreme scores. After comparing the 20 items, 16 items were found to be fit, while 4 items were not because they did not meet the 3 criteria above (table 3). Table 5 shows that the topmost item, number 15, did not fit, as it does not meet the requirements for Outfit ZSTD based on the criteria in table 3 (value 3.32) and had a low measure correlation (0.36). This means that item 15 should be investigated more because it does not contribute adequately to classifying students' mathematical abilities. Another interpretation is that it is too difficult for students, and cannot classify those with high and low mathematical abilities. The next step was to revise the test item. Conversely, the infit mean square statistic showed that the average of each item was 1.0, meaning it was within acceptable limits. The MNSQ outfits for test items 19 and 7 were outside the acceptable limit, while number 1 had a low measure correlation, meaning it was too easy or difficult for students. Therefore, the final decision on the test items should be investigated and revised. However, according Sumintono and Widhiarso (2015) if the item meets one of the criteria (Outfit MNSQ, Outfit ZSTD, or PTMEA-CORR), the item should be retained.

Person Reliability
The Online Mathematics Final Examination had 20 items divided into four subtopics, namely Numbers, Algebra, Geometry and measurement, Probability, and Statistics. Subsequently, the Rasch model analyzed the correlation between the students' mathematical abilities and the test items, using an examination taken by 234 students. The Rasch statistical analysis model was divided into two categories, which are the 234 measured persons and the 20 measured items. Tables 5 and 6 discuss the summary statistics for each category in detail. Cronbach's alpha, which is the person raw score reliability that measures the reliability using the interaction between the person and the item as a whole, was 0.86. According to the rating scale of the items' quality criteria, the Cronbach's alpha fell in the range of 0.81 to 0.90, meaning the overall quality of the items was in the very good category. The person measure was +1.28 logit, showing the average value of all students working on the item given. Meanwhile, the average value was larger than the logit value of 0.0, indicating a tendency for students' mathematical abilities to be higher than the level of difficulty. A value of 0.74 was obtained for the person reliability, indicating the consistency of the students' answers was in the sufficient or acceptable category. The teacher could have more items that were more difficult that would mark the trait a little better. That one reason why the person reliability is not higher than 0.9 [29]. The INFIT MNSQ and OUTFIT MNSQ produced average values of 0.99 and 0.97, respectively, where the ideal value is 1, i.e., the closer to 1, the better. In addition, the INFIT ZSTD and OUTFIT ZSTD gave average table person values of 0.10 and 0.13, where, in this case, the ideal value is 0.0, that is, a value closer to 0.0 depicts better quality. The value of person separation was 1.7. According to Wright and Masters (2002), the number of person separate strata (H) can be culculated from the separation index by using the equation: H = [(4 x separation index) + 1] / 3. The person separation index of 1.7 produces a strata (H) of 2.6 (can be rounded up to 3). This value indicates the students can be well distinguished into three different abilities that is, high, medium, and low ability. Figure 3 shows summary statistics for the 20 items of the Online Mathematics Final Examination. The statistical data results were aimed at analyzing the item categories, either difficult, medium, or easy, and provide overall quality information of the student response patterns, the instrument used, and the interaction between person and the items. Subsequently, the average INFIT MNSQ and OUTFIT MNSQ values were 0.99 and 0.97, where the ideal value is 1, that is, the closer to 1, the better. Conversely, the average INFIT ZSTD and OUTFIT ZSTD values of the item table were 0.03 and 0.06, and in this case, the ideal value was 0.0, where a closer score to 0.0 signifies better quality. Based on the data analysis results, the difficulty levels were obtained from the item measure output results. Then, the items were grouped by combining the logit mean and the standard deviation value, which produced averages of 0.00 and 0.91, respectively. These values were used to identify the item separation. The item separation was 4.59, indicating the items can be divided into 5 categories, namely very easy, easy, medium, difficult, and very difficult. Hence, a larger separation value signifies a better quality of the items for dividing the students' mathematical abilities. According to Linacre (2012), an item separation value which is more than 2.00 is interpreted as good. In this study, the value for item reliability was 0.95. Sumintono and Widhiarso (2015) stated that an item reliability which is higher than 0.94 is interpreted as 'excellent'.  Table 6 shows the specific classification of the students' abilities, which were more in the moderate, high and low ability groups than the very low and very high ability groups. The Guttman scalogram observed the student answers on each item and ranked their abilities from the highest to the lowest [12]. As shown in Figure 4, five people in class 9A, three in 9B as well as two in 9C, 9D, and 9H achieved perfect scores. These 14 students answered all the items on the Online Mathematics Final Examination correctly, while the students with codes 032IXG, 093IXH, and 112IXE had the lowest abilities. The three lowest students were at the bottom and could only answer two items out of the 20 correctly. The text continues here. Besides sorting the students' abilities from the highest to the lowest, the Guttman scalogram also sorts them from the easiest to the hardest. The item on the lowest left was the easiest, while that on the top right was the most difficult. Further analysis shows the students who were not careful when working on the examination. In the green highlight table, students with initials 075IXF, 078IXA, and 095IXD only answered one item incorrectly out of 20 given. They worked on the last five items that had higher difficulty indicating that their ability was sufficient, though they were not careful enough to answer the easier items. Meanwhile, the students with initials 056IXF and 065IXF in the blue highlight box answered the first 19 items correctly but incorrectly answered item number 20, showing their ability was still insufficient for the question. The Guttman scalogram can also detect students that cheat when working on the Online Mathematics Final Examination. In the black box highlight case, students with initials 088IXC and 087IXC had consecutive numbers, similar math test scores, and patterns in answering the examination questions. From the data, the teacher can further analyze and determine whether these two students cheated.

The Wright Maps (or Person-item distribution map)
The Guttman scalogram data is very beneficial for teachers and schools. Teachers and supervisors should ensure the Online Mathematics Final Examination is conducted honestly and a pure final score without cheating is obtained. They can also remind students that attitude is more important than grades in the world of work.
In addition, the Guttman scalogram data can be used by teachers to classify students' abilities. By applying this data on the Rasch model to analyze the daily or midterm test, the teacher can identify the items that are still difficult and need to be explained again. The teachers can also assess their low mathematical ability students to assist them and to focus more on helping students with other learning methods or approaches. Meanwhile, the teachers can provide more challenging items for the high-ability students to improve their other math skills or prepare them for competitions. Figure 5. Probability of response -dichotomous curves. Figure 5 illustrates the probability categories and shows that the logit peak was around 0.9, and numbers 1 and 0 did not cover each other. The correctly and incorrectly answered items were balanced, meaning the test adequately measured and divided the students' mathematical ability and into the low and high categories. Consequently, the items can be concluded to have appropriately divided the students' abilities.

Discussion
The final examinations in grades 9 and 12 are very important for determining the students' ability level in schools, cities, and provinces. According to the results, the government can plan to improve the schools' quality in each province. The schools can measure their students' abilities quality, the teachers can make evaluations on lesson plans and teaching methods, and the goal of improving the quality of education in Indonesia for Teacher training in sustainable education can be achieved. However, the government has not held a national examination for two years, particularly in 2020 and 2021, and has entrusted each school to make its final examination to determine students' graduation. Therefore, final examination items should be made appropriately and professionally and each item should be carefully evaluated to ensure the students' mathematical abilities are measured properly. In addition, the items should be fairly moderate at difficult level to allow the students to pass with good grades and slightly difficult to test their abilities.
Subsequently, the research result shows the importance of validating the final examination items. It indicates that although public schools had the predicate "A," the construct validity results did not reach the minimum score. Furthermore, the Rasch model found several items that were not good, and 4 out of the 20 items needed to be evaluated and revised because they were outside the fit item criteria. Hence, these 4 items require further revision and analysis to correspond with the fit category.
The contribution of this research is the importance of preparing the final examination items appropriately and seriously. It also explains the steps of using the Rasch model to analyze important information needed for preparing the final examination. First, determine whether the construct validity is in a good category (X>40%). Second, investigating the misfits on the statistical item table to observe the difficulty level of the items as a whole and analyze the personal statistics to evaluate the students' overall abilities. Third, using the Wright Maps to assess a comparison between the abilities of each student and the difficulty level of each item in more detail. Fourth, analyzing the Guttman scalogram to observe the students' ability to answer the items in detail. In conclusion, these steps ensure that all items are well-prepared and checked whether the test is unidimensional. It also ascertains that there is no sentence error evidence, luck in answering, distractor analysis error, miscoding, etc. This research concluded that the Rasch measurement model can be used by students, teachers, and schools to adequately prepare the final examination items.

Conclusions and Implications
The Covid-19 pandemic has made the world educational institutions to conduct learning online [31], [32]. This situation made Indonesia stop the national examinations, causing each school to prepare its final examination items for determining the students' graduation. By processing the final examination data using the Rasch measurement model, the items and the students' work can be analyzed. The model can show data and help teachers and schools analyze the students' abilities in working on and answering each test item on the online mathematics final examination. Furthermore, the Rasch model shows the difficulty level of each item as an evaluation to improve the quality of the items properly. Therefore, this model can enhance the achievement of item results and determine better goals, meaning the online final examination distractor analysis in mathematics courses works well.
Also, the findings show that the Rasch model can analyze the quality of the Online Mathematics Final Examination Items at public school in Indonesia. The results of the overall construct validation show that the items were in the accepted category, and after further investigation, four items out of the 20 were discovered not to meet the fitness criteria. This means the four items need to be further observed, revised, or discarded. Also, the results of the person reliability show in the good category and item reliability show that the items were in the 'very good' category. The students' abilities to work on the online final examination was above average. In addition, the Guttman scalogram of responses can show data on the student abilities from the highest to the lowest, as well as the less careful students, the ability limits, and those who likely cheated.
The research suggested that the teachers should prepare well to achieve good construct validity when examining students, especially on important examinations, such as UTS and UAS. It is also recommended that teachers use the Rasch model to obtain more specific items analysis. The model can help produce more objective examination items that can work well to measure the students' abilities. Another important message is that teachers should focus on students with low mathematical abilities and help them improve their learning outcomes. With Rasch analysis, the items and sub-topics that students don't understand can be discovered fully, and the teachers can use new methods or ways to re-explain the topics.

Limitations and suggestions for further research
There were several limitations in this research. First, it was only conducted with a small scale of 234 students, and only one public school with a good predicate in Bandung city was selected, hence the preparation of the final examination items for the 9 th -grade junior high school had good results. However, analysis of public schools in remote areas may get different results, and further research can analyze the final examinations in pri-vate schools. The second limitation is that only the final examination items at the junior high school were analyzed, hence other school levels need to be investigated. Third, only mathematics examination items were analyzed, while the final examination to determine student graduation consists of many subjects such as science, foreign language, and others. Therefore, further research can analyze other subjects tested on the final examination using the Rasch model.
Funding: this research funding by Beijing Normal University.