How to Learn and How to Teach the Five Content Areas: Programming Languages, Game Design, Computer Thinking, Algorithmic Thinking and Robotics Programming - A Systematic Review of Journal Publications From 2015 to 2020”

1. Introduction

With the technology development, the way of education is changing [1]. Therefore, there is a need to include technology [2] and computer science [3] in school curricula. Since 2011, the importance of e-skills has been recognized, so relevant organizations have been involved in defining computer science in schools and it is concluded that computer science topics should be an integral part of the school curricula [4,5,6]. According to [7] curriculum changes have been made throughout the world like: the UK [8,9,10], Europe [11] Australia [12] and New Zealand [13].

The great commitment to K-12 computer science education indicates the recognition of its importance and helps to solve the lack of computer experts around the world. The creation of a viable model of computer science curriculum and its implementation at the K-12 level is a necessary first step toward reaching these goals [14]( p. 3).

In the 1990s, people's interest in programs that use the graphical environment for programming, debugging, etc. began to rise. Increased use of the graphical environment for programming learning can be considered as expected, since learning programming using a traditional approach is more complex for some students. Therefore, it is necessary to realize a graphical environment that will be easy to manage and available to students and teachers interested in "graphic programming" [15] [16]. Numerous studies suggest the use of visual programming languages (when learning programming) like: Logo [17,18,19,20], Scratch [21,22,23,24], Alice [25,26,27]. However, with the growing interest of students in learning graphically oriented programming languages, the gradual introduction of textual programming languages began in schools and, thus, a new field of research was opened for authors [28,29,30,31].

In parallel with the introduction of computer science (learning programming, above all) in the curricula of primary school, there was a need to consider how to develop computer thinking (CT) in children. Computational thinking is a type of analytical thinking that implies approach to problem solving. It is about mathematical thinking where we can tackle problems in a variety of ways. It can also be expanded to imply the understanding of human behavior and intelligence [32]. Lu and Fletcher [33] state that “proficiency in computational thinking helps us to systematically and efficiently process information and tasks”, while Dede, Mishra, and Voogt [34] give the following definition “Computational thinking is seen as a skill set that every child needs to develop”. Voogt, Fisser, Good, Mishra and Yadav [35] state that “programming is an important tool to help develop computational thinking skills”, and Román-González, Pérez-González, and Jiménez-Fernández [36]: “computational thinking must be acquired by the new generations of students to thrive in the digital world”. Kong, Chiu, and Lai [37] in his study confirms the fact of Brennan and Resnick [38] and Papert [39] that, with the development of CT, students can creatively express their ideas. In addition to computer thinking [33,34,40,41,42,43] algorithmic thinking has been studied in the literature [44,45,46,47].

Authors Wilson, Hainey and Connolly [48] present empirical proofs and further guidance on the assessment of the ability of elementary school students to do games-based programming. The after-school club named “Code club” was founded in Great Britain in 2012 to support the elementary school in the field of programming. Students are trained together to create games in the Scratch program [49]. Many other studies have shown increased interest in game design (games-based programming) [50,51,52,53,54].

Robotics is a growing field that has the potential to significantly impact the nature of engineering and science education at all levels, from K-12 to graduate school [55] (p. 1). Robotics provides “source of energy” that can be used to motivate children to learn [56] and, more specifically, to interest students to learn many aspects of robotics, programming, and computational thinking [57,58]. Also, a large number of authors gave their views related to robotics programming issues [59,60,61,62,63].

In the study, presented in this paper, the representation of five content areas (programming languages, game design, computer thinking, algorithmic thinking, robotics programming) in published papers of the cited journals has been investigated. Also, in order to be included in this study, it was necessary that research presented in analysed papers includes participants aged between 6 and 15 years. The research questions are modified version of [64]:

• What are the journal name and year of publication of the articles, country context, education level and age of the participants, learning domain, teaching tools, research design, previous participant experience, research methodology, course duration, grant or project and research purpose?
• What is the impact of the content areas on student learning performance, motivation, attitude and perception?

The paper has been organized as follows: section 2 analyses related work; section 3 presents research method (review process, analysis framework and coding); section 3 analyses the research results and gives the discussion of the results; section 4 provides limitations and future research direction; section 5 presents main conclusions.

2. Related Work

In numerous papers, researchers have studied learning achievement of primary and secondary schools’ students in the following areas: programming language, computer thinking, algorithmic thinking, game design and robotics programming. These areas are usually an integral part of computer science curricula. Some of the listed areas have been also included in a large number of review papers.

In the programming field, Popat and Starkey [65] reviewed research papers that analyse educational outcomes of learning students to code at school, while Costa and Miranda [66] presents a systematic review of the literature about the effectiveness of the use of Alice software in learning programming comparing with using a conventional programming language. The papers [67,68,69,70,71,72,73] deal with research of learning programming languages as well.

According to Hsu, Chang, and Hung [74], the basic problem of computational thinking (CT) is how to teach it. The authors reviewed the academic journals and searched in them adopted learning strategies and teaching tools in CT education. It has been established that CT is applied in the field of computer science and that there are also studies related to other subjects. The systematic review of computational thinking development through programming in Scratch was presented by Zhang and Nouri [75]. Also, authors [76,77,78,79,80] show the review in the computer thinking area.

In the article [81] the methodologies, frameworks, and models applied to game designs, and phases of game development software are given. The review of programming curricula in seven countries and programming games were analysed by Lindberg, Laine and Haaranen [82], as well by [83,84,85,86]. Benitti [87] reviews recently published articles in the use of robotics in schools in three main directions: identifying the potential contribution of the applying robotics as educational tool in schools; presenting a synthesis of the available empirical evidence on the educational effectiveness of robotics; defining future research perspectives concerning educational robotics. Xia and Zhong [88] also present review of empirical studies on teaching and learning robotics content knowledge in K-12 and explore future research perspectives of robotics education (RE), based on the reviewed papers. References [89,90,91,92] show research review in robotics area as well.

3. Methods

In this paper, the general guidelines proposed by Kitchenham and Carters [93], have been adapted. It allows to collect empirical evidence about formulated research questions.

3.1. Review process

In this study, an analysis of the articles published in 13 academic journals has been shown. Analysed journals are: Computers in Human Behavior (CHB), Computers & Education (CAE), British Journal of Educational Technology (BJET), Journal of Educational Technology and Society (JETS), Interactive Learning Environments (ILEs), Learning and Instruction (EARLI), Electronic Liberary (EL), Educational Technology Research & Development (ETR&D), Journal of Computer Assisted Learning (JCAL), The Turkish Online Journal of Educational Technology (TOJET), Journal of Educational Computing Research (JECR), IEEE Transactions on Learning Technologies (TLT), International Journal of Computer-Supported Collaborative Learning (IJCSCL) from 1st January, 2015 to 31st December, 2019. The list of journals can be observed as modification of the list considered in Chauhan [94]. The search process was carried out between January 2020 and June 2020. A detailed description of the review procedure is given below.

We reviewed 7580 articles, i.e. all editions of the journal, listed above (Table 1) were counted and it was systematized 46 articles (Table 2) based on selected content areas: programming languages, game design, computer thinking, algorithmic thinking, robotics programming. The condition was that the review paper covers the listed topics in the context of learning, achievements, etc. of school students (aged 6 to 15 years old). Firstly, it was necessary to exclude journals that are not relevant to our research. The research “exclusion strategy” was performed manually through two phases. The first phase involved reading the title and the abstract of each article. In the next phase, the papers that seemed relevant were reviewed in detail by reading the full text [65]. The articles were limited to those published in academic journals. Book reviews, editorial materials, PhD dissertations, meeting abstracts, proceedings papers, were not included in this study. A detailed overview of the literature research and review process is shown in block diagram in Figure 1.

3.2. Analysis framework and coding

3.2.1. Research design

Elements were chosen for analysis in accordance with the research questions and represent modification of the elements developed by [95,96,97]: 1) the journal of publication, 2) year of publication, 3) the country context (of the first author), 4) the education level and age of the participants (elementary school, secondary school, middle school and mixed subjects- aged 6 to 15 years old), 5) learning domain (programming, robotics programming, game design, computer thinking, algorithmic thinking, and review paper), 6) teaching tools (Scratch, Alice ...etc.), 7) research design, 8) previous participant experience 9) research methodology (research questions, hypotheses, pre/post-test, interviews, control group), 10) the duration of the course, 11) grant or project, 12) research purpose. In this review, each of the research elements were coded individually or were given descriptively. The methods of analysis of each element are described in the continuation:

• journal of publication, year of publication, country context (of the first author). The basic information about listed elements, described in the published articles, is discussed. The aim is to establish which countries have more frequently published articles.

• education level and age of participants – information about: elementary school, primary school, secondary school, middle school (depending on the appropriate level of study of the country in which the research was conducted) and a combination thereof, are coded with the correct name of the study level and grade (considering appropriate interval from 6 to 15 year old). Also, the age of the participants is given numerically (Appendix). The main purpose of the analysis is to determine which educational level(s) are covered in chosen papers, as well as to compare the number of participants in articles per educational level.

• learning domain- covers topics: programming, robotics programming, game design, computer thinking, algorithmic thinking, and review paper from those topics. Topics are coded by their name (Appendix). The aim is to understand which topic is most represented in published articles.

• teaching tools – represents programming languages and software. This category includes subgroups like Scratch (Scratch Jnr, Blockly), Alice, code.org (The Maze, The Canvas’, The Artist 2), programs designed by authors of articles (Koios at al...) (ROBOTCG Graphical - graphical interface, Simulation Creation Toolkit, Unity's 2D development tools, Turtle Graphics Tutorial System, whereby one or more teaching tools may be listed in the analysed papers. Teaching tools are coded by their exact name (Appendix). If the teaching tools are not specified in the article it is coded with "not specified". The main purpose of the analysis of these elements is to determine which teaching tools are used, if any.

• research design – category include: quantitative, qualitative or mixed design. They are coded with their exact name, and if the research design is not specified in the article, it is coded with "not specified".

• previous participant experience – represents the previous experience of participants in working with teaching tools that are done in workshops realized by the article’ author(s). They are coded with "Yes" if the authors stated that the participants have experience, with "No" if it is stated that they have no experience and "not specified" if neither of the previous two attitudes is stated.

• Research methodology - category include: research questions, hypotheses, pre/post-test, interviews, control group. They are coded with "Yes" if the authors have defined them in their papers and with "No" if they are not defined. As the next step of the analysis, a counting is performed in order to determine the exact number for each parameter in relation to the number of papers as well as to determine the paper that has the most represented parameters.

• the duration of the course – is defined by the exact time period specified in the article or by "not specified" if not specified. The aim is to understand which papers had the longest time period of course realization.

• grant or project – coding includes the exact name of the grant or project if the paper was realized within the project, or "not specified" if the grant or project is not specified.

• research purpose – this category is defined by a brief description of the purpose of the research stated by the authors in their papers.

4. Research results and discussions

4.1. Research question one

4.1.1. Journal of publication, year of publication, country context

The Appendix contains the following information: the journal where the article is published, the year of publication, the country context (first author) of articles. The papers were published in 13 educational journals for the period from January 2015 to January 2020. The most of the selected articles (15) were published in the CAE journal, followed by the CHB with 10 articles, JECR and JACL with 5 articles each, JETS with 4, ETR&D with 3, BJET and ILEs journals with 2 articles (see Table 2). The review method, adopted in this paper, was based on the concept of systematic review proposed by (Hsu et al., 2018). The Papers were published by authors from 15 countries: USA 9, Turkey 7, Spain 6, China 5 articles, Taiwan 4, The United Kingdom 3, Greece, Hong Kong and New Zealand 2, Belgium, Italy, Norway, Portugal, Slovenia, Sweden with 1 article each. Figure 2. shows distribution status of the authors by countries. In our study, we only listed the nationality information of the first author in country context. The number of published articles by year is illustrated by Figure 3. From the Figure 3 it can be seen that the trend in the number of publications for the period (2015-2020) is constantly increasing.

4.1.2. Education level and age of participants

In order to have a clearer and more consistent analysis, we have adjusted the Elementary Education category as in [94], to include: Elementary Education, Elementary School, Elementary student, Primary Education, Primary School, Primary Student. The research studies involved elementary students (58%), followed by secondary students (28%) and students in middle education (14%). All students are between 6 and 15 years old (Figure 4). Two papers [98,99] presented a study with the participation of students with special needs (elementary and secondary level, respective).

4.1.3. Learning domain

In systematized papers, the areas of computer thinking and programming are most represented, (16 and 13 articles, respective), which makes (63%) of the total number of papers, followed by game design (7 articles), review articles (6). The area of algorithmic thinking is the least represented (1 article), (see last column a Table 3). This relationship is not surprising because, according to [100], many of the countries in Europe have been attempting to incorporate CT courses into their K-12 education curricula. European Ministries of Education give an overview of their current initiatives and plans regarding computer programming [74,101]. However, it opens up the possibility of further research about the extent to which the computer thinking is included in the existing curricula. Also, in last row the Table 3 we see the number of papers in relation to the education level. The similar analysis was performed in [74](see Table 4, page 306).

4.1.4. Teaching tools

Some of teaching tools were used in 40 selected papers. Figure 5 shows the representation of teaching tools in selected papers. The most common is the Scratch programming language, which has been used in 14 papers. Two or more programming languages [36,102,103] were used in three papers, while one paper used programming software developed by the author of the paper. Such representation of Scratch and similar block-based software is based on the fact that many authors consider visual programming languages as one of the best for "novice programmers" [76,104,105].

4.1.5. Research design

A total of 40 research articles were analysed. In accordance with [106], we defined three types of design (see subsection 3.2.1, item research design) and adjusted our coding based on [64]. All reported methods were included in the analysis. Thirteen articles (32,5%) are based on a quantitative design, followed by eleven articles (27,5%) with a mixed design, and three (7,5%) based on a qualitative research design. The authors in thirteen articles (32,5%) did not explicitly define the research design and, in our coding, we categorized such papers as "not specified". This result of the analysis was obtained because some authors did not use only one research design but mixed. Also, the significant proportion of papers marked as "not specified" is a consequence of the fact that the authors did not explicitly define research design [107]. For example, the authors in [108] predominantly used the qualitative method, and, to a lesser extent, the descriptive analysis. However, since they did not clearly state the research design, for research design "not specified" is put.

4.1.6. Previous experience

In accordance with the coding of this element, out of 40 papers, in 25 (62.5%) papers the authors conducted research with participants who have no previous experience in the field of research. This is followed by 9 (22.5%) papers whose participants had previous experience and 6 (15%) papers where there were no notes on the previous experience of participants. Since some curricula has only recently introduced programming and CT (in primary and/or secondary schools), there is a higher percentage of lack of previous experience in these content areas [12,109,110].

4.1.7. Research methodology

In accordance with the presented coding rules (see subsection 3.2.1, item research methodology), research methodology was modified from the research methodology developed by Crompton and Burke [107]. Many articles reported more than one type of research methodology and all reported methods were included in the analysis. The distribution of the number of research methodologies by papers is shown in Figure 6. However, it should be noted that, of 40 papers included in this paper, only one paper [111] contains all five components of research methodology that we defined in our review.

4.1.8. Course duration

Motivation for the analysis of this aspect was found in the work of the authors Lye and Koh ([95]; see Table 2) who presented the time period of the course in their meta-analysis of review papers. In order to determine the duration of the course as objectively as possible, we considered only those papers that explicitly stated at least two of three parameters month / week / hour (number of weeks and hours per week) (24 papers were considered). Based on that, the longest time courses were conducted in the study presented in (Su, Huang, Yang, Ding, and Hsieh [112] ("three hours per week, 4-month, March to June 2013" (in primary school)) and Ruggiero and Green [99] (“Six-months, 30 workshops, 90 min. per workshop “ (on special needs school for young people)). On the other hand, Basawapatna [113] ("4 days, 16 pattern implementations") and Çakır, Gass, Foster and Lee [114] ("a full day event during the weekend, two workshops"), (both middle school) conducted the research in the shortest time. These results are based on the fact that the longest courses were conducted by authors in primary and special needs schools, where students are introduced to programming for the first time.

However, it is important to note that this analysis did not include papers that realized their workshops as an integral part of compulsory or elective curriculum subjects with a time period of 1-2 semesters [36,108,115,116,117,118], (see Appendix).

4.1.9. Research purpose

In accordance with a significant number of related review papers, in this paper, we have not separately analysed research purpose. For example, the authors of the papers [74,119] in their reviews analysed only two of three elements in their reviews (learning domain, teaching tools and research purpose). Accordingly, in the previous sections we have made a detailed classification in the following categories: learning domain - review of papers based on the content area and teaching tools - review of papers based on the use of software.

4.2. Research question two

4.2.1. Student learning performance, motivation, attitude and perception

In the table 4 there are 40 articles for which we analyzed the impact of studying the specific content area on the following elements: student learning performance, motivation, attitude and perception (based on [64], see Table 5 page 115). Review articles were not included in this analysis. The coding in the Table 4 is performed as follows: listing the previous elements in a positive sense, confirming hypotheses and research questions is marked with “positive”; if the authors stated that the hypotheses and research questions were not confirmed and/or if there are negative attitudes related to the listed elements “negative” mark is put; when it is noted that there is no significant statistical difference or the “positive” or “negative” impact cannot be determined with certainty, the field in the table is marked as “neutral”; in case that there is no statement that caused the use of the aforementioned marks or if the elements are not listed, “not specified” is put.

4.2.1.1. Learning performance

A significant percentage (50%) of the reviewed articles reported a positive impact of the studying some of the listed content areas on student learning performance. Neutral impact was reported by 2% of the articles. This report can be considered as a consequence of the fact that elementary students have limited programming experience (fifth grade). Results showed that there was no statistical difference found for student performance [58]. However, 48% of the articles did not investigate impact on student learning performance (Figure 7).

4.2.1.2. Learning motivation

Positive impact of the content area studying on student learning motivation has been found in 32% of reviewed articles. Only two studies of the reviewed articles reported negative and neutral impact (every per 3%). Witherspoon, Schunn, Higashi and Shoop [120], in results of motivational analyse, show a decline in all our motivational measures, therefore we coded their paper with negative mark. We conclude that coding is neutral for the paper Zhao and Shute [121] since the game, used in the paper, did not have an additional reward system, for what is believed to increase the motivation of players. The rest (62%) of the articles in our review did not investigate the motivational impact of the content area studying (Figure 8).

4.2.1.3. Students' attitude

For 35% of review journals impact according to student attitudes is coded as "positive". Only one article has a negative impact (3%; [122]), because the hypotheses dealing with students’ attitude are rejected in the study. However, two articles are coded with “neutral”. The authors Kong et al. [37] in their results pointed out the fact that “Students with better attitudes towards collaboration had more creative self-efficacy but not more programming self-efficacy”, while the authors Yücel and Rızvanoğlu [123] concluded that girls have “negative attitudes towards serious educational games which were rather the opposite in their male counterparts”. Based on quoted claims that girls have a negative attitude and men a positive one, we coded these cases with “neutral”. Most of the reviewed articles 57% did not refer to student attitudes (Figure 9).

4.2.1.4. Students' perception

Only 25 percentage of the reviewed articles reported a positive impact of the studying some of the listed content areas on student perception. Zhao and Shute [121], in the discussion results of the article, conclude that “the levels more challenging to solve, and thus may have lowered students' perceptions of competency.”, while Yücel and Rızvanoğlu [123] in the results show that “failure perception in a code learning game”. The majority of the reviewed articles (70%) did not investigate the perception impact (Figure 10).

5. Limitation and future research

The literature review, given in this paper, includes articles published in thirteen academic journals in the last five years (January 2015 - January 2020). Therefore, this study can be observed as limited since only chosen journals in predefined time period were analysed. These journals may not include all significant research results related to the following content areas: programming languages, computer thinking, algorithmic thinking, game design, robotics programming. Also, given that the level of education in different countries may differ according to age, the authors took into account the age of children from 6 to 15 years old as well as the names of individual levels (primary, elementary, secondary, middle) that cover the mentioned age range.

There were analysed papers where the authors did not state the age or exact educational level of the participants. Based on the information available on the official website of the Ministry of Education of the countries in which the authors conducted the research, we defined the missing data to perform the analysis.

For the sake of better credibility of the analysis of the results, we coded only those elements that the authors explicitly stated in their works, while in all other cases we coded with “not specified”. It is important to emphasize that the “not specified” label does not necessarily mean that individual elements are not explained in detail in the analysed papers, but that they are not specified in accordance with the defined categories.

The two authors have implemented review and analysis, which potentially increases the relevance of comprehensive review, shown in this paper, in comparison with those that the review carried out by a single person. Nevertheless, although the papers in the field were analysed in detail and all relevant data were entered in the tables, the lack of a broader expert team could be seen as a limitation of the research.

Also, journals written in English were analysed, and, thus, papers written in other languages were not included in shown analysis.

Given these limitations, the future research will be aimed in several directions. One research direction may be targeted at respondents over the age of 15 (high school and college level education) and can include a larger scope of papers. Another future research direction involves considering the involvement of a wider expert team in the analysis of a larger number of papers. One possible future research direction could include defining subcategories so that the percentage of “not specified” data is reduced.

6. Conclusions

This study presents a systematic review of empirical studies in academic journals between January 2015 and January 2020. The purpose of this study is to review, analyse, and classify data from selected papers (research content areas: programming languages, computer thinking, algorithmic thinking, game design, robotics programming; participants in the studies were between 6 and 15 years old). It is important to note that the largest number of papers belongs to the field of CT, where two papers had students with special needs as participants (field: CT and game design). It was found that the number of CT works increased significantly in recent years, and that including CT in curricula has received positive comments from scientists in numerous countries ([74]; (p. 308).

In general, analysed paper indicates that students showed a satisfactory degree of progress in learning programming language, game design and robotics programming. On the practical level, teachers are the keystone in the implementation of the selected content areas. It was noticed that the students began to develop the ability of computer thinking through their active participation in the course of their teaching (by implementing programming languages, game design and robotics programming exercises on a computer). The papers mainly focused on programming skills training and CT, while some papers applied Project-Based Learning as part of their research workshops.

In the process of analysing the papers, it was noticed that some papers lack detailed descriptions in their research methodology. That is, they do not provide a complete background description of the research. The most common shortcomings are: lack of data on the age and number of participants and the length of the workshop, which, according to [75], can impede analysing individual articles and comparison between different articles.

Given research, from the point of view of the analysed works, describes a promising result in selected content areas: programming languages, computer thinking, algorithmic thinking, game design and robotics programming, for the participants between 6 and 15 ages. The future work, mentioned in the previous section, should address potential research limitations.