Design Principles and Impact of a Learning Analytics Dashboard: Evidence from a Randomized MOOC Experiment

1. Introduction

Massive Open Online Courses (MOOCs) have expanded access to education worldwide, but they continue to face persistent challenges with learner engagement and success. One contributing factor is the lack of individualized feedback to support self-regulated learning (SRL), the iterative process of planning, monitoring, and evaluating one’s learning, which is strongly associated with improved achievement [1,2,3]. Although SRL is widely recognized as beneficial, many MOOC learners struggle to engage in it effectively [4,5].

Learning Analytics Dashboards (LADs) have been proposed as tools to foster SRL at scale by visualizing indicators of performance, progress, and engagement [6,7]. When designed well, dashboards can help learners reflect on their behavior and adapt strategies [8,9]. Yet their effectiveness remains contested. Reviews have shown that many LADs lack theoretical grounding or contextual alignment [7,10], and others caution that poorly designed dashboards may increase cognitive load, encourage unhelpful social comparison, or even discourage learners [11,12]. Evidence to date has been mixed, with many studies relying on small-scale pilots or lab settings that limit generalizability [13,14]. This has led to ongoing debate about whether LADs have truly lived up to their promise [11].

A recurring critique of LAD design is the absence of strong theoretical foundations. Matcha et al. [7] found that most dashboards lacked explicit connections to SRL theory, while more recent reviews suggest incremental progress but continued inconsistency. Paulsen et al. [15] argue that the field is moving “from analytics to learning”, but note that theoretical integration is often partial. Similarly, Masiello [16] highlights that dashboards remain largely descriptive, translating data into visualizations but rarely into actionable, pedagogically meaningful guidance.

Another concern is the use of peer-referenced indicators. While showing learners how they compare to their peers can provide useful benchmarks, it can also trigger counterproductive forms of social comparison. Classic theory suggests that upward comparison often undermines motivation [17,18], and empirical studies confirm this risk in online learning settings [12,19]. To mitigate such effects, researchers have proposed using prior cohorts rather than real-time peers as reference points [20], or offering multiple benchmarks (e.g., passing, certificate-ready, mastery) aligned with diverse learner goals [21,22].

A third critique relates to inference cost, the difficulty of interpreting and acting on dashboard information. Complex or abstract visualizations have been shown to impose high cognitive demands [23,24], with their benefits often skewed toward more educated learners [20]. Recent studies emphasize that explanatory, goal-oriented designs reduce extraneous cognitive load [25], while poor designs can undermine motivation by eroding learners’ sense of competence [26]. Reducing inference costs requires not only careful visualization design but also the inclusion of actionable feedback. The ARCS model [27], which structures feedback to capture Attention, highlight Relevance, build Confidence, and foster Satisfaction, has proven effective in online learning environments [28,29].

Finally, LAD evaluation practices have often been limited. Many studies rely on small pilots, lab settings, or usability testing, restricting generalizability [13,14]. Field studies in authentic learning contexts are rarer but crucial, as they reveal confounding factors and provide stronger evidence for both researchers and practitioners [12,30,31,32]. Reviews consistently call for more large-scale, controlled evaluations of dashboards in real courses with diverse learners [11].

This study addresses these gaps by designing a theory- and context-grounded LAD for a credit-bearing MOOC in supply chain management and evaluating it through a large-scale randomized controlled trial (RCT) with 8,745 learners. Guided by the COPES model of SRL [33,34] and informed by historical course data, the dashboard incorporated pacing and progress indicators, with or without actionable ARCS-framed feedback. Our findings show that dashboards without feedback offered no measurable benefits, while dashboards with feedback significantly increased learners’ verification rates (a marker of commitment) but had mixed effects on engagement and no effect on final performance. These results suggest that dashboards are not inherently beneficial, their impact depends on specific design choices. By combining design principles with experimental evidence, this work contributes to ongoing debates about the value of LADs and offers practical guidance for building dashboards that support self-regulated learning at scale.

2. Materials and Methods

This section describes the context, design, and evaluation of the study. We begin with an overview of the MOOC that served as the research setting, then detail the development of the Learning Analytics Dashboard (LAD), and finally outline the experimental design used to assess its impact.

2.1. Course Context

The intervention was implemented in a 14-week MOOC in supply chain analytics, part of a credential-bearing online program in supply chain management. The course ran from April to August 2023 and enrolled 8,745 learners, the majority of whom were working professionals. Learners could audit the course for free or enroll as verified learners by paying a fee. Only verified learners got access to graded assignments and the final exam and were eligible for a certificate if they achieved a grade of 60% or higher.

The course consisted of five instructional modules followed by a final exam. Each module included lecture videos, practice problems, and a graded assignment. Graded assignments contributed 10% of the final grade, while the exam contributed 90%. Survey data from previous course runs (2021–2022) indicated that verified learners typically pursued two main goals: earning a certificate for professional purposes or achieving a high grade as a pathway toward graduate credit. A recurring theme in these surveys was anxiety about pacing and uncertainty about progress, which motivated the development of a learner-facing dashboard to scaffold self-regulated learning.

2.2. Dashboard Design

The Learning Analytics Dashboard (LAD) was designed using the COPES model of self-regulated learning [33,34], which conceptualizes regulation as cycles of conditions, operations, products, evaluations, and standards. From this perspective, dashboards act as external feedback systems that can supplement or correct learners’ often biased self-assessments.

To contextualize the design, we conducted exploratory analyses on clickstream data from earlier runs of the course (2021-2022). Multiple linear regression was used to predict final grades from behavioral traces (see Table 1). Indicators were selected if they showed statistical significance in the linear model ($p<.05$). This process yielded three indicators: (1) number of unique lecture videos completed; (2) number of unique practice problems submitted; and (3) number of practice problem solutions viewed.

Consultations with the teaching staff underscored concerns about learners’ pacing behaviors and highlighted the critical role of structured study planning in online learning. These insights informed the inclusion of time-related features in the dashboard to scaffold effective pacing strategies. The decision was further motivated by evidence on the spacing effect, which demonstrates that learning is more effective when study sessions are distributed over time rather than massed together [35,36].

Two dashboard variants were developed to examine the role of feedback in shaping learners’ interpretation and use of these indicators. The first presented the indicators with basic visualizations and general words of encouragement. The second combined the same indicators with actionable messages structured using Keller’s ARCS model [27] (Attention, Relevance, Confidence, Satisfaction). Messages were designed to highlight pacing strategies, encourage persistence, and reinforce learners’ progress.

2.3. Experimental Design

We conducted a three-arm randomized controlled trial (RCT) to evaluate the LAD’s impact on learner outcomes. Upon enrollment, learners were randomly assigned to one of three groups: (1) Group C (Control): no dashboard; (2) Group A (LAD without feedback): dashboard with indicators and generic encouragement; (3) Group B (LAD with feedback): dashboard with indicators plus ARCS-framed actionable feedback.

Learners in Groups A and B accessed their dashboards through an Engagement Dashboard button on the course landing page. To ensure consistency in the interface, control learners saw a similar button leading to a survey. Engagement with the dashboard was voluntary, with no incentives to click.

Three outcome variables were analyzed: verification status (binary: verified vs. not verified), engagement (total hours spent in the course, derived from clickstream logs), and performance (final course grade, available only for verified learners, on a 0-1 scale). Verification was analyzed with logistic regression, using Group C as the reference category. Engagement and performance were analyzed with one-way ANOVAs, followed by Tukey’s HSD for post-hoc comparisons. Because engagement data were positively skewed, engagement analyses were conducted on both raw and log-transformed values. Effect sizes were reported as odds ratios (ORs) for logistic regression and eta squared (${\eta }^2$) for ANOVAs. Statistical significance was set at $\alpha =.05$. All analyses were conducted in Python 3.10 using the statsmodels, scipy, and statsmodels.stats.multicomp packages.

3. Results

The results are organized into two parts. First, we present the design of the Learning Analytics Dashboard (LAD), derived from theory and contextual data. Second, we report outcomes from the randomized controlled field experiment.

3.1. Learning Analytics Dashboard Design

Grounded in the COPES model and informed by historical learner data, we derived four principles for the design of a theory- and context-driven LAD: (1) use benchmarks from prior cohorts instead of real-time peers to avoid unproductive social comparison; (2) provide multiple goal-aligned standards to reflect the diverse intentions of MOOC learners; (3) combine low-inference visualizations with actionable, ARCS-framed feedback to reduce inference costs and support motivation; and (4) make pacing visible to promote spacing effects. These principles, along with their implementation in dashboard components and theoretical grounding, are summarized in Table 2.

The final LAD (Figure 1) implemented these principles through six components. Engagement KPIs and Time Estimate (a, b) provided progress benchmarks and workload references, aligned with both passing and high-achievement trajectories. Countdown, Weekly Streak, and Time Spent Last Week (c, d, e) emphasized pacing, encouraging learners to spread study sessions rather than relying on cramming. Messages (f) differed by dashboard variant: in the no-feedback version, learners received only generic words of encouragement, whereas in the feedback version, messages were tailored to learners’ current progress and structured using Keller’s ARCS model [27] to strengthen attention, relevance, confidence, and satisfaction. All visualizations were deliberately kept simple (bar and line charts with explanatory tooltips) to minimize extraneous cognitive load, while feedback messages translated indicators into clear next steps.

Design validation with eight former learners suggested strong usability and positive reception. Participants particularly valued the combination of encouragement and actionable guidance, noting that encouragement alone might not suffice for those falling behind. They also expressed interest in countdown and additional pacing features to support planning.

3.2. Experimental Evaluation

We evaluated the impact of the LAD through a randomized controlled trial with three conditions: Group C (control, no dashboard), Group A (dashboard without feedback), and Group B (dashboard with feedback). Outcomes were examined across three dimensions: verification (whether learners paid to pursue a certificate), engagement (time invested in the course), and performance (final grades on the course). These measures capture both learners’ behavioral commitment to the course and their academic achievement.

3.2.1. Verification

Verification rates are summarized in Table 3. Learners who received the dashboard with feedback (Group B) were more likely to upgrade to the verified track (18.1%) than those in either the control condition (15.7%) or the dashboard without feedback (15.7%). Logistic regression confirmed a significant effect of condition, with learners in Group B showing higher odds of verification relative to the control (OR = 1.19, 95% CI [1.04, 1.37], $p=.012$; Table 4). By contrast, the dashboard without feedback (Group A) had no effect compared to the control (OR = 1.00, 95% CI [0.87, 1.15], $p=.990$). These findings indicate that dashboards enhanced with feedback increased learners’ willingness to invest financially in the course to pursue a certificate, whereas dashboards without feedback had no measurable effect.

3.2.2. Engagement

Engagement was measured as total time spent in the course (in hours). Descriptive statistics are reported in Table 5. A one-way ANOVA on raw values indicated significant differences between groups, F(2, 8742) = 4.26, $p=.014$, but the effect size was negligible (${\eta }^2=.001$). Tukey post-hoc tests showed that learners in Group B (dashboard with feedback) spent significantly more time than those in Group A (dashboard without feedback), and marginally more than the control group (Group C; $p=.056$). However, due to the positive skew of the engagement distribution, we repeated the analysis on log-transformed data. In this robustness check, the group effect was no longer significant, F(2, 8742) = 1.53, $p=.217$, and no pairwise contrasts reached significance (Table 7). Taken together, these results indicate that the differences observed in raw data were driven by a small subset of highly engaged learners, rather than a consistent effect across the population.

Table 5. Descriptive statistics for engagement (total time in hours) by experimental group.

Group	n	Mean	SD	Median
Group A (no feedback)	2,952	10.53	28.09	0.48
Group B (with feedback)	2,907	12.64	32.54	0.46
Group C (control)	2,886	10.84	28.72	0.44

Table 5. Descriptive statistics for engagement (total time in hours) by experimental group.

Group	n	Mean	SD	Median
Group A (no feedback)	2,952	10.53	28.09	0.48
Group B (with feedback)	2,907	12.64	32.54	0.46
Group C (control)	2,886	10.84	28.72	0.44

Table 6. One -way ANOVA and Tukey post-hoc results for engagement (raw values). Engagement was measured as total time in hours.

Source	Sum Sq	df	F	p
Experimental group	7.59e3	2	4.26	.014
Residual	7.79e6	8742
Effect size
${\eta }^2$	0.001
Post-hoc Tukey HSD comparisons
Comparison	Mean diff (h)	95% CI	p	Significant
Group A vs. Group B	2.11	[0.28, 3.94]	.019	Yes
Group A vs. Group C	0.31	[–1.52, 2.14]	.917	No
Group B vs. Group C	–1.80	[–3.64, 0.04]	.056	No (trend)

Table 6. One -way ANOVA and Tukey post-hoc results for engagement (raw values). Engagement was measured as total time in hours.

Source	Sum Sq	df	F	p
Experimental group	7.59e3	2	4.26	.014
Residual	7.79e6	8742
Effect size
${\eta }^2$	0.001
Post-hoc Tukey HSD comparisons
Comparison	Mean diff (h)	95% CI	p	Significant
Group A vs. Group B	2.11	[0.28, 3.94]	.019	Yes
Group A vs. Group C	0.31	[–1.52, 2.14]	.917	No
Group B vs. Group C	–1.80	[–3.64, 0.04]	.056	No (trend)

Table 7. One-way ANOVA and Tukey post-hoc results for engagement (log-transformed values). Engagement was measured as total time in hours, log-transformed using $log(1+x)$.

Source	Sum Sq	df	F	p
Experimental group	1.05e1	2	2.60	.075
Residual	1.77e4	8742
Effect size
${\eta }^2$	0.001
Post-hoc Tukey HSD comparisons
Comparison	Mean diff (log h)	95% CI	p	Significant
Group A vs. Group B	0.067	[–0.020, 0.154]	.168	No
Group A vs. Group C	–0.012	[–0.099, 0.075]	.946	No
Group B vs. Group C	–0.079	[–0.166, 0.009]	.088	No

Table 7. One-way ANOVA and Tukey post-hoc results for engagement (log-transformed values). Engagement was measured as total time in hours, log-transformed using $log(1+x)$.

Source	Sum Sq	df	F	p
Experimental group	1.05e1	2	2.60	.075
Residual	1.77e4	8742
Effect size
${\eta }^2$	0.001
Post-hoc Tukey HSD comparisons
Comparison	Mean diff (log h)	95% CI	p	Significant
Group A vs. Group B	0.067	[–0.020, 0.154]	.168	No
Group A vs. Group C	–0.012	[–0.099, 0.075]	.946	No
Group B vs. Group C	–0.079	[–0.166, 0.009]	.088	No

3.2.3. Performance

Performance was analyzed using final course grades. This analysis was restricted to verified learners since they were the only ones who had access to graded assignments and the final exam. Descriptive statistics are reported in Table 8: learners in Group C (control) achieved slightly higher mean grades (M = 0.39) than those in Group B (M = 0.37) and Group A (M = 0.35), but differences were small. A one-way ANOVA revealed no significant differences between groups (F(2, 1438) = 1.46, $p=.233$) with a negligible effect size (${\eta }^2=.002$). Tukey post-hoc comparisons confirmed that none of the pairwise contrasts were significant (all $p>.20$; Table 9). These results indicate that neither dashboard condition, with or without feedback, had a measurable effect on learners’ final grades.

4. Discussion

This study set out to design and evaluate a Learning Analytics Dashboard (LAD) grounded in theory and contextualized in a MOOC, with the goal of enhancing self-regulated learning (SRL) and improving learner outcomes. By combining insights from SRL theory, data-driven indicator selection, and prior critiques of LAD design, we proposed a set of design principles and tested their impact in a large-scale field experiment. The results highlight both the promise and the pitfalls of LADs: dashboards with actionable, ARCS-framed feedback increased learners’ likelihood of verification (paying for the option to get a certificate) and showed tentative signs of boosting engagement, while dashboards without feedback provided no measurable benefit and may have imposed additional cognitive costs. Across all conditions, no significant effects were observed on final performance.

These findings extend the ongoing debate captured by Kaliisa et al. [11], who noted that evidence for positive effects of LADs remains mixed. Our study underscores that dashboards are not uniformly beneficial: simply visualizing learner data, without clear interpretive support, can increase inference costs and fail to motivate action. In contrast, dashboards that integrate low-inference visualizations with actionable feedback can foster motivation and persistence, leading to higher levels of commitment. In this sense, the question is not whether LADs “work,” but under what design conditions they provide value.

The results also align with research on cognitive load and motivation. Explanatory, goal-oriented designs have been shown to reduce extraneous cognitive load [25], while poorly structured feedback can undermine learners’ sense of competence [26]. Our findings fit this dual mechanism: learners who received indicators without guidance saw little benefit, whereas those who received ARCS-framed feedback showed stronger commitment to the course. Feedback thus appears essential for transforming dashboard data into actionable strategies while also sustaining motivation.

A further contribution of this study concerns pacing. Building on evidence that distributed engagement predicts certification more strongly than total time-on-task [38], we incorporated features such as weekly streaks and time-on-task indicators. While these elements alone were insufficient to produce significant effects, their integration with actionable feedback may explain why the feedback condition produced more favorable outcomes. This suggests that pacing supports are most effective when combined with guidance that helps learners interpret and act on them.

Taken together, our findings emphasize that the impact of LADs depends not on their mere presence but on their theoretical grounding, contextualization, and feedback design. Effective dashboards need to reduce inference costs, support motivational needs, and provide meaningful pacing scaffolds. Future research should continue to explore which combinations of features influence different learner outcomes, and how learner characteristics (e.g., prior achievement, goals, self-efficacy) moderate dashboard effectiveness.

4.1. Limitations and future work

This study has three main limitations. First, it was conducted in a single MOOC on supply chain management, primarily targeting working professionals, which constrains generalizability to other subjects, learner populations, or formats such as instructor-paced or non-credit-bearing courses. Second, the dashboard was evaluated as a bundled intervention, making it impossible to isolate the specific contributions of individual components—for example, whether the increase in verification rates was driven by ARCS-framed messages, pacing indicators, or their interaction. Third, the study focused on short-term outcomes within a single course; we did not examine whether exposure to dashboards fostered lasting SRL practices or longer-term academic gains. Future work should therefore test LADs across more diverse settings, employ experimental designs that isolate the contribution of individual features, and track learners longitudinally to capture sustained impacts.

5. Conclusions

This study designed and evaluated a Learning Analytics Dashboard (LAD) grounded in SRL theory and contextualized in a MOOC, testing its impact in a large-scale field experiment. The results show that dashboards are not inherently beneficial: a LAD without actionable feedback offered no measurable advantages and in some cases was associated with negative outcomes, while a LAD with ARCS-framed feedback increased learners’ verification rates and showed tentative benefits for engagement. No differences were found in final course performance across groups.

These findings suggest that LADs are most effective when they combine low-inference visualizations with actionable, motivationally framed feedback, and when they make pacing strategies explicit. For practitioners, the results caution against deploying dashboards that simply display learner data, while highlighting the value of embedding interpretive support that helps learners translate indicators into concrete actions. For researchers, the study underscores the need for designs that isolate the contribution of dashboard features and for evaluations across diverse contexts to determine when and for whom LADs are effective.

Ultimately, the question is not whether LADs have lived up to the hype, but under what conditions they can deliver on their promise. By grounding dashboards in theory, contextualizing them with course-specific data, and embedding feedback that supports both cognition and motivation, future work can move toward LADs that reliably foster self-regulated learning and learner success at scale.