Preprints.org logo
Preprint
Article

This version is not peer-reviewed.

Recognising Learning Phases in FLOSS Mailing Lists: Transformer Language Models Versus Classical NLP and A-Priori Learning Models

Submitted:

03 October 2026

Posted:

06 October 2026

You are already at the latest version

Abstract
FLOSS mailing lists record how newcomers learn, and earlier process-mining studies recognised learning activities in their messages with a-priori keyword models. This paper asks whether the architectures that now dominate natural language processing can perform that recognition step better, and what the language of contributions reveals about how participants mature. In Git and the Linux kernel every contribution is an e-mail that, once accepted, becomes a commit whose trailers (Reviewed-by, Helped-by, Mentored-by) name those who helped. This yields a corpus of 1,432,256 accepted patch e-mails with a learning clock for each author and a graph of helper-learner interactions. On author-disjoint partitions we compare an a-priori phase lexicon, TF-IDF with logistic regression, a Transformer trained from scratch and a Transformer pretrained by masked language modelling on 210,802 e-mails. Newcomers' e-mails (AUROC 0.789 in Git and 0.768 in Linux) and mentored work (0.688) are recognised best by TF-IDF with logistic regression. Pretraining improves the Transformer over training from scratch but leaves it 0.02 to 0.04 AUROC behind, and the a-priori lexicon is weakest (0.644, 0.575, 0.549). A fixed-cohort analysis shows that contributors' language moves towards a collective voice and explicit rationale as they mature, as the a-priori model predicts, whereas help received on the first patches is not associated with retention. A protocol for evaluating large language models on the same test sets is provided.
Keywords: 
learning analytics
;  
mailing lists
;  
transformers
;  
language models
;  
NLP
;  
FLOSS
;  
newcomers
;  
mentoring
Subject: 
Computer Science and Mathematics  -   Artificial Intelligence and Machine Learning

1. Introduction

In many FLOSS projects, the mailing list is where teaching happens. A newcomer sends a patch, reviewers reply with questions and corrections, and the patch is revised and resent until the community accepts it. Knowledge passes from experienced to less experienced participants in natural language, and it does so in public and in writing [16,52,56].
Our earlier work treated mailing archives as records of this learning. We described learning in FLOSS with an ontology of activities and phases [41] and an Abstract State Machine model [40], reviewed how FLOSS data can be turned into event logs [26,42], and verified a-priori learning models on mailing archives by recognising phase-specific activities in messages and mining the resulting process models [25,43,44,45]. In those studies, the step from text to learning activity relied on keywords. The approach is transparent, but it is also brittle, and later work began to replace it with machine learning [27,47,48].
Natural language processing has since been transformed by Transformer language models [59] that are pretrained on unlabelled text and then adapted to specific tasks [8,15,23], and by large language models that can follow instructions without task-specific training [5,58]. These models are now widely used in software engineering research [12,17], and NLP has entered learning analytics, including our own analyses of stakeholder feedback in higher education [11,46]. It is therefore timely to ask whether such models can take over the step that a-priori keyword models performed in earlier FLOSS studies: recognising, from the text of a message, where its author stands in their learning trajectory.
This paper addresses that question. Its aim is to establish how well different NLP paradigms recognise the learning phase of a mailing-list contribution and the mentoring behind it, and what the language of contributions reveals about how participants mature. We study the two largest communities whose entire contribution process runs through e-mail, Git and the Linux kernel. In both, a change is submitted to the list as a patch e-mail, discussed in replies and, once accepted, applied with git am. The resulting commit message is the accepted e-mail, and its trailer lines (Reviewed-by, Helped-by, Mentored-by, Suggested-by and others) record who helped. The two histories yield a corpus of 1432256 accepted patch e-mails by 32606 authors, together with a learning clock for each author (the index of each accepted patch) and a graph of helper–learner interactions. Four research questions structure the study.
RQ1 
How accurately can the text of an e-mail distinguish a newcomer’s contribution (accepted patches 1–3) from that of an established contributor (patch 51 onwards), and how do an a-priori lexicon, TF-IDF with logistic regression, a Transformer trained from scratch and a Transformer pretrained with masked language modelling compare on this task?
RQ2 
How accurately can the same paradigms recognise contributions that were produced with help or mentoring?
RQ3 
How does the language of a fixed cohort of contributors change as they progress from their first patches to later ones?
RQ4 
Is help received on the first patches, as recorded in trailers, associated with staying in the community?
The study contributes a method for turning patch-based mailing lists into a learning-analytics corpus with a learning clock and an interaction graph; a controlled, author-disjoint comparison of four NLP paradigms, including a Transformer pretrained on 210802 e-mails, with transfer between the two communities; cohort-level evidence on how the language of contributors matures and on how early help relates to retention; and a protocol, with prompts, for adding large language models to the comparison on identical test sets.
Section 2 reviews related work and states our expectations, Section 3 describes the corpus and the research design, Section 4 reports the findings, Section 5 discusses them together with the limitations of the study, and Section 6 concludes.

2. Background and Expectations

2.1. Mailing Lists as Learning Environments

Mailing lists carry much of the coordination, review and knowledge sharing of established FLOSS projects [16,24]. Knowledge-sharing activities on them have been quantified [56], the social networks they form have been mined [2], and patch review on lists has been characterised for Apache [52] and for the Linux kernel [18]. Research on newcomers documents how they join, how they are socialised, which barriers they face and what distinguishes those who stay [9,13,20,57,62], and it frequently draws on situated learning theory [21]. Participation in FLOSS has also been proposed as a form of e-learning [7].

2.2. Process-Oriented Learning Analytics on FLOSS Data

Our programme of work introduced a process view of this learning. It defined three learning phases, initiation, progression and maturation, together with their characteristic activities, first in an ontology [41] and then in an ASM model [40]. From these, a-priori learning models were derived whose activities are recognised in messages by keyword search and then mined as process models [25,43,44]. Complementary studies examined educational social networks [28], the temporal distribution of learning activities [29], decision points in learning processes [27], developer trails in code reviews [30] and process maps [45]. Parallel work applied process mining to MOOC learners [37,38,39], combined it with cluster analysis [35] and with machine learning on FLOSS mailing archives and logs [47,48], and argued for explainability in process-based educational analytics [33,36]. Overviews of this line of work and of repository analytics more generally are given in [31,32,34]. Learning analytics and educational data mining are reviewed more broadly in [4,53,55].

2.3. NLP Paradigms for Developer Communication

Classical text analysis weights terms with TF-IDF [54] and feeds them to linear classifiers, and it summarises collections with topic models such as LDA [3] or non-negative matrix factorisation [22]. Tools tuned to the language of developers exist, for example for sentiment [6]. Pretrained Transformers such as BERT [8], RoBERTa [23], CodeBERT [12] and sentence encoders [51] are adapted to a task by fine-tuning, and continued pretraining on in-domain text helps when that text differs from general web text [15]. Large language models add zero-shot and few-shot use [5,50] and have spread rapidly through software engineering research [17]. Their performance depends strongly on scale [19]. Within learning analytics, we have used NLP to analyse stakeholder perceptions in higher education [11,46]; the present study brings these methods to the mailing-list data of our earlier FLOSS work.

2.4. Expectations

Three expectations follow from this literature. First, because learned models can weight many thousands of cues while an a-priori lexicon uses a few dozen, every learned paradigm should recognise newcomer and mentored contributions better than the lexicon (E1). Second, because pretraining exposes a Transformer to the language of the community before it sees any labels, a pretrained Transformer should outperform one trained from scratch and, if pretraining is sufficient, also the classical TF-IDF model (E2). Third, if the phase model is sound, the language of contributors should move from initiation-phase towards maturation-phase cues as they gain experience (E3). For the association between early help and retention we formulate no directional expectation, since help may both support newcomers and mark those who struggle.

3. Research Design

3.1. Why Commit Histories Can Stand in for Mail Archives

In Git and in the Linux kernel, contributions are submitted exclusively as e-mails to the project lists. Maintainers apply the accepted version with git am, which copies the e-mail’s subject and body into the commit and preserves the block of trailer lines at its end. The commit history is therefore the accepted-patch subset of the mailing archive. This subset has two advantages over a raw archive for learning analytics. Authorship is resolved by the projects themselves, through sign-off chains and .mailmap files, and every message is known to have passed community review. Its disadvantage is that the review replies are missing; they survive only as trailers naming the people who reviewed, suggested or helped (Figure 1). We return to this limitation in Section 5.

3.2. Corpus Construction

We extracted all non-merge commits from the full histories of git/git and torvalds/linux (2005–2026), with identities resolved through the projects’ mailmap files. Each message was divided into subject, body and trailer block, the latter being the final block of Key: value lines. Identities were further merged by e-mail address and normalised full name [2,14] and then pseudonymised; people named in trailers were resolved to the same identities. We discarded 437 commits with implausible author dates. For every patch we recorded the author’s patch index, so that a patch with index i is the author’s i-th accepted e-mail; this index serves as the learning clock. We also recorded a directed edge from helper to author for every trailer that denotes help (Reviewed-by, Acked-by, Helped-by, Mentored-by, Suggested-by, Tested-by and similar). Before modelling, URLs, e-mail addresses and commit hashes were replaced by placeholders and identifiers were split at underscores and camel-case boundaries. Table 1 summarises the corpus.

3.3. Recognition Tasks

Newcomer recognition (RQ1).
A positive example is one of an author’s first three accepted patches, provided the author joined after the project’s founding period (June 2006 for Git and January 2007 for Linux), so that veterans whose earlier history predates the repository are not mislabelled. A negative example is a patch with index 51 or higher. Labels are derived from metadata only; the models see the subject and body with the trailer block removed. Negatives were subsampled to a ratio of two to one, and the Linux task was capped at 9000 training e-mails to keep training feasible.
Recognition of helped or mentored work (RQ2).
This task is defined for Git, where the Helped-by and Mentored-by conventions are established. Positive examples carry one of these trailers; negative examples are other Git e-mails written since 2008, when the convention came into use, again at a ratio of two to one. Because the trailers are removed, a model has to find the evidence of help in the prose itself or in what helped patches are typically about.
Author-disjoint partitions.
All partitions separate authors, in the proportion 70/10/20 based on a hash of the pseudonym. No author’s e-mails appear in both training and test data, which prevents a model from succeeding merely by recognising individual writing styles. Table 2 reports the resulting sample sizes.

3.4. Paradigms Compared

A-priori lexicon.
Following the a-priori learning models of our earlier work [43,44], in which messages were assigned to learning phases by keyword matching, we score an e-mail’s newcomer character as the rate of initiation-phase cues minus the rate of maturation-phase cues. Initiation cues express help seeking, tentativeness and a first-person novice stance (I, my, new, sorry, please, maybe, think, hope, tried, understand, typo, GSoC, the question mark, and others). Maturation cues express a collective voice, rationale and architectural or historical reference (we, should, instead, because, callers, semantics, regression, introduced, series, refactor, and others). For RQ2 the score is the rate of help and feedback vocabulary (thanks, suggested, helped, review, feedback, mentor, and others). The three lexicons contain 35, 30 and 22 terms and are released in full. The lexicon learns nothing from data; only its decision threshold is set on the validation set.
Classical NLP.
TF-IDF weights [54] over word unigrams and bigrams (sublinear term frequency, minimum document frequency of two, at most 10 5 features) are passed to an L 2 -regularised logistic regression [49], whose regularisation strength C ∈ { 0.25 , 1 , 4 , 16 } is chosen on validation AUROC. Non-negative matrix factorisation [22] with ten topics is used for the descriptive analysis.
Transformer trained from scratch.
A three-layer pre-LN Transformer encoder [59,61] with model width 96, four heads, a feed-forward width of 192, GELU activations, dropout of 0.1, learned positions and a maximum of 64 tokens reads the e-mail through a word-level vocabulary of 16000 types built from training-author text. Its hidden states are averaged and passed to a linear classifier. It is trained on the labelled task alone for four epochs with Adam, weight decay, warm-up and cosine decay, and the epoch with the best validation AUROC is retained.
Pretrained Transformer.
The same encoder is first pretrained by masked language modelling [8], masking 15% of the tokens with the usual 80/10/10 replacement scheme, for 4000 steps of 64 e-mails. The pretraining corpus consists of 210802 e-mails written by authors in the training and validation partitions of both projects, so that no text by a test author is seen at any stage. The pretrained encoder is then fine-tuned as above with a lower learning rate ( 5 · 10 − 4 ). To examine the effect of pretraining effort, we also fine-tune from the checkpoints after 500 and 1500 steps, and we fit a linear probe, a logistic regression on the frozen, averaged embeddings. With about 1.8 million parameters, the model follows the pretrain-then-fine-tune recipe [15] at a scale that can be trained on a CPU. It is not a substitute for billion-parameter language models, which we address through a separate protocol (Appendix A).

3.5. Evaluation Protocol

We report AUROC, which is independent of the decision threshold and of the two-to-one class ratio, together with AUPRC and with balanced accuracy and F1 at a threshold fixed on validation data. Neural models are trained with three random seeds, and we report means and standard deviations. Paired bootstrap intervals [10] compare seed-averaged scores between paradigms. Transfer between communities is assessed by training on Git and testing on Linux, and vice versa. To see what the models rely on, we inspect the largest logistic-regression coefficients and compute attention rollout [1] from the pooled representation of the Transformer, aggregated by word type.
For RQ3 we follow a fixed cohort, to avoid confounding change within authors with the selective departure of others: all Git authors, and a random sample of 1500 Linux authors, who joined after the founding period and accumulated at least 30 accepted patches. Linguistic rates are averaged per author within bins of the patch index and then over authors. For inference, each author’s patches 1–3 are compared with the same author’s patches 21–23 using the Wilcoxon signed-rank test [60], with a Holm correction over the fourteen tests. Using windows of equal size avoids comparing a single, often sparse message with a long-run average. For RQ4 we consider newcomers who joined after the founding period and before September 2023, so that three years of subsequent activity can be observed, and who reached their third patch. We compare the shares reaching 5, 10 and 20 patches between those whose first three patches carried a help trailer and those whose patches did not, using a χ 2 test and the odds ratio. This analysis describes an association and does not identify a causal effect.

4. Findings

4.1. RQ1 and RQ2: Recognising Newcomers and Mentored Work

Table 2 summarises the comparison. The text of an accepted patch clearly carries information about its author’s stage of learning, but the signal is moderate rather than strong. The best models recognise newcomers’ e-mails with an AUROC of 0.789 in Git and 0.768 in Linux, and work produced with help or mentoring with an AUROC of 0.688.
The a-priori lexicon is the weakest of the paradigms, with AUROCs of 0.644, 0.575 and 0.549 on the three tasks. TF-IDF with logistic regression improves on it by +0.146 (95% bootstrap interval [+0.122, +0.169]), +0.193 [+0.173, +0.213] and +0.139 [+0.100, +0.177], which confirms expectation E1. The lexicon is not mistaken about the direction of the evidence. Its central maturation cue, the collective we, is the strongest single indicator of an established author learned by TF-IDF in both projects. A few dozen hand-selected cues simply cannot match a learned weighting of a hundred thousand word sequences.
Expectation E2 is only partly borne out. Pretraining improves the Transformer over training from scratch on every task (newcomers in Git +0.012 [-0.001, +0.024], in Linux +0.004 [-0.006, +0.015]; help and mentoring +0.029 [+0.006, +0.051]), but the gain is significant only on the smallest task, with 4824 training e-mails, which is where self-supervision is expected to matter most [15]. The pretrained Transformer nevertheless remains below TF-IDF with logistic regression on all three tasks (Git -0.044 [-0.059, -0.031], Linux -0.020 [-0.031, -0.009], help and mentoring -0.038 [-0.067, -0.007]). A linear probe on the frozen encoder performs worse again (0.697, 0.701 and 0.624), so the pretrained representation does not by itself encode an author’s stage of learning. Figure 2 helps to explain this. The masked-language loss falls from 8.92 to 6.19 nats, corresponding to a perplexity of about 488, and is still decreasing slowly after 4000 steps, while downstream AUROC does not rise steadily with the number of pretraining steps. A model of 1.8 million parameters trained on a CPU budget is far from the scale at which pretrained language models typically overtake sparse lexical baselines [19].
What the models attend to.
The TF-IDF coefficients show which cues carry the signal. For newcomers in Git, the strongest indicators are fix, documentation, add, git, option, in, gitweb, email, typo, translation, p xnum, svn; for established authors they are we, batch, xnum xnum, git xnum, xnum, that, convert, let, do, it, pack, out. (Here xnum stands for any number, and “batch” and “git xnum” come from the maintainer’s release notes.) In Linux, the corresponding lists are add, fix, checkpatch, in, this patch, pl, support, checkpatch pl, fixed, added, issue, when and we, it, use, move, just, convert, struct, drop, switch, that, up, don. Two kinds of evidence are mixed in these lists. One concerns where newcomers work: documentation, typo corrections, gitweb and translations in Git, and in Linux the checkpatch clean-ups of staging drivers that have long served as the kernel’s entry route. The other concerns how they write. Newcomers describe their change in terms of the patch itself (“this patch”, “fixed”, “added”), whereas established authors write in the collective voice (“we”, “let’s”) and describe design operations (“convert”, “move”, “drop”, “switch”). Attention rollout of the pretrained Transformer falls mostly on words that name the area of the code (in Linux drm, the, staging, net, alsa, mtd, scsi, aso, iio, amd, perf, and; in Git makefile, http, gitweb, gui, documentation, builtin, remote, completion, the, diff, send, gitk), which suggests that the neural model relies more on where newcomers work than on how they write.
To separate the two sources of evidence, we removed the area prefix from subject lines (such as “drm/amd:” or “gitweb:”) together with administrative e-mails such as release notes, of which there were 2 in the Git test set. The AUROC of TF-IDF changes only from 0.789 to 0.771 in Git and from 0.768 to 0.770 in Linux, and the strongest indicators of an established Git author are then we, that, convert, it, let, do, out, object, let s, our. The linguistic signal therefore survives once the location of the work is hidden.
For help and mentoring, the positive indicators are dominated by the subjects of mentored projects, namely ref filter, subcommand, and, filter, is, command, reftable, introduce, repo, by, repo config, t reftable. These correspond to Google Summer of Code and Outreachy projects on ref-filter, reftable and repository configuration. The classifier thus learns what mentored work is about rather than recognising explicit acknowledgements, which rarely appear in the prose once the trailers record them.
Transfer between communities.
A TF-IDF model trained on Git newcomers reaches an AUROC of 0.659 on Linux, against 0.768 when trained on Linux itself; the pretrained Transformer reaches 0.669. In the opposite direction the values are 0.709 and 0.675, against 0.789 in-domain. The language of newcomers therefore transfers partially between the two communities, at a cost of 0.08 to 0.11 AUROC for TF-IDF. The pretrained Transformer transfers slightly better than TF-IDF from Git to Linux but worse in the other direction. A plausible explanation is that Linux e-mails make up most of the pretraining corpus, so that the encoder is better adapted to Linux text.

4.2. RQ3: How the Language of Contributors Matures

Figure 3 follows the same authors from their first patch onwards. Comparing each author’s patches 1–3 with the same author’s patches 21–23 shows how their writing changes. In the Linux cohort ( n = 1500 ), the collective we, our and us become more frequent, rising from 0.27 to 0.34 occurrences per 100 tokens ( p < 0.001 ). Rationale markers such as because, since, instead and so rise from 0.61 to 0.71 ( p < 0.001 ), and the first-person singular declines from 0.33 to 0.28 ( p = 0.018 ). Taken together, the lexicon score moves towards the maturation pole, from -0.44 to -0.61 ( p = 0.005 ). In the smaller Git cohort ( n = 163 ), e-mails become longer, from 62.08 to 74.63 words ( p = 0.005 ), and rationale markers increase from 0.56 to 0.80 ( p = 0.005 ); the remaining changes point the same way without reaching significance. After Holm correction for the fourteen tests, the Linux increases in collective voice and in rationale remain significant at the 5% level. Hedging and expressions of gratitude are rare in accepted patch e-mails and show no detectable change, which is unsurprising, since tentative and grateful language belongs mostly to review replies, which the corpus does not contain. The direction of these changes supports expectation E3.
Topics change less than style (Figure 4). In the Git cohort, the topic that gathers documentation and typo corrections (T10) falls from 11.3% of early e-mails to 8.2% of later ones, while the topic of references and packing internals (T7) rises from 5.9% to 8.3%. The movement from the periphery towards the core machinery of the system is what legitimate peripheral participation describes [21], and it matches the progression from initiation to maturation in the a-priori models [41,44].

4.3. RQ4: Early Help and Retention

The helper–author graph derived from trailers contains 8251 edges from 399 helpers to 852 authors in Git and 792547 edges from 13506 helpers to 18731 authors in Linux. Help is highly concentrated. The most active 1% of helpers account for 23.3% of all help mentions in Git and 42.0% in Linux, and the distribution of helper degree has a heavy tail (Figure 5, left).
Among newcomers who reached their third accepted patch, those whose first three patches carried a help trailer were not more likely to continue (Figure 5, right). In Git, 45.3% of helped newcomers ( n = 258 ) and 49.8% of the others ( n = 460 ) went on to reach ten patches (odds ratio 0.84, χ 2 test p = 0.288 ). In Linux the shares were 56.2% ( n = 7204 ) and 56.4% ( n = 4726 ), with an odds ratio of 0.99 ( p = 0.898 ). The absence of a positive association should not be read as evidence that help is ineffective. Trailers are given to newcomers whose first patches were difficult enough to need help, and in Git many mentored newcomers take part in programmes with a fixed end date. Neither form of selection can be removed with observational data.

5. Discussion

5.1. Recognising Learning Phases in Text

The comparison ranks the paradigms consistently across tasks and communities. Learned lexical models are best, followed by the small pretrained Transformer, the Transformer trained from scratch and, last, the a-priori lexicon. Three consequences follow for process-based learning analytics on mailing archives.
The a-priori phase models remain valuable as theory, but not as classifiers. Their central maturation cue, the collective we, is rediscovered independently by every learned model, and the cohort analysis confirms the drift from initiation to maturation that the models posit [41,44]. As instruments for labelling messages, however, fixed keyword lists are clearly outperformed. The text-to-activity step of process-mining pipelines on mailing archives [43,47,48] should therefore rely on learned weightings, with the lexicon retained as a theoretically motivated check on what the classifier learns.
Transformers pay off only with pretraining, and at the scale we could afford pretraining helped most where labelled data were scarcest. This argues for pretraining on a community’s own archives before fine-tuning, and it identifies billion-parameter language models as the natural next step of the comparison. The protocol in Appendix A prepares that step on identical test sets.
Finally, every model mixes evidence about where newcomers work with evidence about how they write. Used naively, a classifier of newcomer language would largely rediscover that newcomers fix typos and clean up staging drivers. The prefix-stripping analysis shows that a genuine linguistic signal remains once the location of the work is hidden, and any application that aims at learners’ writing should control for location in the same way.

5.2. Interpretability and Responsible Use

Attention rollout located the Transformer’s evidence mainly in words naming the area of the code, whereas the TF-IDF coefficients exposed both area and style. In this setting the more transparent model was also the more accurate one, a combination that Explainable Process Mining would favour [33,36]. Labelling contributors as newcomers or as having received help, on the basis of their e-mails, is a form of profiling. Such classifiers are appropriate for analytics at the level of the community, for instance to monitor whether entry routes such as documentation or staging drivers remain open, rather than for decisions about individual contributors.

5.3. Limitations

The corpus contains accepted patches only. Rejected patches, review replies and questions, where much of the asking and thanking of the initiation phase takes place [44], are absent. This explains the flat trajectories of hedging and gratitude and biases the study towards contributors who succeeded. Trailer conventions also differ between projects: Helped-by and Mentored-by are specific to Git, and the absence of a trailer does not prove the absence of help. Our lexicon operationalises the earlier phase models but does not reproduce their exact keyword lists. On the side of internal validity, author-disjoint partitions prevent the memorisation of individual styles, while the two-to-one subsampling and the Linux cap alter class prevalence; AUROC is insensitive to this, but AUPRC and F1 are not. The retention analysis is observational and confounded by the difficulty of early patches and by time-limited mentoring programmes. With regard to external validity, Git and Linux are mature, e-mail-centred projects, and communities organised around pull requests may write differently. Finally, our Transformers are small and were pretrained for 4000 steps on a CPU, so their results are a lower bound for the Transformer paradigm. Public large language models, on the other hand, have very probably seen these histories during training, so an evaluation of such models must control for contamination.

6. Conclusion

This paper set out to establish how well current NLP paradigms recognise the learning phase of a FLOSS mailing-list contribution and the mentoring behind it. Patch-based mailing lists turn commit histories into learning records with a clock and an interaction graph, and on 1432256 accepted e-mails from Git and Linux the text of a contribution identifies newcomers and mentored work with moderate accuracy. Learned lexical models outperform both the a-priori lexicon and small self-supervised Transformers, and pretraining narrows the gap without closing it. The language of contributors matures in the direction predicted by the a-priori models, towards a collective voice and explicit rationale, whereas early help recorded in trailers is not associated with retention. Future work will add review replies and rejected patches from the raw archives, run the prepared language-model protocol with contamination controls, and feed learned text-to-activity classifiers into process-mining analyses of learning [28,29,30] and into cluster-based learner profiling [35].

Supplementary Materials

The following supporting information can be downloaded at the website of this paper posted on Preprints.org.

Data Availability Statement

The scripts for corpus construction, the model implementation with its gradient check, the experiment and analysis scripts, the lexicons, the complete results and the language-model protocol are provided as supplementary material. Both repositories are public.

Appendix A. Protocol for Evaluating Large Language Models

The script llm_zero_shot.py scores each test e-mail (subject and body, trailers removed, truncated to 4,000 characters) with an instruction-tuned causal LLM under the model’s chat template. The score is the log-probability difference between the two answer tokens at the first generated position, which yields a continuous score for AUROC without sampling. Test sets are exported with experiments.py –export-tasks, so the LLM sees exactly the author-disjoint test e-mails of Table 2. The prompts are:
Preprints 236461 i001
Two cautions apply. First, public LLMs have almost certainly been trained on Git and Linux histories, so a strong zero-shot score may partly reflect memorisation of authors and patches rather than inference from text; a contamination check (e.g. scoring e-mails written after the model’s training cut-off separately) is required. Second, the prompts encode the same a-priori notion of “newcomer language” as the lexicon, and should be reported together with it.

References

  1. Abnar, S.; Zuidema, W. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020), 2020; pp. 4190–4197. [Google Scholar]
  2. Bird, C.; Gourley, A.; Devanbu, P.; Gertz, M.; Swaminathan, A. Mining email social networks. In Proceedings of the 2006 International Workshop on Mining Software Repositories (MSR 2006), 2006; pp. 137–143. [Google Scholar]
  3. Blei, D.M.; Ng, A.Y.; Jordan, M.I. Latent Dirichlet allocation. J. Mach. Learn. Res. 2003, 3, 993–1022. [Google Scholar]
  4. Bogarín, A.; Cerezo, R.; Romero, C. A survey on educational process mining. WIREs Data Min. Knowl. Discov. 2018, 8(1), e1230. [Google Scholar] [CrossRef]
  5. Brown, T.B.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.; et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems; 2020; Volume 33, NeurIPS 2020, pp. 1877–1901. [Google Scholar]
  6. Calefato, F.; Lanubile, F.; Maiorano, F.; Novielli, N. Sentiment polarity detection for software development. Empir. Softw. Eng. 2018, 23(3), 1352–1382. [Google Scholar] [CrossRef]
  7. Cerone, A.; Sowe, S.K. Using free/libre open source software projects as e-learning tools. Electronic Communications of the EASST, 2010; 33. [Google Scholar]
  8. Devlin, J.; Chang, M.W.; Lee, K.; Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACL-HLT 2019, 2019; pp. 4171–4186. [Google Scholar]
  9. Ducheneaut, N. Socialization in an open source software community: A socio-technical analysis. Comput. Support. Coop. Work 2005, 14(4), 323–368. [Google Scholar] [CrossRef]
  10. Efron, B.; Tibshirani, R. J.: An Introduction to the Bootstrap; Chapman & Hall, 1993. [Google Scholar]
  11. El Nekiti, A.; Mukala, P.; Ullah, O.; Yatsyshyn, D. Advanced learning analytics with AI: Analyzing stakeholder perceptions of feedback, assessment, engagement, and faculty development in MENA higher education using NLP. Proceedings of Eighth International Conference on Information System Design and Intelligent Applications, 2025; Springer Nature Singapore; pp. 117–129. [Google Scholar]
  12. Feng, Z.; Guo, D.; Tang, D.; Duan, N.; Feng, X.; Gong, M.; Shou, L.; Qin, B.; Liu, T.; Jiang, D.; Zhou, M. CodeBERT: A pre-trained model for programming and natural languages. Find. Assoc. Comput. Linguist. EMNLP 2020 2020, 1536–1547. [Google Scholar] [CrossRef]
  13. Gerosa, M.; Wiese, I.; Trinkenreich, B.; Link, G.; Robles, G.; Treude, C.; Steinmacher, I.; Sarma, A. The shifting sands of motivation: Revisiting what drives contributors in open source. In Proceedings of the 43rd International Conference on Software Engineering (ICSE 2021), 2021; pp. 1046–1058. [Google Scholar]
  14. Goeminne, M.; Mens, T. A comparison of identity merge algorithms for software repositories. Sci. Comput. Program. 2013, 78(8), 971–986. [Google Scholar] [CrossRef]
  15. Gururangan, S.; Marasović, A.; Swayamdipta, S.; Lo, K.; Beltagy, I.; Downey, D.; Smith, N.A. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020), 2020; pp. 8342–8360. [Google Scholar]
  16. Guzzi, A.; Bacchelli, A.; Lanza, M.; Pinzger, M.; van Deursen, A. Communication in open source software development mailing lists. In Proceedings of the 10th Working Conference on Mining Software Repositories (MSR 2013), 2013; pp. 277–286. [Google Scholar]
  17. Hou, X.; Zhao, Y.; Liu, Y.; Yang, Z.; Wang, K.; Li, L.; Luo, X.; Lo, D.; Grundy, J.; Wang, H. Large language models for software engineering: A systematic literature review. ACM Trans. Softw. Eng. Methodol. 2024, 33(8), 220:1–220:79. [Google Scholar] [CrossRef]
  18. Jiang, Y.; Adams, B.; German, D.M. Will my patch make it? and how fast? case study on the Linux kernel. In Proceedings of the 10th Working Conference on Mining Software Repositories (MSR 2013), 2013; pp. 101–110. [Google Scholar]
  19. Kaplan, J.; McCandlish, S.; Henighan, T.; Brown, T.B.; Chess, B.; Child, R.; Gray, S.; Radford, A.; Wu, J.; Amodei, D. Scaling laws for neural language models. arXiv 2020, arXiv:2001.08361. [Google Scholar]
  20. von Krogh, G.; Spaeth, S.; Lakhani, K.R. Community, joining, and specialization in open source software innovation: A case study. Res. Policy 2003, 32(7), 1217–1241. [Google Scholar] [CrossRef]
  21. Lave, J.; Wenger, E. Situated Learning: Legitimate Peripheral Participation; Cambridge University Press, 1991. [Google Scholar]
  22. Lee, D.D.; Seung, H.S. Learning the parts of objects by non-negative matrix factorization. Nature 1999, 401, 788–791. [Google Scholar] [CrossRef] [PubMed]
  23. Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; Stoyanov, V. RoBERTa: A robustly optimized BERT pretraining approach. arXiv 2019, arXiv:1907.11692. [Google Scholar]
  24. Mockus, A.; Fielding, R.T.; Herbsleb, J.D. Two case studies of open source software development: Apache and Mozilla. ACM Trans. Softw. Eng. Methodol. 2002, 11(3), 309–346. [Google Scholar] [CrossRef]
  25. Mukala, M.P. Process Models for Learning Patterns in FLOSS Repositories. Ph.D. thesis, University of Pisa, 2015. [Google Scholar]
  26. Mukala, P. Foundational imperatives for process mining learning patterns in FLOSS repositories. Int. J. Inf. Netw. Secur. (IJINS) 2014, 3(6), 17–32. [Google Scholar]
  27. Mukala, P. Decision point analysis on learning process models in FLOSS mailing archives. In Tech. rep.; Università di Pisa, 2016. [Google Scholar]
  28. Mukala, P. Mining educational social network structures from FLOSS repositories. In Tech. rep.; Università di Pisa, 2016. [Google Scholar]
  29. Mukala, P. A temporal visual distribution of learning activities in FLOSS repositories. In Tech. rep.; Università di Pisa, 2016. [Google Scholar]
  30. Mukala, P. Mining reviews in open source code for developers trail: A process mining approach. arXiv 2023, arXiv:2308.00686. [Google Scholar]
  31. Mukala, P. SSRN preprint 5391607; Exploring data analytics on open-source software repositories: A review of development activities and tools for mining logs. 2025.
  32. Mukala, P. Exploring data analytics on open-source software repositories: Insights into development activities and tools for mining logs. TechRxiv 2025. [Google Scholar] [CrossRef] [PubMed]
  33. Mukala, P. A framework for explainable process mining (XPM): Steps toward adopting AI transparency principles applied to process mining. Proceedings of International Conference on Recent Advancements in Artificial Intelligence, 2025; Springer Nature Singapore; pp. 1–14. [Google Scholar]
  34. Mukala, P. Leveraging process mining, learning analytics, and AI to explore learning behaviors in MOOCs and FLOSS communities: A comprehensive mise-au-point of key findings. SSRN 2025, 2025. [Google Scholar]
  35. Mukala, P. Unlocking learning patterns and personal constructivism in MOOCs and FLOSS: A cutting-edge big data analytics approach with cluster analysis and process mining. TechRxiv 2025. [Google Scholar] [CrossRef] [PubMed]
  36. Mukala, P. Unveiling the synergy of process mining, explainable AI, and learning analytics in advancing educational data interpretability: Paving the way for a new era in educational analytics. SSRN 2025, 2025. [Google Scholar]
  37. Mukala, P.; Buijs, J.C.A.M.; van der Aalst, W.M.P. BPM Center Report BPM-15-09; Uncovering learning patterns in a MOOC through conformance alignments. 2015; Available online: BPMcenter.org.
  38. Mukala, P.; Buijs, J.C.A.M.; Leemans, M.; van der Aalst, W.M.P. BPM Center Report BPM-15-10; Exploring students’ learning behaviour in MOOCs using process mining techniques. BPMcenter.org / Eindhoven University of Technology, 2015.
  39. Mukala, P.; Buijs, J.C.A.M.; Leemans, M.; van der Aalst, W.M.P. Learning analytics on Coursera event data: A process mining approach. Proc. 5th Int. Symp. Data-Driven Process Discov. Anal. (SIMPDA 2015) CEUR Workshop Proc. 2015, vol. 1527, 18–32. [Google Scholar]
  40. Mukala, P.; Cerone, A.; Turini, F. An abstract state machine (ASM) representation of learning process in FLOSS communities. In Software Engineering and Formal Methods – SEFM 2014 Collocated Workshops. Lecture Notes in Computer Science; Springer: Cham, 2014; vol. 8938, pp. 227–242. [Google Scholar]
  41. Mukala, P.; Cerone, A.; Turini, F. OntoLiFLOSS: Ontology for learning processes in FLOSS communities. In Software Engineering and Formal Methods – SEFM 2014 Collocated Workshops. Lecture Notes in Computer Science; Springer: Cham, 2014; vol. 8938, pp. 164–181. [Google Scholar]
  42. Mukala, P.; Cerone, A.; Turini, F. Process mining event logs from FLOSS data: State of the art and perspectives. In Software Engineering and Formal Methods – SEFM 2014 Collocated Workshops. Lecture Notes in Computer Science; Springer: Cham, 2014; vol. 8938, pp. 182–198. [Google Scholar]
  43. Mukala, P.; Cerone, A.; Turini, F. Mining learning processes from FLOSS mailing archives. In Open and Big Data Management and Innovation – 14th IFIP WG 6.11 Conference on e-Business, e-Services, and e-Society (I3E 2015) Lecture Notes in Computer Science; Springer: Cham, 2015; vol. 9373, pp. 287–298. [Google Scholar]
  44. Mukala, P.; Cerone, A.; Turini, F. An empirical verification of a-priori learning models on mailing archives in the context of online learning activities of participants in free/libre open source software (FLOSS) communities. Educ. Inf. Technol. 2017, 22(6), 3207–3229. [Google Scholar] [CrossRef]
  45. Mukala, P.; Cerone, A.; Turini, F. An exploration of learning processes as process maps in FLOSS repositories. arXiv 2023, arXiv:2307.07841. [Google Scholar]
  46. Mukala, P.; El Nekiti, A.; Ullah, O.; Yatsyshyn, D. AI-based learning analytics: Evaluating MENA higher education stakeholders’ views on curriculum, resources, class size, learning environment, and support services using NLP. Proceedings of Eighth International Conference on Information System Design and Intelligent Applications, 2025; Springer Nature Singapore; pp. 103–115. [Google Scholar]
  47. Mukala, P.; Ullah, O. Enhancing learning analytics in open-source software mailing archives using machine learning and process discovery techniques. In Proceedings of the 14th International Conference on Industrial Engineering and Operations Management, 2024; IEOM Society. [Google Scholar] [CrossRef]
  48. Mukala, P.; Ullah, O. Improving learning analytics from open-source software data logs using machine learning and process mining techniques. Proceedings of Eighth International Conference on Information System Design and Intelligent Applications, 2025; Springer Nature Singapore; pp. 57–72. [Google Scholar]
  49. Pedregosa, F.; Varoquaux, G.; Gramfort, A.; et al. Scikit-learn: Machine learning in Python. J. Mach. Learn. Res. 2011, 12, 2825–2830. [Google Scholar]
  50. Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; Sutskever, I. Language models are unsupervised multitask learners. OpenAI technical report, 2019. [Google Scholar]
  51. Reimers, N.; Gurevych, I. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. Proceedings of EMNLP-IJCNLP 2019, 2019; pp. 3982–3992. [Google Scholar]
  52. Rigby, P.C.; German, D.M.; Storey, M.A. Open source software peer review practices: A case study of the Apache server. In Proceedings of the 30th International Conference on Software Engineering (ICSE 2008), 2008; pp. 541–550. [Google Scholar]
  53. Romero, C.; Ventura, S. Educational data mining: A review of the state of the art. IEEE Trans. Syst. Man. Cybern. Part C 2010, 40(6), 601–618. [Google Scholar] [CrossRef]
  54. Salton, G.; Buckley, C. Term-weighting approaches in automatic text retrieval. Inf. Process. Manag. 1988, 24(5), 513–523. [Google Scholar] [CrossRef]
  55. Siemens, G.; Baker, R.S. J.d.: Learning analytics and educational data mining: Towards communication and collaboration. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK 2012), 2012; pp. 252–254. [Google Scholar]
  56. Sowe, S.K.; Stamelos, I.; Angelis, L. Understanding knowledge sharing activities in free/open source software projects: An empirical study. J. Syst. Softw. 2008, 81(3), 431–446. [Google Scholar] [CrossRef]
  57. Steinmacher, I.; Graciotto Silva, M.A.; Gerosa, M.A.; Redmiles, D.F. A systematic literature review on the barriers faced by newcomers to open source software projects. Inf. Softw. Technol. 2015, 59, 67–85. [Google Scholar] [CrossRef]
  58. Touvron, H.; Martin, L.; Stone, K.; et al. Llama 2: Open foundation and fine-tuned chat models. arXiv 2023, arXiv:2307.09288. [Google Scholar]
  59. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser; Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems; 2017; Volume 30, NeurIPS 2017, pp. 5998–6008. [Google Scholar]
  60. Wilcoxon, F. Individual comparisons by ranking methods. Biom. Bull. 1945, 1(6), 80–83. [Google Scholar] [CrossRef]
  61. Xiong, R.; Yang, Y.; He, D.; Zheng, K.; Zheng, S.; Xing, C.; Zhang, H.; Lan, Y.; Wang, L.; Liu, T.Y. On layer normalization in the transformer architecture. Proc. 37th Int. Conf. Mach. Learn. (ICML 2020) PMLR 2020, vol. 119, 10524–10533. [Google Scholar]
  62. Zhou, M.; Mockus, A. What make long term contributors: Willingness and opportunity in OSS community. In Proceedings of the 34th International Conference on Software Engineering (ICSE 2012), 2012; pp. 518–528. [Google Scholar]
Preprints 236461 g001
Figure 1. From mailing-list thread to corpus (top) and the paradigms compared (bottom). Review replies are not part of the corpus; their outcome is recorded in the trailers of the accepted e-mail.
Figure 1. From mailing-list thread to corpus (top) and the paradigms compared (bottom). Review replies are not part of the corpus; their outcome is recorded in the trailers of the accepted e-mail.
Preprints 236461 g001
Preprints 236461 g002
Figure 2. Left: masked-language-modelling loss (means over 100 steps). Right: test AUROC after fine-tuning from successive pretraining checkpoints (0 denotes random initialisation). Points at 0 and 4000 steps average three seeds, the intermediate checkpoints use one seed; dashed lines give TF-IDF with logistic regression on the same task.
Figure 2. Left: masked-language-modelling loss (means over 100 steps). Right: test AUROC after fine-tuning from successive pretraining checkpoints (0 denotes random initialisation). Points at 0 and 4000 steps average three seeds, the intermediate checkpoints use one seed; dashed lines give TF-IDF with logistic regression on the same task.
Preprints 236461 g002
Preprints 236461 g003
Figure 3. Linguistic trajectories of a fixed cohort of authors with at least 30 accepted patches (Git n = 163 ; Linux sample n = 1500 ). Per-author means within bins of the patch index, averaged over authors (±1.96 standard errors).
Figure 3. Linguistic trajectories of a fixed cohort of authors with at least 30 accepted patches (Git n = 163 ; Linux sample n = 1500 ). Per-author means within bins of the patch index, averaged over authors (±1.96 standard errors).
Preprints 236461 g003
Preprints 236461 g004
Figure 4. Topics of the Git cohort’s e-mails (11530 e-mails, patches 1–100), estimated by non-negative matrix factorisation, and their share by bin of the patch index (normalised per column, in %).
Figure 4. Topics of the Git cohort’s e-mails (11530 e-mails, patches 1–100), estimated by non-negative matrix factorisation, and their share by bin of the patch index (normalised per column, in %).
Preprints 236461 g004
Preprints 236461 g005
Figure 5. Left: complementary cumulative distribution of helper out-degree in the trailer graph. Right: share of newcomers who reached their third patch and went on to reach 5, 10 and 20 accepted patches, by whether their first three patches carried a help trailer.
Figure 5. Left: complementary cumulative distribution of helper out-degree in the trailer graph. Right: share of newcomers who reached their third patch and went on to reach 5, 10 and 20 accepted patches, by whether their first three patches carried a help trailer.
Preprints 236461 g005
Table 1. Corpus statistics. Patch and author counts are after cleaning; trailer counts and median lengths were computed before discarding the 437 commits (427 Linux, 10 Git) with implausible dates.
Table 1. Corpus statistics. Patch and author counts are after cleaning; trailer counts and median lengths were computed before discarding the 437 commits (427 Linux, 10 Git) with implausible dates.
Git Linux
accepted patch e-mails 60998 1371258
distinct authors 2348 30258
median body length (words) 42 31
Reviewed-by / Acked-by trailers 2249 / 2458 456631 / 231134
Helped-by / Mentored-by trailers 2194 / 873 – / –
Suggested-by / Reported-by trailers 538 / 1463 23733 / 71828
patches with ≥1 help trailer 8289 588507
Table 2. Test performance on author-disjoint partitions. Neural models give mean ± s.d. over three seeds; the best value per column is in bold. A classifier using only the class prior has an AUROC of 0.5.
Table 2. Test performance on author-disjoint partitions. Neural models give mean ± s.d. over three seeds; the best value per column is in bold. A classifier using only the class prior has an AUROC of 0.5.
Newcomer (Git) Newcomer (Linux) Help/mentoring (Git)
Paradigm AUROC Bal. acc. AUROC Bal. acc. AUROC Bal. acc.
(a) A-priori lexicon (zero-shot) 0.644 0.619 0.575 0.556 0.549 0.500
(b) TF-IDF + logistic regression 0.789 0.693 0.768 0.685 0.688 0.607
(c) Transformer, from scratch 0.726±0.004 0.644±0.007 0.729±0.001 0.660±0.005 0.613±0.012 0.568±0.010
(d) Transformer, MLM-pretrained (4,000 steps) 0.740±0.001 0.658±0.009 0.737±0.004 0.667±0.004 0.643±0.002 0.600±0.012
(d’) MLM encoder, frozen + linear probe 0.697 0.625 0.701 0.641 0.624 0.575
test e-mails (positives) 2376 (792) 3999 (1333) 879 (293)
train e-mails 8811 9000 4824
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.