Stylometric Fingerprinting with Contextual Anomaly Detection for Sentence-Level AI Authorship Detection

1. Introduction

The advent of large language models (LLMs) like GPT-3 has revolutionized content generation, enabling students to produce high-quality text with minimal effort [1]. However, this raises concerns about academic integrity, as AI-generated text can be seamlessly integrated into student writing, often at the sentence level, making detection challenging. Traditional tools like Turnitin and GPTZero focus on document-level analysis and rely on computational resources, which may not be accessible to all students, especially under constraints like the "Student+Student" rule that prohibits LLM usage. Moreover, these tools often lack the granularity to detect AI authorship at the sentence level, where students might intersperse AI-generated sentences with their own writing.

This paper introduces a novel approach, Stylometric Fingerprinting with Contextual Anomaly Detection, to address this gap. Our method leverages manual stylometric analysis to create a "fingerprint" of a student’s writing style and uses contextual coherence checks to identify AI-generated sentences as anomalies. Unlike existing models, our approach is entirely manual, requiring only basic statistical tools and search engines, making it feasible for student implementation. We compare our method to Turnitin, GPTZero, and Moss, demonstrating its uniqueness in combining stylometric and contextual analysis at the sentence level. The paper is structured as follows: Section 2 reviews related work, Section 3 details the proposed method, Section 4 compares it with existing models, Section 5 discusses its uniqueness, and Section 6 concludes with future directions.

2. Related Work

The detection of AI-generated text in academic writing has gained attention with the rise of LLMs. Turnitin, a widely used plagiarism detection tool, has integrated AI detection capabilities, claiming 98% accuracy in identifying AI-generated content [4]. However, it struggles with false positives (4% at the sentence level) and often fails to detect modified AI text, as it primarily focuses on document-level patterns. GPTZero, another tool, uses metrics like perplexity and burstiness to detect AI text, achieving 90% accuracy [5]. While effective for whole documents, GPTZero requires computational resources and overlooks contextual coherence, limiting its applicability for sentence-level analysis.

Moss (Measure of Software Similarity), developed by Schleimer et al. [2], is a tool designed for detecting plagiarism in programming code by comparing structural similarities across submissions. It is widely used in computer science education but is irrelevant for natural language AI detection, as its algorithms are tailored for code syntax rather than linguistic patterns. Stylometric analysis, as explored by Brennan et al. [3], has been used for authorship attribution by analyzing features like word choice and sentence structure. However, these methods typically target document-level authorship and do not address the specific challenge of AI-generated text at the sentence level, nor do they incorporate contextual coherence as a detection mechanism.

Our approach builds on stylometric principles but extends them with contextual anomaly detection, focusing on sentence-level granularity and manual implementation. This combination sets our method apart from existing tools, offering a human-centric solution for academic integrity.

3. Proposed Method

3.1. Overview

The Stylometric Fingerprinting with Contextual Anomaly Detection method aims to identify whether each sentence in a student’s writing report is authored by the student or an AI. It creates a stylometric "fingerprint" of the student’s writing style using linguistic features and detects AI-generated sentences as anomalies through statistical and contextual analysis.

3.2. Algorithm

• Collect a Student Writing Sample: Gather a verified sample of the student’s writing (e.g., prior essays) to establish a baseline style. This sample should be at least 500 words to ensure sufficient data for analysis.
• Extract Stylometric Features: For each sentence in the sample, compute the following features:

  ◦

  Average word length (in characters).

  ◦

  Sentence complexity (e.g., number of clauses or depth of dependency tree, approximated manually).

  ◦

  Vocabulary richness (e.g., type-token ratio, calculated as the number of unique words divided by total words).

  ◦

  Function word usage (e.g., frequency of prepositions, conjunctions).

3.

Build a Student Style Fingerprint: Aggregate these features into a statistical profile by calculating the mean and variance of each feature across all sentences in the sample. This profile represents the student’s typical writing style.

4.

Analyze the Target Report: For each sentence in the student’s report, extract the same stylometric features as in Step 2.

5.

Contextual Anomaly Detection: Compare each sentence’s features to the student’s style fingerprint using a distance metric, such as the Mahalanobis distance, which accounts for feature correlations:

6.

$D=\sqrt{{\left(x-\mathsf{\mu }\right)}^T{S}^{-1}\left(x-\mathsf{\mu }\right)}$

7.

where x is the feature vector of the sentence, μ is the mean feature vector from the fingerprint, and S is the covariance matrix. Flag sentences as anomalies if their distance exceeds a threshold (e.g., 2 standard deviations).

8.

Incorporate Contextual Coherence: Manually score the semantic and syntactic coherence between consecutive sentences using a rubric:

◦

Topic continuity: Does the sentence logically follow the previous one in terms of subject matter? (Score: 0-2)

◦

Pronoun consistency: Are pronouns used consistently across sentences? (Score: 0-2)AI-generated sentences often disrupt coherence due to differing generation objectives, scoring lower (e.g., <2 total).

9.

Label Sentences: Classify each sentence as "student-authored" if it aligns with the fingerprint (low distance) and maintains coherence (high score), or "AI-generated" if it is an outlier or breaks contextual flow.

10.

Output: Provide a report annotating each sentence with its likely authorship (student or AI).

3.3. Implementation Details

The method can be implemented using basic tools like spreadsheets for feature computation (e.g., Excel for calculating means and variances) and manual scoring for coherence. Students can use search engines to research stylometric features (e.g., definitions of type-token ratio) and statistical methods (e.g., Mahalanobis distance formula). No computational tools beyond a calculator are required, ensuring accessibility.

4. Comparison with Existing Models

4.1. Turnitin

Turnitin, a widely used tool in academia, has integrated AI detection capabilities, claiming 98% accuracy in identifying AI-generated content [4]. However, its focus is on document-level analysis, and it struggles with sentence-level granularity, reporting a 4% false positive rate at this level. Additionally, Turnitin often fails to detect modified AI text, as its algorithms rely on pattern matching against a database of known AI outputs, which can be easily circumvented by paraphrasing.

4.2. GPTZero

GPTZero, designed specifically for AI detection, uses metrics like perplexity (a measure of text predictability) and burstiness (variation in sentence length) to identify AI-generated text, achieving 90% accuracy [5]. While effective for whole documents, GPTZero requires computational resources to process text and calculate these metrics. Moreover, it overlooks contextual coherence, a key indicator of AI authorship at the sentence level, where abrupt shifts in style or topic may occur.

4.3. Moss

Moss (Measure of Software Similarity), developed by Schleimer et al. [2], is a tool for detecting plagiarism in programming code by comparing structural similarities across submissions. It is widely used in computer science education but is irrelevant for natural language AI detection, as its algorithms are tailored for code syntax rather than linguistic patterns. Moss cannot address the challenge of distinguishing AI-generated sentences in textual content, making it inapplicable to this task.

4.4. Comparison Summary

Our method outperforms these tools in several ways:

• Sentence-Level Granularity: Unlike Turnitin and GPTZero, which focus on document-level analysis, our approach detects AI authorship at the sentence level, addressing the specific challenge of mixed authorship in student writing.
• Manual Implementation: Our method requires no computational tools, unlike GPTZero, which relies on digital processing, or Turnitin, which requires a subscription-based platform.
• Contextual Coherence: By incorporating coherence checks, our method captures disruptions in semantic and syntactic flow, a feature absents in existing tools.
• Relevance: Unlike Moss, which is designed for code, our method is tailored for natural language, making it directly applicable to writing reports.

5. Uniqueness of the Approach

The Stylometric Fingerprinting with Contextual Anomaly Detection approach is unique in several aspects:

• Sentence-Level Focus: Existing tools like Turnitin, GPTZero, and Moss primarily operate at the document level or focus on code similarity, lacking the granularity to detect AI authorship sentence by sentence. Our method addresses this gap by analyzing each sentence individually, making it ideal for identifying mixed authorship in student writing.
• Integration of Stylometric and Contextual Analysis: While stylometric analysis has been used for authorship attribution [3], our approach uniquely combines it with contextual anomaly detection. This dual mechanism leverages both linguistic patterns (e.g., word length, vocabulary richness) and semantic coherence (e.g., topic continuity), providing a more robust detection framework.
• Manual and Accessible Design: Unlike existing models that require computational resources or labeled datasets, our method is entirely manual, relying on basic statistical tools (e.g., spreadsheets) and human judgment for coherence scoring.
• Human-Centric Interpretation: The inclusion of contextual coherence checks, scored manually, adds a human-centric layer to the analysis, allowing for interpretability and adaptability in academic settings where automated tools may not be feasible.

This combination of features sets our approach apart, offering a novel solution that balances precision, accessibility, and interpretability for detecting AI authorship in academic writing.

6. Conclusions and Future Work

The Stylometric Fingerprinting with Contextual Anomaly Detection method provides a novel, feasible solution for detecting AI authorship in student writing at the sentence level. Its human-centric design, requiring no LLMs or advanced tools, ensures accessibility for students while maintaining academic integrity. By combining stylometric analysis with contextual coherence checks, the method addresses the limitations of existing tools like Turnitin, GPTZero, and Moss, offering a unique approach tailored for mixed authorship detection.

Future work could explore semi-automated implementations of this method, such as developing a simple script to compute stylometric features, while still adhering to manual constraints. Additionally, expanding the feature set to include more advanced linguistic markers (e.g., syntactic patterns, sentiment consistency) could improve detection accuracy.