A Dataset for Evaluating Large Language Models in Web Accessibility Auditing and Remediation

1. Summary

Web accessibility is a fundamental pillar of an inclusive digital society [1]. It ensures that individuals with disabilities can perceive, understand, navigate, and interact with online content effectively, thus supporting equitable participation in economic, educational, and civic life [2]. With the increasing reliance on the Web as a primary interface for services and information, accessibility is no longer a niche concern but a mainstream requirement [3]. In particular, adherence to global standards such as the Web Content Accessibility Guidelines (WCAG) has become a legal and ethical imperative across many sectors and jurisdictions [4].

Despite the formalization of accessibility standards and the proliferation of automated tools, such as WAVE [5], Axe [6], and Lighthouse [7], web accessibility remains a persistent challenge [8]. Existing tools are often limited to detecting syntactic issues and fail to capture semantic or context-dependent violations [9], especially in modern web applications that rely heavily on JavaScript frameworks, dynamic content rendering, and component-based architectures [10]. These limitations contribute to widespread accessibility gaps, particularly in websites developed without expert oversight or adequate accessibility training [11].

Artificial Intelligence (AI) and Machine Learning (ML) have recently emerged as promising avenues for addressing these limitations by enabling more nuanced and scalable evaluations of web accessibility [12]. For example, Makati [13] presented DocTTTTTQuery, a method that employs a sequence-to-sequence model (i.e., T5) to generate a set of anticipated queries for a given document. These generated queries are then appended to the original document, creating an enhanced version. The underlying principle is that by enriching the document with potential queries, the overall retrieval performance is improved. The author demonstrated that using these enhanced documents leads to superior retrieval results. Instead, Ara and Sik-Lanyi [14] evaluated website accessibility using a Decision Tree. The model is trained on data generated by the Mauve tool, which assesses websites against various checkpoints from the WCAG 2.1. The authors aimed to identify accessibility barriers and demonstrate the need for stronger web accessibility policies. Their experiments, validated through a confusion matrix and classification report, found that none of the tested webpages were free from accessibility errors, highlighting a significant need for improved directives to enhance social inclusiveness.

In such a context, Large Language Models (LLMs) such as GPT-4, Claude, and LLaMA have shown considerable potential due to their ability to interpret source code, understand natural language requirements, and reason about content structure and intent [15]. Rather than relying solely on rigid rule-based systems, LLMs can offer more flexible and context-aware assessments. This represents a paradigm shift in accessibility validation: from conventional DOM-level checks and static analyzers to AI-powered agents that can analyze raw HTML/CSS/JS code, interpret it semantically, and suggest or apply fixes [16]. Several recent studies illustrate this emerging capability. For instance, Othman et al. [17] investigated how ChatGPT can remediate WCAG 2.1 violations identified by the WAVE tool, showing significant improvement in compliance through automated fixes. Similarly, Delnevo et al. [18] assessed LLMs’ ability to validate and correct HTML forms and tables, finding that ChatGPT could successfully detect and fix structural issues such as missing labels and incorrect semantic tags. López-Gil and Pereira [19] further explored LLMs for automating manual WCAG criteria evaluation, such as interpreting link purposes or identifying language mismatches, reaching detection accuracies beyond 85%. However, these works also highlight the limitations of current models, especially in dealing with missing structural elements and generating consistent remediations.

The dataset described in this article was created to systematically investigate and benchmark the capabilities of LLMs in detecting and remediating accessibility issues directly at the source code level. It is motivated by the need for a structured, reproducible, and scalable resource that allows researchers to explore how LLMs perform in realistic, code-centric accessibility validation scenarios. The dataset includes a diverse set of code snippets, some intentionally injected with accessibility violations, and others designed to meet best practices.

The dataset is structured into two primary directories, each targeting distinct facets of accessibility evaluation within modern web development. The first directory, titled “Dynamic Generated Content”, comprises a collection of code snippets developed using four widely adopted frontend and backend technologies, Angular, React, Vue, and PHP. Each snippet contains two functionally equivalent versions of a tabular component: one that adheres to established WCAG and another that deliberately incorporates common accessibility violations. These violations reflect typical real-world pitfalls, such as missing ARIA attributes, poor semantic structure, or inaccessible keyboard interactions, thereby simulating realistic development scenarios where accessibility might be overlooked.

The second directory, named “Vue Table Components”, includes 25 distinct Vue-based implementations of table components. These examples were carefully selected and constructed to represent a diverse range of development patterns and accessibility practices. They vary in several dimensions, including the programming paradigm (with examples written using both the Composition API and the Options API), the degree and type of accessibility support provided, and the complexity of internal structure (e.g., use of slots, scoped components, or dynamic data binding). This portion of the dataset is particularly valuable for studying how different architectural choices in Vue development influence the interpretability and accessibility of the resulting components.

Together, these two parts provide a rich and heterogeneous source of annotated web components for training, evaluating, and benchmarking LLMs and accessibility auditing tools, particularly in contexts involving dynamically generated content, which remains a significant challenge for traditional static analysis methods.

This dataset supports a broader line of research aimed at exploring the responsible integration of AI into web development workflows, with an emphasis on equity, sustainability, and inclusivity. The dataset is intended to serve multiple audiences: AI researchers developing accessibility-aware models, Human-Computer Interaction (HCI) practitioners evaluating usability, accessibility auditors seeking AI assistance, and educators looking to train students in inclusive web development. By releasing this dataset publicly, we aim to facilitate a deeper understanding of the strengths and limitations of LLMs in real-world accessibility use cases.

The initial results based on this dataset have been published in [20], where GPT-4o mini, o3-mini, and Gemini 2.0 Flash were employed to systematically evaluate the capabilities of LLMs to identify and assess accessibility issues

2. Data Description

The dataset is structured into two primary directories, each targeting distinct facets of accessibility evaluation within modern web development. The first directory, titled “Dynamic Generated Content”, comprises a collection of code snippets developed using four widely adopted frontend and backend technologies, Angular, React, Vue, and PHP. Each snippet contains two functionally equivalent versions of a tabular component: one that adheres to established WCAG and another that deliberately incorporates common accessibility violations. These violations reflect typical real-world pitfalls, such as missing ARIA attributes, poor semantic structure, or inaccessible keyboard interactions, thereby simulating realistic development scenarios where accessibility might be overlooked.

The name of files is defined as: “$framework-table-$validity” where:

• $framework indicates the Javascript framework (i.e., “angular”, “react”, and “vue”) or the “php” language.
• $validity indicates whether the generated table will be valid according to the WCAG guidelines (“accessible”) or not (“invalid”).

Hence, the full list of files of the “Dynamic Generated Content” folder is:

• angular-table-accessible.js
• angular-table-invalid.js
• php-table-accessible.php
• php-table-invalid.php
• react-table-accessible.js
• react-table-invalid.js
• vue-table-accessible.html
• vue-table-invalid.html

The second directory, named “Vue Table Components”, includes 25 distinct Vue-based implementations of table components. These examples were carefully constructed to represent a diverse range of development patterns and accessibility practices. They vary in several dimensions, including the programming paradigm (with examples written using both the Composition API and the Options API), the degree and type of accessibility support provided, and the complexity of internal structure (e.g., use of slots, scoped components, or dynamic data binding). This portion of the dataset is particularly valuable for studying how different architectural choices in Vue development influence the interpretability and accessibility of the resulting components.

Each distinct Vue-based implementation is structured in a separate folder, named “delivery-$n” where $n ranges from “01” to “25”. Each folder contains the following files and folders:

3. Methods

3.1. Languages and Frameworks Covered

The dataset has been designed to reflect real-world patterns in contemporary web development, with a particular focus on technologies that dominate the current frontend and backend ecosystem. In selecting the technologies to be included, priority was given to JavaScript frameworks and PHP, based on their widespread adoption and relevance in both legacy and modern web applications.

JavaScript remains the cornerstone of interactive web development. Within the JavaScript ecosystem, the dataset includes examples written in Vue, React, and Angular, three of the most prominent frontend frameworks. These frameworks offer declarative syntax, component-based architecture, and varying levels of support for accessibility, making them particularly relevant for studying how accessibility violations can manifest and be remediated in different environments. Their differences in templating systems, rendering strategies, and reactive paradigms also provide a useful contrast for evaluating the reasoning capabilities of LLMs when analyzing source code.

Vue.js was selected for special attention due to its flexibility in allowing developers to write components using two distinct paradigms: the Options API and the newer Composition API. The Options API relies on a traditional object-based syntax, promoting readability and clarity for beginners. In contrast, the Composition API uses the setup() function and encourages function-based organization, improving code reuse and scalability in complex applications. Including both paradigms in the dataset enables fine-grained analysis of how code structure and abstraction style may affect accessibility practices and their interpretability by LLMs.

In parallel, PHP was included as a representative of server-side technologies that still play a significant role in generating dynamic HTML content. Though often overlooked in modern single-page applications, PHP remains widely used in content management systems and traditional websites. Including PHP in the dataset allows the evaluation of LLMs in contexts where accessibility violations arise from back-end generated content, such as dynamic tables or forms rendered from server-side scripts.

Together, these frameworks and languages ensure that the dataset captures both client-side and server-side sources of accessibility issues, supporting comprehensive analysis and benchmarking.

3.2. Details on Development

The dataset was constructed through a two-phase process involving both professor-authored and student-authored code. The first phase involved manually crafting dynamic web snippets in Angular, React, Vue, and PHP. For each technology, two versions of a table component were created: one accessible, adhering to WCAG 2.1 guidelines (e.g., semantic markup, ARIA roles, keyboard support), and one non-accessible, featuring common violations such as missing labels, improper heading structures, or lack of alternative text. These components were written to reflect typical developer practices and errors, ensuring realistic input for LLM evaluation.

The second phase involved collecting contributions from a cohort of computer science students participating in a web development course. Each student was instructed to design and implement four independent Vue table components as Single-File Components (SFCs) using .vue files. The deliverables included:

• A minimal table using the Options API
• A minimal table using the Composition API
• An accessible table using the Options API
• An accessible table using the Composition API

To guide development and promote consistency, the following constraints were imposed:

• Each student was required to select a unique dataset of at least 50 rows and 5 attributes, representing real-world data such as songs, movies, environmental measurements, etc.
• Students were encouraged to modularize their code using multiple components where appropriate and to leverage native Vue features (e.g., props, slots, reactivity).
• All components had to remain autonomous, meaning no shared logic or global imports were allowed across implementations. Code duplication was explicitly permitted to ensure isolation.
• Deliverables were submitted as compressed .zip archives, with source code and data included, but build artifacts (e.g., node_modules, dist) excluded for portability and reproducibility.

This rigorous protocol ensured that each submission represented a self-contained, fully functional Vue project capable of being evaluated in isolation. Importantly, the dual-paradigm structure (Options vs. Composition API) allows researchers to study whether the underlying code organization influences the likelihood of accessibility errors or the ease with which LLMs can interpret and remediate them.

All student submissions were manually reviewed and sanitized to remove any personally identifiable information, ensure consistent formatting, and verify the presence of both accessible and non-accessible versions.

3.3. Selected Datasets

As part of the data collection process, students were assigned the task of implementing Vue-based table components using a dataset of their choice. Each student selected a unique dataset, ensuring diversity in content while adhering to a common structure requirement: a minimum of 50 records and 5 features per dataset. The topics ranged from entertainment and technology to science and public health, providing a rich variety of use cases for web interface implementations.

Table 1 summarizes the datasets selected by each student. It is worth noticing that in the first delivery, the student employed a self-constructed dataset.

3.4. Usage and Application

This dataset is designed to support a wide range of research and practical applications in the field of AI-assisted web accessibility auditing and remediation. It can be used in isolation (via source code analysis) or in combination with rendered output (i.e., the DOM produced by each component) to investigate accessibility behaviors from both a static and dynamic perspective.

There are several potential use cases:

• Source Code Analysis Only: Researchers may focus solely on the raw HTML, Vue, React, Angular, or PHP source code to test the ability of LLMs or static analyzers to detect accessibility issues without relying on runtime context. This is especially relevant for pre-deployment code review tools.
• Rendered Output Analysis: By executing the components and inspecting the resulting DOM, researchers can simulate what a user’s browser would process. This method is suitable for testing tools that analyze accessibility post-rendering (e.g., using Axe-core or Lighthouse).
• Combined Static and Dynamic Analysis: The most comprehensive use involves combining code-level insights with runtime behaviors, allowing for comparison of LLM predictions and traditional tool outputs. This approach is particularly useful for validating whether LLMs can correctly infer the final structure and accessibility state of dynamic components.

Additionally, the dataset provides an excellent testbed for fine-tuning LLMs, evaluating prompt engineering strategies, or developing hybrid accessibility checkers that integrate rule-based systems with generative AI models. Its diverse inclusion of languages, paradigms, and component architectures makes it uniquely suited for advancing research at the intersection of web development, accessibility, and AI.

4. User Notes

4.1. Compliance with FAIR Principles

The dataset adheres to the FAIR principles—Findability, Accessibility, Interoperability, and Reusability [45]:

• Findability: The dataset is uniquely identified by a Digital Object Identifier (DOI). This permanent reference acts like a digital fingerprint, ensuring it can always be found and cited accurately.
• Accessibility: The dataset is hosted on Zenodo, a repository dedicated to open science, that supports long-term preservation.
• Interoperability: The included examples are composed of plain text files. This makes the data highly flexible and easy to work with, as the source code can be analyzed directly with any standard text editor or programming tool.
• Reusability: The content is published under a Creative Commons Attribution 4.0 International (CC BY 4.0) license. This a very permissive license that allows for unrestricted use, modification, and distribution of the dataset, as long as the original creators are credited.

4.2. Dataset Limitations

While the dataset offers a rich and diverse collection of web components designed to evaluate accessibility in dynamic web applications, several limitations must be acknowledged.

First, the dataset is inherently synthetic and educational in nature. A significant portion of the components, particularly those implemented in Vue, were developed by students as part of a university assignment. Although the submissions were reviewed for correctness, completeness, and adherence to accessibility principles, the code may not fully represent the complexity, scalability, or design patterns commonly found in large-scale, industry-grade applications. As such, the dataset may lack examples of deeply nested components, complex interactivity, or third-party library integrations.

Second, the dataset is currently limited to table-based components, both accessible and non-accessible. While tables are a critical element for accessibility—frequently subject to violations and requiring semantic precision—they do not cover the full breadth of accessibility challenges faced in modern web interfaces, such as modal dialogs, custom widgets, form validation flows, or responsive navigation menus.

Third, the focus on Vue, React, Angular, and PHP excludes other emerging or legacy technologies. Frameworks such as Svelte, Nuxt.js, or server-side technologies like ASP.NET and Python-based frameworks (e.g., Django, Flask) are not included. This may limit the generalizability of findings derived from this dataset to other technological contexts.

Lastly, while the Vue components were carefully designed to be diverse, the controlled structure of the assignment may have led to similarities in implementation approaches, code style, or problem-solving strategies. Furthermore, the requirement for autonomy across implementations—while methodologically useful—may introduce unnatural code duplication not typically seen in production systems.

Despite these limitations, the dataset serves as a valuable and novel resource for the community, particularly for studying how large language models interpret and reason about accessibility in source code across different frameworks and paradigms.

5. Conclusions and Future Work

This paper presents a novel dataset of accessible and non-accessible web components, spanning multiple languages and frameworks with a particular focus on Vue.js implementations. The dataset was carefully curated to provide controlled variations in accessibility, code organization, and development paradigms. It enables diverse research directions, including automated accessibility auditing, LLM-based code reasoning, and the training of supervised or reinforcement learning systems.

The collection bridges a gap between synthetic benchmarks and real-world web code, offering a unique testbed for evaluating the ability of AI models to reason about dynamic and component-based content. By combining static and dynamic code representations, the dataset supports multi-faceted analysis pipelines involving source-level analysis and DOM-based audit tools.

In future work, we plan to:

• Extend the dataset with additional User Interface (UI) components such as modals, forms, and menus that pose more advanced accessibility challenges.
• Incorporate runtime accessibility audit results (e.g., from Axe-core, Lighthouse) as labels to enable supervised learning approaches.
• Develop benchmark tasks for evaluating the accessibility-awareness of LLMs through code completion, explanation, or repair prompts.
• Expand to additional frameworks such as Svelte Nuxt.js, and potentially server-rendered environments such as Django or ASP.NET.

We believe this dataset will serve as a valuable foundation for the development of next-generation accessibility tools and AI-driven development environments that embed inclusive design by default.