Early Experiences Using AI Software Agents on OLCF Systems: From Installation and Environments to Simulation Workflows for Molecular and Materials Sciences

1. Introduction

Modern computational science increasingly depends on workflows that span theory, simulation, machine learning, and experimental validation. In molecular simulation, a single research question can involve physical reasoning, simulation analysis, deployment on large high-performance computing (HPC) clusters, and recently, machine-learning with deep learning (DL) tools. The software stack includes GPU-accelerated and distributed codes, hybrid HPC/DL simulations, and scheduler-controlled parallel and distributed execution on large computing clusters, including on leadership-class systems such as the OLCF. The scientific value of the workflow depends on the entire chain. Even a correct model or simulation engine can fail scientifically when the programming environment, data conversion methods, launch patterns, or validation steps silently diverge from scientific assumptions. Furthermore, with these increasingly complex software stacks on modern HPC computing resources, building and deployment can be a monumental task and represents a substantial hurdle, creating a persistent productivity problem for HPC-based computational science. Domain experts must repeatedly translate scientific intent into software environments, data schemas, Slurm scripts, validation checks, and post-processing logic. The translation is often fragile because the relevant information is distributed across papers, notebooks, source repositories, cluster documentation, local logs, and interactive conversations. Leadership systems amplify the difficulty because each platform exposes a distinct combination of modules, compilers, GPU runtimes, MPI implementations, file systems, and scheduler behavior.

Agentic AI coding systems such as OpenAI’s Codex1 and Anthropic’s Claude Code2 could offer a new way to reduce this integration burden. Their value for HPC efforts within computational science, however, remains relatively unexplored. As opposed to commercial software development, scientific computing research involves numerous open-ended discovery processes, which may lack definitive specifications and verification gates, making verification more difficult. In addition to the long-standing considerations about model accuracy and approximations, numerical stability of the algorithms and their implementations, and finding bugs in large code bases that contain multiple programming models for multi-level parallelism, AI coding agents introduce more verification challenges because of the inherent stochasticity and sensitivity of neural network models, combined with the problematic reproducibility and reliability, and the tendency for large language models to hallucinate and cheat [1,2,3,4]. Whether they can help computational scientists by providing coherent and functional assistance across software prototyping, system integration, and experimental validation while preserving enough provenance for later inspection remains to be seen. This paper studies that question through early molecular simulation workflows on OLCF resources. Here, our case studies focus on HPC interatomic and molecular simulations for biological, chemical, and materials science, including machine learning interatomic potentials (MLIPs) and molecular dynamics (MD).

The paper is organized around three research questions. The first asks whether an AI agentic coding system can help resolve complex scientific software builds on nonstandard HPC platforms quickly enough to change the practical workflow. The second asks whether the same system can assist scientific method development after code is already running, including numerical testing, uncertainty analysis, prototyping, deployment, and revisitation of a method after failures. Finally, we evaluate whether the agent can assist with theory development, addressing the bottleneck of translating theoretical scientific concepts into testable calculations.

We find that both agentic systems can resolve complex build and deployment problems on nonstandard HPC systems, including Frontier configurations that require architecture-specific GPU software, scheduler-aware launch tests, and source-built simulation backends. We estimate that production speedups in these areas can approach several orders of magnitude compared to single expert-human unassisted work. Our testing also shows that coding agents can assist scientific method development beyond code generation, by helping experts connect theory, numerical tests, uncertainty and validation reasoning, prototyping, deployment, and method revision. We also identified some reliability practices that made agentic coders more usable for scientific computing: expert supervision, source documentation, run-scoped artifacts, negative-result preservation, and explicit validation checks for reported claims.

2. Background

Scientific computing workflows have multiple layers of conceptual and operational complexity. AI-driven molecular simulation increases this complexity while reducing production bottlenecks. The scientific computing use cases we study in this paper represent some of the more complex molecular simulation workflows we encounter, and are representative of current trends in molecular and materials sciences. For example, an MLIP workflow using a neural network trained on interatomic energies and forces from atomistic or quantum mechanical simulations requires numerous steps, including selecting or adapting a physical representation, preparing atomic or coarse-grained data, validating units and labels, running molecular dynamics, detecting failure modes, selecting new configurations for training, computing reference labels, and repeating these loops until convergence [5,6]. Errors in any stage can contaminate downstream results[7,8,9]. This is especially clear in active-learning and fine-tuning workflows for MLIPs. A model can run molecular dynamics quickly, but the simulation must be checked for stability, chemical plausibility, and consistency with the training domain. When the trajectory reveals a broken bond or an out-of-domain configuration, the failure becomes evidence for the next labeling or fine-tuning iteration. The workflow therefore needs both computational throughput and scientific judgment. The HPC further defines the character of this problem. On a leadership system, a failure can consume queue time, block a scientific campaign, or hide inside a batch script that only runs after a long wait; after a run, hundreds to thousands of output files may be generated. Coarse-graining [10] adds another layer of theory. A mapping from atomistic coordinates to coarse-grained sites is a modeling decision that determines which degrees of freedom remain visible to the learned potential. In the coarse-graining MLIP workflow considered here, the mapping needs to support mass-weighted coordinates, force aggregation, energy propagation, unit conversion, and MACE-compatible training data, while being compatible with GPU memory limits and other efficient deployment considerations, representing a combined theoretical and computational problem. Uncertainty also appears at several levels. There is physical uncertainty about whether a selected mapping preserves the coordinates needed to represent solvent-mediated forces. There is numerical uncertainty about whether a training curve reflects model capacity, data quality, or a collapsed representation. There can be systems uncertainty, for example whether an observed failure is caused by launch configuration, Python environment, GPU kernel compatibility, or a software bug. The practical consequence is that an AI-assisted workflow presents additional uncertainty on top of these already-present sources, whether the agentic AI’s documentation and reporting on its actions is faithful, reliable, reproducible, and trustworthy.

Here we report that coding agents greatly reduced time for installation, deployent and testing of these complex workflow stacks, but introduced multiple levels of uncertainty and verification challenges that may require co-design together with the vendors producing the agents. Successful build and deployment stages included a validated Frontier ROCm stack for training and performing simulations with a machine learning interatomic potential (MLIP) that couples PyTorch and the LAMMPS simulation engine with the MACE MLIP package, and resolving a Spack package build for the large molecular and materials simulation package CP2K. In addition, Codex helped move coarse-graining theory discussions into executable tests and prototype numerical experiments and simulations, including creating different mappings from atoms to beads and performing distributed training of an MLIP bead model from generated atomistic polarizable molecular dynamics data.

3. Methodology

3.1. Target HPC Cluster Platforms

OLCF Frontier is an HPE Cray EX system with AMD EPYC CPUs, AMD Instinct MI250X accelerators, eight GPU-visible GCDs per node, Slingshot interconnect, and Orion Lustre storage [11]. Wombat is a testbed system which provides an Arm and NVIDIA portability target with several AArch64 node classes, including Fujitsu A64fx CPU nodes with EDR InfiniBand, Ampere Computing Altra CPU nodes with two NVIDIA Ampere GPUs and two BlueField-2 DPUs, and Grace Hopper nodes with BlueField-3 DPUs. The GH200 nodes include a 72-core Arm Neoverse V2 CPU at 3.52 GHz, 480 GB ECC LPDDR5X memory, and a GH200 GPU with 96 GB ECC HBM3 memory [12].

3.2. Agentic AI Coding Assistants

We used Codex with GPT 5.4 in high accuracy mode, and Claude Code v2.1.232. Across the tests, the work remained human supervised. The user provided scientific goals and corrections. The records tracked run state: machine context, loaded modules, Python environment, command line, input file hashes when available, output paths, scheduler identifiers, and final status.

3.3. Molecular Simulation Packages Used

We used the MACE program and the MACE-OFF23 potential as the primary framework for MLIP training and development [5,13,14], and its coupling with the LAMMPS and OpenMM HPC-based MD engines as representative simulation backends [15,16,17], deployed on Frontier, an HPE Cray EX system with AMD GPUs, and Wombat, an AArch64 Arm and NVIDIA testbed. On Frontier, the MACE setup used a ROCm Python environment with PyTorch 2.8.0+rocm6.4, MACE 0.3.15, and PyTorch GPU visibility. The OpenMM simulations with the AMOEBA force field used the NPT ensemble with one-femtosecond timesteps, one-nanosecond per-replica targets, selected-frame export, and a single-srun eight-replica launch pattern. The coarse-grained RNA implementation computed bead positions from mass-weighted centers of mass, summed atomistic forces onto bead forces, and wrote MACE-ready extxyz datasets using the REF_forces key. For distributed training, we compared torchrun, Slurm-based launch, and MPI-style launch across two to eight GPU-visible GCDs. We used Spack for testing of installation of CP2K[18].

4. Results

4.1. Installing and Testing MLIP and MD Stacks

Codex was used to successfully install, test, and prototype several different simulation workflows with the MACE and OpenMM packages on Frontier. The LAMMPS MD engine was successfully coupled to MACE to run MLIP-MD. The agent recorded missing batch-environment commands, compile-mode incompatibilities, and ASE HIP out-of-memory behavior for the benchmark calculations.The final MACE-LAMMPS installation used the ACEsuit LAMMPS fork, the ML-MACE package, Kokkos, HIP targeting for gfx90a, Torch CMake metadata workarounds, and mace_create_lammps_model –format libtorch. We completed a 20,000-step ASE MACE MD trajectory and preserved the trajectory, log, manifest, and reasonableness-check artifacts.

Table 1. Systems used in the case studies.

System	Relevant hardware	Target	Integration difficulty
Frontier	HPE Cray EX, AMD EPYC CPUs, AMD MI250X GPUs, Slingshot, Orion Lustre	MACE, ASE MD, LAMMPS, coarse-grained RNA training	ROCm PyTorch selection, HIP/Kokkos build choices, Slurm and one-rank-per-GPU execution
Wombat A100 partition	AArch64 Arm/NVIDIA testbed environment	MACE plus LAMMPS/OpenMM portability exploration	Python packaging, CUDA wheel selection, and backend compatibility
Wombat GH200 nodes	72-core Arm Neoverse V2 CPU, 480 GB ECC LPDDR5X, GH200 GPU with 96 GB ECC HBM3, BlueField-3 DPU	Platform context for Arm + NVIDIA systems	Distinct CPU ISA, memory hierarchy, accelerator packaging, and power/energy instrumentation

Table 1. Systems used in the case studies.

System	Relevant hardware	Target	Integration difficulty
Frontier	HPE Cray EX, AMD EPYC CPUs, AMD MI250X GPUs, Slingshot, Orion Lustre	MACE, ASE MD, LAMMPS, coarse-grained RNA training	ROCm PyTorch selection, HIP/Kokkos build choices, Slurm and one-rank-per-GPU execution
Wombat A100 partition	AArch64 Arm/NVIDIA testbed environment	MACE plus LAMMPS/OpenMM portability exploration	Python packaging, CUDA wheel selection, and backend compatibility
Wombat GH200 nodes	72-core Arm Neoverse V2 CPU, 480 GB ECC LPDDR5X, GH200 GPU with 96 GB ECC HBM3, BlueField-3 DPU	Platform context for Arm + NVIDIA systems	Distinct CPU ISA, memory hierarchy, accelerator packaging, and power/energy instrumentation

4.2. Distributed Training Results

For distributed MACE training, a distributed-method check swept two to eight GCDs and found that torchrun passed all seven tested cases, while the tested Slurm and MPI launch methods failed all seven cases.

A later eight-GCD debug scaling run completed training for 10 epochs on 12,800 graphs and reported 106.3 configurations per second before the top-level Slurm job reached its 20-minute walltime during post-processing. The smoke-run dataset split contained 649 training frames, 81 validation frames, and 82 test frames.

4.3. Creating Spack Environments on Frontier

Installing software on HPC systems is a notoriously difficult task for all users, as HPC environments are non-standard. HPC systems also offer multiple versions of the major mathematical libraries, such as MPI or BLAS/LAPACK, which complicates the task for users regardless of their experience. Machine learning and chemistry codes also come with multiple dependencies that must be built separately, increasing the amount of work required to obtain a working stack. In the past decade, dedicated package managers targeting HPC have been developed to mitigate these build issues. The package manager is responsible for building every component of a given software stack, provided that an initial configuration, specific to each cluster, is written. Writing these configurations remains tedious. We found the process can be greatly accelerated using Claude Code. We used the Spack package manager[19] together with Claude Code to create a Spack environment on Frontier from scratch, containing a minimal version of CP2K[18] and reusing as many pre-installed packages as possible. In practice, CP2K only requires cray-mpich and cray-libsci, and Spack builds the remaining dependencies.

Writing a full Spack environment by hand is time-consuming. Even though Spack can detect most of the installed packages automatically, some editing of the configuration files is often required to obtain a working configuration. Coding assistants can generate the basic configuration files from the cluster documentation, or by directly probing the installed software and module environments on the login nodes.

This prompt creates a full environment that, once installed, contains a basic version of CP2K with GPU support: “Create a spack environment named frontier-cp2k including gcc, mpi, rocm and cp2k with GPU support with their latest officially-supported version on Frontier. Use the latest version of spack. Keep an history of all commands. Use spack concretize -f to check that the environment file works."

The artifact produced by this prompt can be found in the supplementary material. Generating the Spack environment (before compilation) takes between 10 and 20 minutes in semi-automatic mode, compared to one or two hours when written by an experienced user with Spack alone. This is, however, not the most difficult part, as many issues can be resolved automatically by the coding assistant with minimal user intervention.

The most sensitive part of the process is the compilation step, since compilation errors cannot be detected while the Spackenvironment is being generated. During the build with the latest official version of Spack(v1.2.2), Claude Code identified several issues with packages that require MPI. The fftw library, for instance, failed to compile because its configuration step could not link cray-mpich against libfabric. Cray’s libmpich.so has a transitive runtime dependency on libfabric (the OFI provider library underlying Frontier’s Slingshot-11 interconnect) that is not expressed anywhere in Spack’s ‘cray-mpich’ package recipe, and is not resolvable through the compiler’s default library search path. The issue is narrow and is related to how the Spackcompilation environment is configured during the installation stage, and to how the initial user environment is set before running spack install. This bug applies to v1.2.2 and earlier; on the current development branch the same call returns without error. A fix is present on the development branch of Spack(PR #52752, merged 2026-07-22, which adds an early return when the module is already present in LOADEDMODULES) but has not yet appeared in a tagged release, the latest being v1.2.2 at the time of writing.

The coding assistant used a legitimate and documented mechanism when configuring the Spackenvironment; however, a combination of the cluster’s user environment and the mishandling of already-loaded modules by Spackraises a compilation error during the final installation stage that can be detected only after extensive debugging.

4.4. Codex and Deep Research for Theory Development Assistance

We tested whether Codex could be used for assistance with translating theoretical concepts into testable numerical experiments. We supplied the agent the chemical question, literature context and existing repositories with similar strategies and corrections. Codex helped turn those materials into explicit modeling alternatives with prototype scripts, numerical checks, and Frontier-executable tests. This was especially important for RNA coarse-graining, where the choice of mapping determines which molecular degrees of freedom remain visible to the learned potential. The initial question was whether a very small RNA mapping could support MACE training against atomistic reference trajectories in water. The discussion quickly exposed a central modeling risk: compressing the nucleotide too aggressively can erase sugar pucker, base orientation, phosphate electrostatics, and the RNA-specific 2’-OH hydration environment. Figure 1 shows the different coarse-grained mapping strategies for RNA simulation that were developed in the iterative theory/computing loop, assisted by the agentic workflow.

Table 2 displays the details of the coarse-grained mapping within the theory/compute workflow, including the scientific risks and failure modes identified and how improvements were organized. The workflow treated projected atomistic forces as the training target for force matching. Frame-level forces were propagated to the coarse-grained frame for force-matching compatibility, and the generated datasets used the MACE key REF_forces. Candidate MACE models were trained on alternative mappings, and the resulting logs, learning curves, model artifacts, and launch behavior were used to determine whether the pipeline was numerically executable.

Codex enabled a substantial speedup in the time it takes to connect theory revisions to executable numerical tests; in some cases, the speedups could be estimated as over 1,000×. This result is different from the previous testing: the build work showed that coding agents can navigate unusual HPC software stacks, while here, theory work testing showed that, with expert supervision, the same systems can help scientists develop and test theoretical ideas using numerical experiments, although these coding agents were not trained for this purpose.

5. Discussion

Our observations support the idea that coding agents increase productivity for installation, testing, deployment, and management of HPC molecular simulation workflows. The results are early experience, not a controlled productivity benchmark, and the systems results are platform-specific. The coarse-grained RNA results are intermediate milestones rather than a completed force field. A formal study should measure expert time, failure recovery time, reproducibility quality, and whether the assistant proposes tests that distinguish physical, numerical, and systems failures. The broader implication is that AI coding systems can become useful HPC reasoning and integration tools when experts define acceptance criteria, validate scientific meaning, and require an audit trail for every accepted result.

AI assistance is valuable when it accelerates the production of reviewable data, but it is risky when the amount of data to be verified grows too large. For this reason, logs, model files, scheduler records, and generated datasets can be used as evidence in a verification process and should be required to be written by the agent. However, the amount of files produced by even a short session with a coding agent like Codex or Claude Code on an HPC system can rapidly become overwhelming for human verification. Our RNA coarse-graining work with Codex produced around 94,000 files, including scripts, input files for simulations and MLIP training, and simulation and training output files.

The task of verifying 94,000 files written by an AI agent that could potentially be hallucinating, over-interpreting, or even cheating at some point in the process is a daunting one. Work is already underway across many fields using AI agents to develop systems of verification agents to test production coding assistants and reduce this verification burden [20]; these results show that reliability still remains a challenge in the verification efforts as well. Future efforts in this area will need to codify what we mean by truth and correctness in scientific computing in this new paradigm, similar to what has been said about mathematics in this new AI era3. This perspective informs how HPC facilities might support AI-assisted science and what agent developers can focus new efforts on for supporting computational science on HPC systems. In scientific computing, the unit of useful work is often a validated computational state. The broader implication is that AI coding systems must be able to faithfully document and reproduce these states, which include the machine, the loaded environment, the input data, the launcher, the generated artifacts, the failure modes, numerical considerations, reproducible and testable executables, and the scientific interpretations attached to the result. Final verification may need to focus on what can be checked mechanically: whether claimed statuses, states, output files and results exist contain all expected information and records and nothing more, and whether they can be manually reproduced without the agent.

Language models are strong at producing coherent and plausible explanations, which can lead to over confidence in their methods, summaries and outputs. The practical response is to keep the assistant focused on discriminating tests and to reserve interpretation for the point at which testable artifacts are available. Future tools should support negative evidence as a first-class object. Most automation systems focus on successful runs because those are the runs that produce final artifacts. Scientific debugging requires the opposite habit. Failed runs explain the boundary of the result. They show which assumptions were tested and rejected. They can also reveal hidden constraints, such as a scheduler launch pattern that serializes work, an accelerator wheel that targets the wrong architecture, or a backend interface that requires a different model export. Capturing those failures in a structured but lightweight way would make AI assistance safer because the assistant could reason from the full path, not only from the final success.

The useful role for the assistant is to shorten the path to an artifact that a domain expert can judge. The expert must still decide whether a mapping preserves the right degrees of freedom, whether a model error is chemically meaningful, and whether a simulation is suitable for the scientific question. These requirements are modest compared with fully autonomous science. They require the assistant to be explicit and faithful about evidence, uncertainty, and provenance. Nevertheless, there requirements are currently out of the scope of state-of-the-art tools.

6. Related Work

This work connects machine-learned interatomic potentials, molecular dynamics software, workflow provenance, and AI-assisted programming. MACE introduced higher-order equivariant message passing for accurate force fields, and foundation models such as MACE-MP and MACE-OFF extended this approach to transferable potentials [5,13,14]. OpenMM and LAMMPS represent two important styles of MD infrastructure. OpenMM supports rapid development of high-performance MD algorithms and machine-learning potential integration [15,16]. LAMMPS provides a broad compiled simulation framework for particle-based materials, chemistry, and mesoscale modeling [17]. Our experience shows that backend selection also depends on runtime compatibility, model export format, MPI behavior, scheduler launch pattern, and validation feasibility.

Prior studies of Codex and GitHub Copilot show that large language models can improve programming productivity while introducing correctness, usability, and security concerns [21,22,23,24]. Recent work with faithfulness, reliability, and reproducibility has worked to elucidate and characterize such challenges in scientific discovery processes [2,3,4,20] .

7. Conclusions

We presented early experiences using coding agents to accelerate scientific computing research tasks for molecular simulation on HPC systems. Codex and Claude Code helped accelerate production and solve difficult installation issues substantially on the Frontier and WOmbat systems at the OLCF.

The main lesson is that AI coding systems are most valuable when they reduce friction between scientific intent and machine-specific execution, and when they help scientists revisit methods after numerical evidence returns. Codex resolved complex builds on unusual HPC platforms and helped organize scientific reasoning around coarse-grained mapping, numerical testing, uncertainty, deployment, prototyping, and method revision. These benefits depended on documented evidence, preserved negative results, and expert validation of scientific meaning. The immediate implication for HPC centers is practical. AI-assisted scientific computing should be evaluated by whether it leaves reviewable evidence: documented environments, replayable commands, accepted and rejected launch paths, validation checks, and explicit boundaries on what was not proven.

This experience also suggests a more conservative standard for reporting AI-assisted scientific computing. The useful contribution is not only that an assistant produced correct code, but also that the assistant can both expose and faithfully document enough machine state, failure state, and scientific context for the human team to make a defensible decision and to maintain the scientific method. That distinction is important for cluster-style systems research because HPC productivity is shaped by many details that are invisible in ordinary software benchmarks, and yet can generate massive amounts of files and data that must be verified.

The broader opportunity is to make this discipline routine. Future AI-assisted HPC tools should preserve process, local context, record why paths were rejected, distinguish direct observations from interpretation, and ask for missing artifacts before strengthening a claim. If those practices become part of the interaction model, AI coding systems can become practical instruments for scientific computing rather than occasional sources of useful scripts. They can help researchers navigate unusual architectures, port complex software stacks, construct numerical tests, and revisit scientific methods as evidence accumulates. The central requirement is accountability. The assistant must remain tied to the files, commands, logs, and expert judgments that make a computational claim trustworthy.