Submitted:
14 October 2026
Posted:
15 October 2026
You are already at the latest version
Abstract
Biological computing needs the application of the high throughput machine learning approach to biological sequences. DNA, RNA and protein are the building blocks of the biological sequences and to use them in biological deep learning several sequence based tokentization such as one-hot, k-mer encodings, character, k-mer and trainable byte-pair-encoding (BPE) tokenizers, BERT-style masked-language-model, FASTA/FASTQ input and output are needed. Additionally to use them in the biological deep learning, padding-aware Transformer encoders, dilated residual CNNs, bidirectional LSTMs, and a masked language model are needed. Bio-keras fills this gap with the ready to use the tokenizers and the bio-layers for machine and deep learning. Bio-Keras is TensorFlow, JAX and PyTorch backends compatible and data handling depends on the Numpy.
Keywords:
bioinformatics
; deep learning
; biological sequences
; Keras
1. Introduction
Deep learning is at the forefont of the AI especially after the development of the transformers, which uses self attention mechanism to address questions using the pre-trained language models (Song et al. 2021). It has wide applications from normal sequencing data to the high dimensional single cell sequencing data ( Hu et al. 2026). Genome language models (gLMs) have revolutionzed the genomic language modelling of the genomic data and has played a key role in the prediction of the splice-site, exon scale models, masked language modelling and motif predictions (Shu et al. 2026). Tokenzier, batch encoding and embedding generation of the sequences are the pre-requisite for the classification language models and plays an intricate role in the training of the transformers and the language models (Dotan et al. 2024). Transformer models like BERT, GPT, and DNABERT have been effectively used for classifying introns and exons in genomic sequences. BERT, in particular, achieved the highest classification accuracy, demonstrating the importance of bidirectional context modeling in genomic sequence analysis(Cruz & Pozo, 2025). The Nucleotide Transformer, a series of foundation models, has been recently developed to predict molecular phenotypes from DNA sequences. These models, which are pre-trained on extensive genomic data, can be fine-tuned for various genomics applications, even in low-data settings (Dalla-Torre et al., 2024).
I present bio-keras, which features data modules (alphabets, encoding, tokenizers, preprocessing, io) for sequence preparation along with several modulation layers for the sequence embeddings and to easily classify the language models. Three tokenizers share one interface (encode, batch_encode, decode, save, load). CharTokenizer assigns one token per nucleotide or residue and accepts a named alphabet or a custom symbol set. KmerTokenizer builds the complete vocabulary of k-mers for the chosen alphabet, with a configurable stride so that overlapping (stride 1) and non-overlapping (stride = k) schemes, as used in k-mer-based genomic language models as proposed by recent genomic language models (Zhou et al. 2014) for developing large scale language models specific to the biological sequences (Lee et al. 2026). All tokenizers reserve identifiers for padding, unknown, classification, separator and mask tokens, with padding fixed at 0. For convolutional models, one_hot and batch_one_hot produce (length, alphabet) arrays. IUPAC ambiguity codes in nucleotide sequences are spread uniformly over the bases they represent (for example N becomes 0.25 per base), and padding is encoded as all-zero rows.
2. Software Quality
Bio-Keras has been tested using pytest and contains a total of 18 tests, which span across encoding, all tokenizers and FASTA input/output and model family. To test the models, training for one epoch was done and prediction and serialization was done with the .keras format using JAX backend and Keras 3.15.1.
3. Benchmarking
All the benchmarking was done on single single virtual CPU core with 3 GB of memory (Python 3.12.3, NumPy 2.4.4, Keras 3.15.1, JAX 0.11.2, CPU only), using synthetic uniformly random sequences. Data-handling timings are the best of 3 to 5 repetitions; model timings are the best of three epochs after one warm-up (compilation) epoch. Tokenization has a major influence on the genome language models (Dotan et al. 2024). Most data-handling operations process millions of bases per second on one core (Table 2). One-hot encoding of 2,000 sequences of 1 kb takes 0.24 s, non-overlapping k-mer tokenization is about five times faster than overlapping tokenization, and reading plain FASTA runs at several hundred megabases per second. The two exceptions are BPE tokenization and gzip-compressed FASTA writing. BPE encoding ran at about 23 kb/s, roughly 250 times slower than k-mer tokenization, and training a 512-token vocabulary on 300 sequences of 500 bp took 9.8 s.
4. Results
bio-Keras has been built to support low end computing resources also as fine tuning pre-trained models has been shown applicative in the genomic language model space (Dalla-Torre et al., 2024) and to evaluate the same, we evaluated bio-Keras on 512 sequences of 126 nt encoded to 128 tokens (batch size 32; 3-mer tokens for Transformers, one-hot input for the CNN, single-nucleotide tokens for the BiLSTM) (Table 3). A small two-layer Transformer (embedding dimension 64, four heads, 117k parameters) processed 137 sequences/s in training and about 1,170 sequences/s at inference. The dilated residual CNN (171k parameters) trained about 2.5 times faster than this Transformer. Increasing the Transformer to four layers with embedding dimension 128 (827k parameters) reduced training throughput to 39 sequences/s. Therefore, computational cost is minimal and for long genomic window and convlutional models with chunking and pooling layers available from bio-Keras will benefit the genomic language models.
5. Discussion
Sequence transformers and attention mechanism has gained wide applicability and has shown promisable evidence of the sequence models in genome language modelling including the representation learning using the vectorized biological sequences (Iuchi et al. 2021). While transformers offer significant advancements in genomics, their complexity and the “black box” nature can be a barrier for some researchers. Efforts to demystify these models, such as providing annotated scripts and conceptual explanations, are crucial for broader adoption and effective application in biological research (Friedman, 2025). Genome wide application of the genome language models in intron-exon classification, splice-site prediction, DNA-Protein translation and sequence labelling has been shown and demonstrated (Iuchi et al. 2021, Song et al. 2021). bio-Keras is intended as a compact, composable starting point rather than a replacement for large pretrained genomic foundation models. Bio-Keras acts as a inter-connecting bridge between the use data and the genome language models by providing tokenization, encoding, augmentation including pre-training data preparation and model construction. We believe that bio-keras will play a major role in the development of the large language models and efficient language models for the biological sequences quickly and with minimal effort (Song et al. 2021, Madan et al. 2024).
Supplementary Materials
The following supporting information can be downloaded at the website of this paper posted on Preprints.org, Benchmark scripts and raw results are available in the repository / Supplementary Data.
Author Contributions
GS concieved the software, developed the software and wrote the paper.
Funding
GS received no funding for this work.
Conflicts of Interest
GS declares no conflict of interest.
Availability and Implementation
bio-keras is implemented in Python (>= 3.9) and released under the MIT license. Source code, documentation and examples are at https://github.com/sablokg/bio-keras. It can be installed with pip install -e.
References
- Zhou Z, Ji Y, Li W, Dutta P, Davuluri RV, Liu H. DNABERT-2: efficient foundation model and benchmark for multi-species genomes. In: International Conference on Learning Representations (ICLR 2024). (2024).
- Song B, Li Z, Lin X, Wang J, Wang T, Fu X. Pretraining model for biological sequence data. Brief Funct Genomics. (2021) 20:181–195. [CrossRef]
- Hu T, Wei Z. Transformers for single-cell RNA sequencing: a survey. Brief Bioinform. (2026) 27:bbag332. [CrossRef] [PubMed]
- Shu L, Tang J, Guan X, Zhang D. A comprehensive survey of genome language models in bioinformatics. Brief Bioinform. (2026) 27:bbaf724. [CrossRef] [PubMed]
- Lee G, Kim S, Lee D, Goh K, Kim H. Towards efficient language giants: a comprehensive survey on structural optimizations and compression techniques for large language models. Neural Netw. (2026) 201:108900. [CrossRef] [PubMed]
- Madan S, Lentzen M, Brandt J, Rueckert D, Hofmann-Apitius M, Fröhlich H. Transformer models in biomedicine. BMC Med Inform Decis Mak. (2024) 24:214. [CrossRef]
- Dotan E, Jaschek G, Pupko T, Belinkov Y. Effect of tokenization on transformers for biological sequences. Bioinformatics. (2024) 40:btae196. [CrossRef]
- Iuchi H, Matsutani T, Yamada K, Iwano N, Sumi S, Hosoda S, et al. Representation learning applications in biological sequence analysis. Comput Struct Biotechnol J. (2021) 19:3198–3208. [CrossRef]
- Cruz G, Pozo ATR. Large language models for genomic sequence understanding: classifying introns/exons and translating DNA to protein. In: 2025 IEEE 25th International Conference on Bioinformatics and Bioengineering (BIBE). (2025). p. 653–657. [CrossRef]
- Dalla-Torre H, Gonzalez L, Mendoza-Revilla J, Lopez Carranza N, Grzywaczewski AH, Oteri F, et al. Nucleotide Transformer: building and evaluating robust foundation models for human genomics. Nat Methods. (2025) 22:287–297. [CrossRef] [PubMed]
- Friedman R. Anatomy of a Transformer: an essay on the code and concepts for biological sequence analysis. Preprints.org. (2025). Posted June 9, 2025. [CrossRef]
Table 2.
Throughput of data-handling functions (single core).
| Operation | Throughput |
|---|---|
| One-hot encoding, DNA (2,000 × 1 kb) | 8.3 Mb/s |
| One-hot encoding, protein (2,000 × 300 aa) | 6.2 M aa/s |
| Character tokenizer, batch encode | 10.5 Mb/s |
| k-mer tokenizer, k = 3 or 6, stride 1 | 5.7 Mb/s |
| k-mer tokenizer, k = 6, stride 6 | 29.7 Mb/s |
| k-mer frequency vector, k = 4 | 3.8 Mb/s |
| Reverse complement | 25.8 Mb/s |
| Six-frame-capable translation (one frame) | 17.7 Mb/s |
| MLM token masking | 80 M tokens/s |
| FASTA read / write (plain) | ~400 / ~360 Mb/s |
| FASTA read / write (gzip) | 125 / 1.5 Mb/s |
| BPE encode | 0.023 Mb/s |
Table 3.
Training and inference throughput (sequences/s) on one CPU core, JAX backend.
| Model | Parameters | Training | Inference |
|---|---|---|---|
| Transformer, d = 64, 2 layers, <cls> pooling | 117k | 137 | 1,167 |
| Transformer, d = 64, 2 layers, mean pooling | 117k | 137 | 1,001 |
| Transformer, d = 128, 4 layers, mean pooling | 827k | 39 | 261 |
| Dilated residual CNN, 64 filters, 4 blocks | 171k | 337 | 1,510 |
| CNN with reverse-complement augmentation | 171k | 346 | 1,496 |
| BiLSTM, 64 units, 2 layers | 174k | 276 | 1,485 |
| Masked language model, d = 64, 2 layers | 117k | 133 | n/a |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.