The Application and Development of Grapheme-Phoneme Conversion

1. Introduction

In the theory of writing systems, grapheme is defined as the smallest distinctive writing unit in the orthographic system, whose function is to establish the correspondence between phonological representation and visual symbol[1]. Phoneme is a physical speech unit divided based on acoustic features[2].Grapheme-to-Phoneme (G2P) is dedicated to building a mapping relationship between the smallest unit of written text, grapheme, and the sound unit of the speech system, phoneme. Its mapping function is expressed as:

$$f:G\to P$$

(1)

where $f$ is the conversion function, which maps the grapheme sequence $G=(g_1,g_2,\dots ,g_n)$to the phoneme sequence $P=\left(p_1,p_2,\dots ,p_3\right)$. This technology plays an important role in the field of speech information processing, and its value is mainly reflected in three aspects: First, G2P is a key part of the front-end text processing module in the text-to-speech synthesis system (TTS), and the mapping accuracy directly affects the naturalness of the synthesized speech. Secondly, in the field of Automatic Speech Recognition (ASR), G2P effectively expands the vocabulary coverage of the ASR system by generating phoneme sequences of new words, proper nouns, and unregistered words, and improves the adaptability of the ASR system to dynamic language environments. Finally, in speech simulation technology, precise G2P technology provides reliable underlying support for the acoustic modeling of speech signals, thereby enhancing the system's ability to represent speech features and the quality of generation, helping machines to more accurately simulate human speech and improve the understanding of speech signals.

The research on G2P began in the mid-twentieth century, and its research methodology system has undergone three major paradigm shifts: the initial deterministic modeling based on artificial linguistic rules (1950-1980s), the mid-term statistical probability model (1990s-2010s), and the current technical system dominated by deep learning architecture (2010s). Each paradigm shift stems from the limitations of previous methods in processing complex language phenomena. Against this background, in recent years, researchers have used complex neural networks such as Long Short-Term Memory (LSTM) [3] and Convolutional Neural Network (CNN) [4] models to perform G2P conversion and reconstruct the G2P technical path. In 2018, the Google research team proposed the Bidirectional Encoder Representations from Transformers (BERT) [5] model, which pioneered the "pre-training-fine-tuning" transfer learning paradigm. This method is a milestone in the G2P field. It effectively alleviates the three major problems faced by traditional methods in multilingual speech synthesis systems: the modeling dilemma of irregular phonology, the complexity of traditional TTS steps, and the generalization bottleneck in multiple languages. This paradigm of transfer learning marks that G2P has entered a new stage of scalability and generalization.

This paper aims to sort out the application fields and research method development in the field of G2P, and finally focus on the innovative breakthroughs of neural network architecture and pre-training models. The content is arranged as shown in Figure 1. First, from the perspective of speech engineering, the important role of G2P technology in the two major application fields of TTS and ASR was discussed; then, through a diachronic analysis, the methodological transformation of G2P research from symbolism to connectionism was carried out, and a comparative analysis of G2P methods based on deep learning under the same data set was conducted; finally, the two hot topics of different language dialects and multilingual G2P were discussed, and the future direction of the development of G2P technology was proposed, establishing a theoretical reference system for subsequent research.

2. G2P Application Areas

As an important component of speech information processing technology, G2P plays a key role in the speech technology ecosystem. Its technical value is mainly reflected in the following three application dimensions: front-end processing of TTS, vocabulary expansion mechanism of ASR, and cross-modal alignment of multimodal speech retrieval[6].

2.1. Speech Synthesis System

In the technical framework of TTS, G2P assumes the core function of the grapheme-phoneme interface. Its essence is to provide the underlying representation for the parameter generation of the acoustic model by constructing a deterministic mapping from grapheme sequence to phoneme sequence (Figure 2). With the widespread application of TTS technology in intelligent voice interaction systems (such as virtual digital humans, IoT terminal devices, and barrier-free reading tasks), the performance requirements of the G2P module have evolved from basic intelligibility guarantees to multi-dimensional indicators such as accuracy, rhythmic naturalness, and emotional adaptability.

The evolution of TTS technology can be divided into three stages. The first stage is the synthesis stage based on unit selection and waveform splicing (1980s-2000s) [7,8] (such as the Diphone system). Its idea is to rely on a large-scale pre-recorded speech database to splice the target phoneme units in the time domain through a dynamic programming algorithm. This method has defects such as obvious splicing traces and exponential growth of database storage in synthesized speech, so it has been replaced [9]. The second stage is the speech synthesis technology based on statistical parameters, which has achieved the transformation of speech generation models [10,11,12,13]. Its technical path can be decomposed into three layers: first, in the front-end text analysis layer, the G2P module outputs the phoneme sequence and generates prosodic features through context-dependent modeling; then, in the acoustic parameter generation layer, the acoustic parameters such as Mel-Frequency Cepstral Co-efficients (MFCC) are synthesized based on the maximum likelihood parameter generation algorithm [14]; finally, in the waveform reconstruction layer, the time domain signal is reconstructed through the Griffin-Lim algorithm [15], such as the typical Hidden Markov Model (HMM)-based speech synthesis system (HMM-based Text-to-Speech, HTS) [14]. Third, under the speech synthesis technology driven by deep learning, the end-to-end architecture based on neural networks is used to achieve direct mapping from text to waveform (2010s to present). The end-to-end architecture represented by WaveNet [16] and Tacotron [17] has completely reconstructed the TTS technology stack. Its system architecture is usually composed of three core modules: linguistic front-end processing module, neural acoustic model and neural vocoder [9], as shown in Figure 2. The front-end processing module is responsible for text normalization, among which G2P, as a key submodule, needs to complete complex functions such as word boundary detection and polyphone disambiguation. Acoustic models (such as FastSpeech2) [18] map phoneme sequences to mel-spectrogram features through an encoder-decoder architecture. Vocoders (such as WaveGlow) [19] convert mel-spectrograms into 24KHZ high-fidelity waveforms through a reversible flow model (Glow-based Model), and their signal-to-noise ratio is improved by 14.6dB compared to the traditional Griffin-Lim algorithm.

2.2. Automatic Speech Recognition

As an interdisciplinary field of computational linguistics and speech signal processing, the essence of ASR technology is to construct a mapping function $f:X\to Y$ from a time-domain speech signal $X\left(t\right)$ to a discrete symbol sequence $Y=\{y_1,y_2,\dots ,y_n\}$ where $X$ is the speech signal space and $Y$ is the text space consisting of a finite character set. The engineering application of ASR technology has penetrated into multiple dimensions of human-computer interaction, such as the virtual assistant Siri, electronic medical record transcription in the medical field, and automatic TOEFL oral scoring.

The technical evolution of ASR can be divided into four iterative stages, and its methodological transformation reflects the coordinated development of computational linguistics and machine learning theory. Early ASR systems were based on dynamic time adjustment algorithms and used fixed template matching strategies to achieve isolated word recognition (1950s-1980s). Representative systems include IBM's "shoebox" [20]. The contribution of this stage is the establishment of a basic framework for speech time domain alignment, which laid the foundation for subsequent probabilistic modeling. In the statistical modeling stage (1990s-2010s), the joint architecture of HMM and Gaussian Mixture Model (GMM) [21] became the mainstream paradigm. The mathematical form can be expressed as:

$$\mathrm{P}(\mathrm{O}|\mathrm{W})=\prod _{\mathrm{t}=1}^{\mathrm{T}}\sum _{\mathrm{s}\in \mathrm{S}}{\mathrm{a}}_{{\mathrm{s}}_{\mathrm{t}-1}{\mathrm{s}}_{\mathrm{t}}}·\mathsf{{\rm N}}({\mathrm{o}}_{\mathrm{t}};{\mathsf{\mu }}_{\mathrm{s}},{\mathrm{Ʃ}}_{\mathrm{s}})$$

(2)

where $a_{s_{t-1}{s}_t}$ is the state transition probability, and N represents the GMM emission probability. Limited by the ability of GMM to represent nonlinear acoustic features, the rise of the deep learning paradigm was born. In the neural hybrid architecture stage (early 2010s), the deep neural network (DNN) replaced GMM to construct the HMM-DNN hybrid model [22]. Its hidden layer activation function $h_l=\sigma (W_l{h}_{l-1}+b_l)$significantly improved the acoustic modeling capability. From the late 2010s to the present, it is in the end-to-end architecture stage. The end-to-end model based on sequence-to-sequence (seq2seq) modeling abandons the traditional acoustic-language model separation architecture for direct modeling, see Equation 3.

$$\mathrm{P}\left(\mathrm{Y}|\mathrm{X}\right)=\prod _{\mathrm{t}=1}^{\mathrm{T}}\mathrm{P}\left({\mathrm{y}}_{\mathrm{t}}\right|{\mathrm{y}}_{1:\mathrm{t}-1},\mathrm{X})$$

(3)

where X is the speech feature, $Y$ is the target text [23], and $P\left(y_t\right|y_{1:t-1},X)$ represents the conditional probability of generating $y_t$ at the current time under the condition that the generated prefix sequence $y_1,y_2,\dots ,y_{t-1}$ is input into $X$. Nevertheless, G2P still has independent research value,

The necessity stems from the following technical demands: First, the data-intensive training requirements of the end-to-end ASR system conflict with the low-resource language scenario. The G2P model can help alleviate this dependence on large-scale data. Second, for the black-box characteristics of the end-to-end system, G2P can provide more fine-grained control and debugging. Third, for fields with special G2P rules such as medical terminology and scientific and technological vocabulary, the G2P model can be optimized.

Previously, self-supervised pre-training models (such as Wav2Vec 2.0[24] and HuBERT[25]) achieved speech representation learning by contrasting the learning objective function. Thanks to the self-attention mechanism and pre-training technology, ASR efficiency and accuracy have been further improved, which points out a new direction for the ASR field. In the end-to-end ASR architecture (Figure 3[26]), the technical implementation path can be formally described as:

$$\mathrm{Ŷ}={\mathrm{f}}_{\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{t}}\left({\mathrm{f}}_{\mathrm{d}\mathrm{e}\mathrm{c}}\right({\mathrm{f}}_{\mathrm{e}\mathrm{n}\mathrm{c}}{\mathrm{f}}_{\mathrm{f}\mathrm{e}\mathrm{a}\mathrm{t}}\left(\mathrm{X}\right)\left)\right)$$

(4)

The input of the system $X\in R^T$ is a continuous speech signal, usually a time-series audio waveform. It is converted into a time-frequency domain representation by the feature extraction module, and is typically implemented as a Mel-filterbank spectrogram [27]. This process is implemented through short-time Fourier transform [28] and Mel-scale mapping, and the expression is:

$${\mathrm{F}}_{\mathrm{d},\mathrm{t}}=\sum _{\mathrm{k}=0}^{\mathrm{K}-1}|{\mathrm{X}}_{\mathrm{t}}\left[\mathrm{k}\right]{|}^2·{\mathrm{M}}_{\mathrm{d}}(\mathrm{k})$$

(5)

where $M_d\left(k\right)$ is the number of the dth Mel filter group. The encoder $f_{enc}$ extracts context-sensitive representations of speech through hierarchical feature abstraction. Mainstream implementations include convolutional temporal encoders [29] and Transformer encoders [30]. The decoder $f_{dec}$ implements the probabilistic mapping from acoustic features to character sequences. Technical paths include the Connectionist Temporal Classification (CTC) decoding method: solving the optimal alignment path through dynamic programming [31]. The post-processing enhancement module $f_{post}$ improves the output quality (such as correction, word segmentation, and punctuation) by integrating linguistic constraints. The G2P component in its module can convert the output graphemes of ASR into phoneme sequences to verify whether they meet the expected pronunciation (such as polyphones and proper nouns). It can also infer the pronunciation of unregistered words through G2P to assist in generating more reasonable text.

3. G2P Method Evolution

The G2P methodology system has undergone three major paradigm shifts: modeling based on artificial linguistic rules (1950-2000s), modeling based on statistical probability (2000s-2010s), and the current technology system dominated by deep learning architecture (2010s). The brief history of its development is shown in Figure 4.

3.1. Pre-Pretraining Stage

Early G2P research was based on generative phonology theory and focused on the symbolic encoding of linguistic rules (1950s-1990s), such as the English Letter-to-Sound (LTS) rule system [32], alignment through manual seed algorithms, and decision tree technology [33]. The rule system approach is limited by the completeness of linguists’ prior knowledge. The expansion of speech databases (such as CMUdict) and the maturity of Bayesian statistical theory prompted G2P research to enter the stage of statistical methods (1990s-2010s), such as the G2P mapping method based on decision trees [34], the bidirectional G2P conversion method (G↔P) based on the joint n-gram model [35], the G2P model based on conditional maximum entropy and joint maximum entropy n-gram [36], the G2P conversion method based on HMM [37], the G2P conversion method based on the joint sequence model (Joint-sequence) [38], the G2P method based on conditional random field (CRF) [39], and the joint n-gram model conversion method based on failure transition (φ-transitions) [40].

The transition of the two methods presents the following significant features: first, the methodology changes from symbolism to statistics; second, from decision trees, n-grams to HMM, CRF, WFST, the model gradually enhances the modeling capabilities of long-distance dependencies, many-to-many alignment, and context sensitivity; finally, the core contradiction changes from being limited by expert knowledge to data sparsity (such as unregistered words and low-frequency words), and further to model efficiency and robustness. This technological change has promoted breakthroughs in downstream tasks such as speech synthesis and multi-language processing, and provided a historical mirror for the development of subsequent Neuro-Symbolic AI hybrid models.

The deep learning-based G2P technology has gradually established its dominance since 2010. Its core advantage lies in effectively capturing the nonlinear laws of letter-sound mapping through deep learning models. This technology can be traced back to 2008 when Beatrice Bilcu used three types of neural networks: multilayer perceptron (MLP), recurrent neural network (RNN), and bidirectional recurrent neural network (BRNN) to perform G2P tasks [41]. In 2015, Kanishka Rao et al. achieved significant progress in the G2P task by combining the Deep Bidirectional Long Short-Term Memory - Connectionist Temporal Classification (DBLSTM-CTC) model with CTC [3]. In the same year, Kai Yao's team systematically explored the performance differences of three neural network architectures, encoder-decoder LSTM, unidirectional LSTM and bidirectional LSTM (Bi-LSTM), in the G2P task. In 2019, S. Yolchuyeva's team successively proposed a hybrid architecture based on deep residual convolution [4] and a G2P method based on the Transformer structure. Although deep learning methods are significantly better than statistical models in terms of accuracy, they still face challenges such as homophone disambiguation and a surge in model parameters. To address these two challenges, researchers have proposed the following two model methods.

3.1.1. SoundChoice

In 2022, Alexey Ploujnikovt's team proposed a new G2P model called SoundChoice[42], with a structure as shown in Figure 5[42]. The grapheme-level features are encoded into continuous vectors through a trainable lookup table (green in the lower left corner of Figure 6), and are concatenated with the word-level semantic embedding of the pre-trained BERT (blue in the upper left corner of Figure 6) (pink in the middle left corner of Figure 6) to form a composite representation that combines character details and semantic context. The LSTM-based encoder (light green in the lower middle corner of Figure 6) extracts multi-level contextual information through bidirectional temporal modeling, and combines the gated recurrent unit (GRU) decoder and attention mechanism (light blue in the upper middle corner of Figure 6) to achieve accurate phoneme sequence generation. In terms of training strategy, a hybrid loss function was innovatively constructed. The encoding end uses the CTC loss function (middle right of Figure 6) to enhance the alignment capability, while the decoding end combines the negative log-likelihood loss with the homonym-specific loss to significantly improve the semantic disambiguation capability. Finally, a hybrid beam search mechanism using CTC and final prediction was adopted (bottom right of Figure 6). By integrating BERT word embedding and designing a new loss function, the model uses sentence context to improve the disambiguation accuracy of homographs, thereby improving the pronunciation accuracy in the speech synthesis system.

3.1.2. LiteG2P

In 2023, ZChen Wang et al. proposed a new G2P conversion model called LiteG2P[43], see Figure 6[43]. This model has a breakthrough design of a dual feature enhancement mechanism: first, the input letters are processed by phoneme length expansion, and each letter is expanded to the corresponding maximum phoneme mapping length to ensure the complete mapping of the phoneme sequence; secondly, the local position embedding technology GRU (pink in the lower right corner of Figure 6) is introduced to effectively solve the positioning ambiguity problem of repeated letters. After that, the letter position encoding vector (embed in the lower right corner of Figure 6) and the expanded letter embedding vector (embed in the lower left corner of Figure 6) are concatenated as a new feature vector and input into the subsequent feature extraction module (CNN and BiGRU in Figure 6). A binary mask matrix (0 or 1) is generated from a lightweight expert dictionary to suppress unreasonable phoneme mappings, and a pure CTC loss function is used to optimize the alignment process, which improves the speed and accuracy of the G2P task while keeping the model lightweight, making it suitable for cloud and mobile devices. Experiments were conducted on the CMUdict dataset, and the results showed that the LiteG2P model outperforms existing CTC-based methods with 10 times fewer parameters, and is similar to the Transformer-based Seq2Seq model with fewer parameters and less computation.

3.2. Pre-Training Model Stage

With the innovative breakthroughs in the Transformer architecture and its derivative models (such as BERT[5] and GPT), the transfer learning mechanism based on pre-training and fine-tuning has injected new development momentum into G2P technology. This technical paradigm captures deep phonological rules across languages through self-supervised pre-training, and then achieves knowledge transfer through task-oriented fine-tuning, which significantly improves the generalization ability and data utilization efficiency of the model. For example, in 2021, Y. Jia proposed the BERT model (Phonemes and Graphemes PnG BERT)[44], which takes the phoneme and grapheme representation of the text as input. By extending the BERT architecture and innovating the pre-training strategy, the model can learn the joint representation of phonemes and graphemes on a large-scale text corpus and fine-tune it in the TTS task. M. Řezáčková et al. used the pre-trained Text-to-Text Transfer Transformer (T5) model for G2P conversion in 2021 [45]. The T5 model was fine-tuned on English and Czech to adapt to the G2P tasks in these two languages. The T5 model achieved excellent performance on the G2P tasks in both English and Czech, especially in dealing with long words and homographs. Pre-training models have become the mainstream technical paradigm for current natural language processing and speech tasks due to their advantages such as low core data support, improved training performance, and strong cross-task and cross-language generalization capabilities. The paper focuses on the following pre-training methods

3.2.1. GBERT

In 2022, Dong Lu et al. proposed a BERT-inspired pre-trained grapheme model (Grapheme BERT, GBERT) [46], which uses only grapheme information and is trained on a large-scale specific vocabulary through self-supervised learning. The authors proposed two methods to integrate GBERT into the existing Transformer-based G2P model. The first method is to replace GBERT with the encoder in the original Transformer model and train the new model in an end-to-end manner. The second method, as shown in Figure 7 [46], integrates GBERT into the Transformer-based G2P model through an attention mechanism. In each encoder and decoder layer, additional GBERT-Enc and GEBRT-Dec attention modules are added to adaptively control the interaction between these layers and the GBERT representation. A dropout network is used in the structure to regularize network training. Experimental results show that under medium-resource and low-resource data conditions, the G2P model based on GBERT can improve performance, and the fusion of GBERT through the attention mechanism can reduce WER and PER compared with the Transformer model.

3.2.2. LLM

Large Language Model (LLM) has become a research hotspot in the field of natural language processing due to its massive knowledge reserve, powerful language understanding and generation capabilities, and continuously optimized adaptive characteristics.

M.F. Qharabagh (2024) conducted a study on Persian and compared the phoneme transliteration performance of 9 large language models (LLMs), including LLaMA 3.1 and GPT-4 [47]. The study found that the optimal model LLaMA-3.1 (version 405B) achieved an accuracy of 54.00% in polyphone transliteration, significantly better than traditional models (such as Epitran's PER of 47.53%). In addition, the author systematically explored a multi-stage prompting strategy: starting from the naive method of directly generating the International Phonetic Alphabet (IPA) (PER 31.61%), the author gradually introduced Finglish (Latinized) intermediate conversion, contextual learning examples, and dictionary prompts (embedded word phonetic options) and other techniques, successfully reducing the PER to 8.30%. The study also tested various combinations of these prompting strategies. For Persian, the most effective strategy is to combine dictionary prompts with LLM-based correction methods, see Figure 8. First, input a piece of text, and find the words in the dictionary to clarify the basic information of each word. Then give the operation instructions, ask to return the pronunciation content of the Persian text, and list the candidate pronunciations of some words, so that the model can choose the correct pronunciation. Input these contents into the big model, and the model processes them according to the instructions and candidate information. Finally, the model outputs the mapping result of graphemes to phonemes, completing the conversion process from input text to specific voice-character mapping.

In 2024, Dongrui Han et al. studied the problem of homonym disambiguation in G2P conversion [48] and proposed two prompt learning-based methods, namely one-shot prompt and in-context knowledge retrieval (ICKR). The experiment directly used GPT-4's one-shot prompt on the Librig2p dataset with a weighted average PER of 8.7%. Through fine-tuning models (such as Llama2-7B-chat and Gemma-2B-it), PER was significantly reduced to 5.5% and 5.2%. The latter ICKR system uses GPT-4 as an “AI linguist” to analyze the target word and its context in the sentence through prompts and retrieve the most matching pronunciation in the dictionary[48]. It achieves the lowest weighted average PER (4.9%) and the highest homograph accuracy (95.7%) on the Librig2p dataset, which shows that the latter method is superior to the former.

The ICKR algorithm flow is shown in Figure 9. The core is to dynamically match the correct pronunciation of polyphones in a specific context through semantic analysis of structured dictionaries and GPT-4. First, the input sentence is split into word sequences, and then the target word is detected to see if it is in the predefined homograph dictionary (the yellow color in the lower left of Figure 9). If it is not in the dictionary, the designed prompt is used to let the LLM directly generate the phoneme sequence of the word in combination with the sentence context of the target word (the dotted line on the left of Figure 9); If it is in the dictionary, it will continue to detect whether it is a homonymous word (yellow in the lower right corner of Figure 9). If it is a homonymous word, it will enter the LLM semantic analysis module (GPT-4), otherwise it will directly retrieve the unique phoneme (dashed line on the right of Figure 9). The LLM semantic analysis module (green in Figure 9) passes the target word and the sentence it is in to GPT-4, requesting analysis of its contextual meaning, part of speech, and related semantic information, and compares the output semantic information with the dictionary entries one by one to find the most matching entry. Each word in the homonym dictionary contains multiple sub-entries, each of which corresponds to a unique meaning, part of speech, and related semantic information. Finally, the target phoneme sequence is output.

3.3. Evaluation Indicators and Comparative Analysis

3.3.1. Evaluation Indicators

Phoneme Error Rate (PER) is a quantitative indicator for evaluating the performance of G2P systems. Its theory originates from the sequence alignment algorithm in the field of speech recognition. Its mathematical definition is the minimum edit distance (MED) between the predicted phoneme sequence and the reference phoneme sequence, also known as the normalized ratio of the Levenshtein distance to the length of the reference sequence [49]. It is formally expressed as:

$$\mathrm{P}\mathrm{E}\mathrm{R}={\displaystyle \frac{\mathrm{D}({\mathrm{P}}_{\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{d}},{\mathrm{P}}_{\mathrm{r}\mathrm{e}\mathrm{f}})}{{\mathrm{N}}_{\mathrm{r}\mathrm{e}\mathrm{f}}}}$$

(6)

where $P_{pred}$ represents the output phoneme sequence of the model, $P_{ref}$ is the reference phoneme sequence annotated by experts, and $D(P_{pred},P_{ref})$ is the editing operation function based on Levenshtein distance. This function calculates the minimum number of editing operations required to achieve sequence alignment through a dynamic programming algorithm, where normalization processing enables PER to be applicable to phoneme sequence evaluation of different lengths.

The word error rate (WER) is the core evaluation indicator for measuring the prediction performance of the G2P model. It reflects the overall performance of the system mapping by calculating the percentage of words that do not completely match the predicted phoneme sequence with the reference pronunciation [49]. The mathematical definition is:

$$\mathrm{W}\mathrm{E}\mathrm{R}={\displaystyle \frac{{\mathrm{N}}_{\mathrm{e}\mathrm{r}\mathrm{r}\mathrm{o}\mathrm{r}\mathrm{s}}}{{\mathrm{N}}_{\mathrm{w}\mathrm{o}\mathrm{r}\mathrm{d}\mathrm{s}}}}\times 100\mathrm{\%}$$

(7)

$N_{errors}$ is the number of incomplete matches between the predicted phoneme sequence and the reference pronunciation sequence, and $N_{words}$ is the total number of words in the reference pronunciation. Compared with the PER indicator at the phoneme level, WER pays more attention to the accuracy of the overall vocabulary mapping. For example, when a phoneme error occurs in word prediction ("cat" [kæt] → [kæp]), the PER is 33.3%, and WER will be directly judged as 100%.

3.3.2. Comparative Analysis

This paper collects the standardized evaluation results (PER/WER) of the G2P field on the CMUdict benchmark dataset in recent years, and conducts a longitudinal comparative analysis according to the time sequence of technology evolution, as shown in Table 1. The "N/A" in the table indicates that the author of the original document did not provide the corresponding indicator measurement results of the model. The data in the table show that the performance of the model architecture based on deep learning (LSTM, Transformer, GRU) significantly exceeds that of traditional statistical methods. The integrated architecture of 5 global attention encoder-decoder models [50] achieves breakthrough performance of WER=20.24% and PER=4.69%, which is a relative improvement of 18.1% and 20.5% respectively compared with the optimal traditional model (WER=24.7%, PER=5.9%). This performance gain is due to the probabilistic fusion of multi-model decision space and the coordinated optimization of the attention mechanism.

The single-layer LSTM architecture is limited by the unidirectional constraint of time series modeling and cannot effectively capture reverse contextual dependencies, and its performance is usually poor. For example, the PER of uni-directional LSTM[51] is 8.22% and the WER is 32.64%. However, by stacking bidirectional LSTM layers to build a deep time series structure, the optimization effect of PER=5.45% and WER=23.55% is achieved in a 3-layer configuration[51], indicating that deeper network structures can better capture contextual information and thus improve performance. Transformer models also performed well, especially Transformer 4x4[49]. As can be seen from the table, after the introduction of the attention mechanism into the model based on the Encoder-Decoder architecture (such as Encoder-Decoder Bi-LSTM with attention layer), its performance has been significantly improved, with lower values, which means that the attention mechanism has indeed effectively improved the performance.

In addition, the use of a multi-model ensemble architecture [50] further improved the performance, indicating that ensemble learning is an effective method in G2P tasks. The Token-Level Ensemble Distillation model [50] performed relatively well, indicating that this ensemble distillation framework, combining the advantages of multiple models and unsupervised learning, can break through the limitations of traditional G2P methods and improve robustness. In addition, the LiteG2P-medium model [43] achieves WER = 24.3% with only 1.2M parameters, proving that the efficiency bottleneck of traditional models can be broken through through lightweight architecture innovation.

4. Current Research Hotspots and Future Research Directions

4.1. Current Research Hotspots

The research hotspots in the G2P field have transitioned from rule-based and statistical models to technologies based on deep learning and pre-trained models, and are moving towards multilingual and cross-lingual directions. Pre-training and transfer learning, multilingual data sharing, and unsupervised learning will drive further innovation and development in this field in the future.

4.1.1. Challenges of Low-Resource Languages and Dialects

Factors such as language complexity, differences in phonetic rules, and scarcity of data resources have caused low-resource languages to face many challenges in the G2P field,as shown in Table 2. Many languages have complex many-to-many mappings between graphemes and phonemes, such as agglutinative languages such as Turkish and Finnish, and polysynthetic languages such as Inuktitut. To address this problem, Novak et al. proposed the WFST and n-gram grammar model in 2016 [58]. The use of WFST can efficiently handle complex input-output alignments, demonstrating its advantage in handling languages with complex morphological changes. The challenge of low-resource languages is particularly typical in the case of Mongolian, where annotated data is relatively scarce and it is difficult to train powerful neural network models. In response to this, Sun et al. proposed an integrated distillation method in 2019 to effectively compress multi-model knowledge into a small model[59], effectively improving model performance under data-scarce conditions. In addition, it is also possible to consider using a pre-trained model combined with transfer learning and fine-tuning on limited data, such as the PnG BERT model proposed by Yunhui Jia in 2021[44]. For languages with irregular mapping relationships such as Thai, Yamasaki used a neural regression model in 2022 to improve conversion accuracy[60]. Some graphemes have different pronunciations in different words, and phonemes cannot be inferred from spelling rules at all. This is also a major problem in languages where polyphones are common (such as English, Chinese, and Vietnamese). This requires the G2P model to accurately predict phonemes based on context. In this regard, the model proposed by Kim, Hwa-Yeon et al. in 2023 improved the phoneme prediction of polyphones by utilizing neighbor word information [61]. These research points indicate that future research on G2P conversion needs to consider language specificity and the application of model generalization capabilities under low-resource conditions.

4.1.2. Multilingual G2P

In the context of globalization, developing a G2P system that supports multiple languages and improving the performance of models on data-scarce languages through transfer learning and other means have become important research directions. In recent years, researchers have been committed to building cross-language representation models through transfer learning and other technical means, promoting the practical application of multilingual speech recognition and synthesis systems, and improving the phoneme conversion capabilities of low-resource languages. Thanks to the deep collaboration between the technology of large-scale pre-trained language models and the transfer learning technology system, this field has made significant progress. The representative research results are summarized as follows.

Ben Peters' team[62] took the lead in building a multilingual encoder-decoder framework based on the attention mechanism, proving that the shared parameter model can effectively utilize cross-language similarities to provide solutions for low-resource languages. David R. proposed the large-scale multilingual G2P system Epitran[63] in 2018. High-precision pronunciation conversion is achieved through rule-driven mapping files and post-processing technology, supporting 61 languages. Experiments show that in ASR tasks in 11 languages, Epitran significantly reduces WER (for example, Amharic from 58.6% to 57.2%), proving its effectiveness in low-resource scenarios.

Sokolov et al. (2020)[64] proposed a multilingual G2P model based on a bidirectional LSTM encoder-decoder, which distinguishes language rules by language ID and distribution vector, and achieved an average PER reduction of 7.2% on 18 languages (including low-resource languages), without degrading the performance of high-resource languages. In 2020, Mingzhi Yu et al. proposed a multilingual G2P model based on the attention mechanism Transformer architecture[65], using bytes as input representation to adapt to different grapheme systems. The authors used data from multiple European and East Asian regions to evaluate the performance of character-level and byte-level G2P. The experimental results show that the model using byte representation has a relative word error rate improvement of 16.2%-50.2% over the corresponding character-based model. In addition, the size of the byte-level model is reduced by 15.0%–20.1%.

In 2022, Jian Zhu et al. trained a large-scale multilingual G2P model based on ByT5 using a G2P dataset in about 100 languages[66]. Experiments show that in terms of multilingual G2P, ByT5 running on byte-level input significantly outperforms the token-based mT5[67] model. ByT5 can handle more than 100 languages, including Latin letters and East Asian languages (such as Chinese and Japanese), and improves the pronunciation prediction accuracy of low-resource languages through multilingual joint training. In the future, as the demand for low-resource languages grows and the generalization ability of models improves, research in this field will continue to deepen and further promote the inclusiveness of speech technology.

4.2. Future Research Directions

4.2.1. Optimization of Phoneme Mapping for Low-Resource Languages

The development of Seq2Seq models and pre-trained language models based on deep learning frameworks has significantly improved the performance of G2P systems in multilingual scenarios. However, due to the scarcity of labeled data and complex mapping rules, the performance improvement of low-resource languages still faces the following bottlenecks. First, the complexity of language morphology, agglutinative languages (such as Turkish) and polyglot languages (such as Inuktitut) have irregular mapping relationships between graphemes and phonemes; second, the lack of training data for low-resource languages limits the generalization ability of the model.

The following development strategies are proposed to address the above bottlenecks. In recent years, multilingual pre-trained models such as XLM-RoBERTa [68], mT5 [67], and ByT5 [66] have demonstrated excellent performance in NLP tasks. Their deep semantic representation space can effectively capture the phonological correlation characteristics between languages. Future research can explore the migration of such multilingual pre-trained models to low-resource language G2P tasks, further benefit low-resource G2P by performing zero-shot predictions on unseen languages, or provide pre-trained weights for fine-tuning, which helps the model converge to a lower PER than randomly initialized weights.

Applying multimodal fusion technology to low-resource language G2P can also optimize the phoneme mapping of low-resource languages. The introduction of auxiliary modalities (such as audio) can provide additional information and enhance the robustness of G2P. For example, as early as 2019, James Route and other scholars proposed that relying solely on IPA characters to represent pronunciation may not capture the subtle differences in speech. They innovatively introduced audio data as an auxiliary supervision signal and dynamically optimized the intermediate representation of source language characters through a multi-task learning mechanism to improve the accuracy and cross-language generalization ability of the G2P model [69]. Experimental results show that the multimodal model reduces PER to 2.46%, which is more than 65% lower than the unimodal model (7.05%), fully verifying the technical advantages of the multimodal model in enhancing robustness. In 2023, Manuel Sam Ribeiro et al. proposed using speech data to improve the G2P model for low-resource languages[70]. The authors first trained a baseline G2P model with a small amount of manually annotated pronunciation dictionaries, then used the model to generate pronunciation hypotheses for the target language speech corpus and combined it with multilingual data to train the phoneme recognition system. By decoding the target language audio, the forced alignment algorithm is used to find word boundaries and learn the pronunciation of new words. Finally, the learned pronunciation dictionary is combined with the original annotated data to retrain the G2P model. Experiments show that this method effectively improves the PER of the G2P system across languages and available data. Most current research focuses on the integration of text and audio modalities. In the future, this technology can integrate more modalities, such as text, speech, vision (such as lip movement) and even pronunciation physiological signals (such as tongue position, vocal cord vibration) to improve the model's ability to model pronunciation rules. See Figure 9. Each modality extracts features through an independent encoder, integrates features using joint training and multi-task learning strategies, and generates phoneme sequences through a decoder. In the future, it is also possible to solve the ambiguity or low resource problems of a single modality by building a multimodal dataset for low-resource languages.

4.2.2. Hybrid Architecture Innovation

The end-to-end model demonstrates high accuracy and strong generalization capabilities through a unified learning framework, especially when processing complex language mappings. However, it also has shortcomings, such as large data requirements, poor interpretability, and excessive reliance on hardware. Traditional G2P models such as the HMM model [37] and CRF [39] are mainly characterized by strong regularity and interpretability, and good adaptability to specific languages. However, they also have limitations, such as difficulty in processing complex languages, the need for manual design of features and rules, and high development costs.

The hybrid architecture that integrates the end-to-end model and the traditional rule-based approach can effectively break through the limitations of a single approach. In the Mongolian G2P conversion task, Zhinan Liu and other scholars innovatively proposed a collaborative mechanism of "rule priority + LSTM supplement", as shown in Figure 10[71]. This strategy first processes graphemes and phonemes with clear mapping relationships through the rule system, and then leaves the complex cases that the rule system cannot cover to the LSTM model for learning. Experimental data show that compared with the single LSTM model, the hybrid model achieves a significant decrease of 2.5% and 0.6% in the WER and PER indicators, respectively, which fully verifies the synergistic advantages of the high efficiency of the traditional rule system and the strong representation ability of the deep learning model.

The application of Neural Architecture Search (NAS) [72] technology in the G2P field is also worthy of special attention. This technology allows the system to dynamically select the optimal processing path based on the input features: for regular word-sound mapping, the traditional model is called; for complex nonlinear mapping, the end-to-end model is selected, and the performance of the selected model is evaluated and then the structure search is iterated. The process is shown in Figure 11. Currently, NAS has shown its application potential in NLP tasks such as text classification (Auto-Keras [73]) and question-answering systems [74], but it has not yet formed a complete theoretical system in the G2P field. Especially in low-resource scenarios, automatically optimizing the model structure through NAS to improve the accuracy and robustness of pronunciation prediction will become an important direction for technological breakthroughs. In summary, the deep integration of traditional models and end-to-end models has multi-dimensional value: first, it improves data utilization efficiency through complementary advantages, especially in scenarios where labeled data is scarce; second, the interpretability of the rule system and the strong fitting ability of the neural network form a benign complement; third, the hybrid architecture can enhance the model's adaptability to different language features. These characteristics collectively point to a research paradigm worthy of in-depth exploration - building an intelligent G2P system that is efficient, interpretable, and has strong generalization capabilities.

4.2.3. Mapping Optimization Based on Prompt Learning

Prompt Learning is a cutting-edge method in the field of natural language processing. It guides pre-trained language models (such as BERT, GPT, T5, etc.) to efficiently complete downstream tasks by designing specific forms of natural language instructions or templates, without the need for large-scale structural adjustments or complete fine-tuning of the model. Although this paradigm is still in the exploratory stage in the G2P field, it has shown breakthrough potential in low-resource language scenarios[47] and polyphone disambiguation[48].For example, the research conclusion of reference [47] shows that LLM is significantly better than traditional models in context-aware tasks, especially in low-resource language scenarios. The experimental results of reference [48] show that the GPT-4 version achieves the lowest weighted average PER (4.9%) and the highest homograph accuracy (95.7%) on Librig2p. The fine-tuned model (such as Gemma-2B-it) performs stably in processing unregistered words, but its semantic understanding ability is still weaker than GPT-4.

The above experimental results show that prompt learning has opened up a new technical paradigm for G2P tasks: (1) Hierarchical prompt engineering can enhance the grapheme reasoning ability of large models; (2) Knowledge-enhanced prompts can narrow the performance gap with fine-tuned models; (3) The fusion application of intermediate representations (such as Finglish) and external knowledge bases provides innovative ideas for low-resource language processing. In the future, we can further explore how prompt learning can enhance the cross-language transfer ability of G2P models and the collaborative optimization path of lightweight models and prompt learning.

5. Conclusions

G2P aims to convert grapheme sequences into phoneme sequences, and its performance directly affects the performance of TTS, ASR and other application fields. This paper systematically reviews the evolution of G2P technology and proposes research hotspots and future directions based on the challenges faced by the G2P field. By combing the evolution of methodology, it is pointed out that the early rule-driven methods relied on manually designed linguistic rules, which were highly interpretable but had limited generalization capabilities; statistical modeling methods significantly improved generalization capabilities through data-driven optimization; deep learning technology (such as end-to-end neural networks) further broke through the performance bottleneck, especially in complex contexts and multilingual scenarios. Current research focuses on multilingual G2P and low-resource language performance improvement. Future directions emphasize the coordinated optimization of technology and applications: on the one hand, it is necessary to improve the robustness of models in low-resource scenarios and explore small sample learning and unsupervised methods; on the other hand, multimodal fusion (such as text-speech joint modeling) technology may promote the application of G2P in edge computing. In addition, prompt learning and hybrid architecture innovation (such as combining rules and neural models) also show potential, providing new ideas for adapting to new languages or dialects.