AI Image Content Extraction Survey

1. Introduction

We are experiencing the rapid proliferation of AI in our everyday lives. We can now find AI applications in virtually every domain, from virtual assistants and AI researchers to cybersecurity tools, intelligent industrial automation systems, and much more. Driven by Generative AI, these tools are increasingly multi-modal. They are no longer constrained to text and can be used to generate images, videos, sounds, and voices. They can also be used to analyze multi-modal sources, such as comprehending text, summarizing image content, and intelligently editing videos. In particular, the ability to extract meaningful information from visual data has long been a central focus of computer vision research, dating back to traditional algorithmic Optical Character Recognition (OCR) methods. Initially, OCR was developed to recognize typed or handwritten text in scanned documents, converting them from pixel-based images into machine-readable formats, such as plain text, source code, and formatted text. Robotics applications have pushed the boundaries of what computer vision algorithms can achieve, but they have always been limited by the capabilities of the underlying algorithms.

In recent years, the rise of artificial intelligence and machine learning has led to significant advancements across many fields, including image content analysis. While traditional OCR was limited to structured, text-heavy inputs, such as books, forms, or printed documents, modern deep learning techniques have been introduced that can support more complex visual data.

One particular emerging application domain for AI image analysis is the automation of extracting complex descriptions of both text and structure from technical diagrams and other technical illustrations. These types of illustrations are commonly found in engineering, software development, and electronics, and contain a mixture of textual, symbolic, and graphical elements. Increasingly, AI models are being used to extract not only the textual content but also vital structural and relational information from these diagrams, enabling greater automation, analysis, and digital transformation. This capability is critical for applications such as Retrieval Augmented Generation (RAG) [1], where Large Language Models (LLMs) are used together with information extracted from documents to answer specific questions related to these documents. For any RAG pipeline, it is critically important to capture all content of the provided documents, not just their textual aspects. It also greatly aids in question-answering processes, enabling the generation of full and accurate answers to user questions by extracting relevant text and visual information from technical documents. This also supports accessibility applications, such as screen readers, in accurately describing visual content, as well as many sophisticated and AI-driven automation applications. Thus, the ability to extract visual details from technical illustrations within these documents, alongside the textual content, is crucial for accurately answering user questions, informed decision-making, and more.

This image content extraction survey focuses on four specific, widely used types of technical illustrations: flowcharts, block diagrams, electrical schematics, and timing diagrams. These are important elements in technical documentation, user guides, network protocol specifications, software and algorithm documentations, and many other types of documents crucial to virtually all scientific fields and end-user applications. Our aim for this survey is to provide an overview of the current state-of-the-art in image content analysis for technical illustrations, including their current capabilities, shortcomings, and suggested areas of further research.

Flowcharts and block diagrams excel at representing complex systems in a clear and easy-to-understand format. They can be used to describe processes, systems, data flow, computer programs and algorithms, document structure, workflows, and decision-making. Also, closely related to flowcharts are State Machine Diagrams and more structured forms such as swimlane flowcharts. Both diagram types are widely used in fields such as engineering, electronics, and telecommunications, where systems need to be broken down into smaller components for analysis, design, and troubleshooting. Figure 1 shows a typical example of a flowchart, and a simple block diagram is shown in Figure 2.

When it comes to electrical systems and electronics, scientists and engineers often rely on electrical circuit diagrams, which provide a detailed and standardized visual representation of circuits and electronic components, as well as the connections between them, to achieve a specific function. Circuit diagrams serve as blueprints for electrical systems, converting abstract design concepts into a visual representation that is easy to understand. A circuit diagram that describes a microcontroller (MCU) pin-out is shown in Figure 3.

Finally, this survey also specifically reviews approaches related to the analysis of timing diagrams. They are a type of visual representation that describes the behavior of signals or events over time, for example, in digital circuits, systems, or communication networks. Timing diagrams provide a clear and organized way of observing how various signals evolve and interact over specific intervals. An example timing diagram is shown in Figure 4, illustrating the relationship between the clock, reset, address, data, write enable, and ready signals.

These diagram types all pose unique challenges. While OCR can typically be applied to extract the textual content, a comprehensive representation of these diagram types encompasses far more than simply text. It needs to include shapes, color, and the structural relation between all these elements, in order to fully capture their content and meaning.

OCR itself has evolved over the years, progressing from a purely algorithmic approach to incorporating various Deep Learning techniques.

Despite these improvements, significant limitations remain. One major challenge arising from the use of AI models is hallucination, where AI models generate text or structures that are not grounded in actual content. Another challenge is the lack of contextual understanding. Many image content analysis systems still struggle to accurately interpret connections such as directional arrows, grouped symbols, or hierarchical structures, all of which are crucial in understanding the intent and logic behind technical diagrams. These can lead to flawed outputs, misinterpretations, or even cause system failures when used in automated pipelines.

This paper presents a comprehensive survey and historical analysis of image content extraction techniques, including OCR, applied to the four major diagram types mentioned above. It includes both traditional and AI-driven approaches, highlighting their methodologies, strengths, and specific limitations. By analyzing a wide range of models and algorithms, we assess their effectiveness in extracting both textual and structural information. We also evaluate their readiness for deployment in high-accuracy, real-world environments.

The remainder of this paper will be organized as follows. Section 2 will go over the steps on how this survey was conducted. In Section 3, we present a brief review of the evolution of image content analysis approaches and related evaluation techniques. Section 4 represents our survey, analyzing and discussing the various research papers we included for this survey, exploring their inspiration, techniques, and results. Finally, in Section 5 we present our conclusions and suggest future research directions.

2. Survey Methodology

The literature review method used in this study was designed based on a self-directed process to suit the specific aims of the research. We first conducted numerous literature searches across a wide range of academic publishing platforms. Our search focused on available resources on the following platforms:

• IEEE Xplore,
• ACM Digital Library,
• SpringerLink,
• Elsevier,
• MDPI, and
• Google Scholar.

From these searches, we then collected and compiled all relevant peer-reviewed papers within the scope of the topic. The literature search began with the use of targeted keywords, including "flowchart," "optical character recognition" (OCR), and "extraction," to retrieve publications relevant to the research scope. During this initial phase, no restrictions were applied to the publication date, thereby maximizing the inclusivity of the search results. This approach yielded a total of 210 research papers for preliminary consideration. The search strategy and retrieval process are visually summarized in Figure 5.

In our pre-filtering step, we then used carefully selected keywords and search techniques to manually identify and filter publications based on relevance, quality, and publication year. Once selected, the papers were reviewed in depth to extract key insights, common themes, and understand the methodological approaches used across the literature. We also, at that time, established relations between these papers. These findings helped establish a strong foundation for the study, ensuring that our work was informed by a broad and up-to-date understanding of current research in the field.

Following the broad initial search and pre-filtering, a more rigorous and focused screening process was implemented to enhance the relevance and modernity of the literature base. Specifically, additional papers were carefully selected for inclusion that were published between 2020 and 2025, ensuring the inclusion of the most recent advancements and findings highly pertinent to the research topic. In addition to this date criterion, further inclusion parameters were applied to align the selected studies with the specific scope and objectives of this paper, with a particular emphasis on research in information retrieval and text recognition from diagrammatic sources.

This systematic refinement culminated in a curated set of 56 research papers, which serve as the foundation for the detailed analysis and discussion in this survey. Among those, 37 papers drive the technical discussion presented below, with other papers providing supporting insights and closely related techniques discussed in relation to the core papers.

During our literature search, we identified a survey published in 2024 [2] that provides an in-depth analysis of advancements in text and shape extraction specifically from Piping and Instrumentation Diagrams (P&ID). Thus, for our survey, we excluded this diagram type and instead refer the reader to the referenced publication, as that paper comprehensively covers the significant progress in the domain of P&ID schematic analysis.

3. Background

3.1. Early Image Content Extraction Approaches

Over the years, numerous methods have been developed for extracting textual and visual information from documents and images. Earlier approaches focused heavily on OCR, which aims to detect printed or handwritten text and convert it into machine-readable formats. As image processing improved, shape and pattern recognition methods emerged, allowing systems to detect and recognize shapes, diagrams, and geometric figures. In recent years, the emergence of deep learning methods for computer vision tasks, including Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), has dramatically improved the accuracy and versatility of text and image content extraction techniques. These advancements have enabled the development of more intelligent systems, capable of handling complex layouts, handwritten variants, and providing contextual understanding.

Traditional OCR techniques are the earliest methods for text extraction from images. They typically follow a multi-step process comprised of [3]:

• Pre-Processing involves noise reduction, binarization, and skew correction to enhance image quality.
• Segmentation separates text regions from non-text elements, and individual characters or words are isolated.
• Feature Extraction identifies key characteristics of each symbol, such as edges, corners, and stroke widths.
• Classification and Recognition compares the features to already trained character models to output machine-readable text.
• Post-Processing is often utilized by modern OCR systems to correct recognition errors using language models.

While OCR is primarily limited to text recognition, research into shape recognition techniques focuses on the detection and extraction of symbols, icons, diagrams, and geometric figures. The steps associated with shape recognition typically involve [4]:

• Edge Detection or Contour Extraction is utilized to identify shape boundaries.
• Shape Description computes area, perimeter, or moments.
• Feature Matching uses template matching or feature descriptor techniques.
• Classification maps the detected shapes to predefined categories.

These methods are the cornerstone of content extraction from complex images, which comprise technical diagrams, engineering drawings, flowcharts, and other intricate diagrammatic representations.

The advent of deep learning has enabled significant advances in image content extraction, particularly for complex technical imagery. For example, CNNs [5] are comprised of three primary types of layers, which are typically stacked to extract and combine features from input data before producing a resulting value. By optimally structuring CNNs using these different layer types, the resulting CNNs excel at a variety of computer vision tasks. The three distinct layer types are:

• Convolutional layers apply a series of learnable filters or kernels that perform convolutional operations over the input image. These filters detect edges, textures, and color gradients.
• Pooling layers perform a down-sampling operation on the feature maps, reducing their spatial dimensions while retaining the significant information.
• Fully Connected (FC) layers operate after the feature maps are flattened into a one-dimensional feature vector. Flattening converts the multidimensional feature maps into a single continuous vector. These layers integrate the extracted features and compute the final prediction.

The convolving capabilities of CNNs make them ideal foundational approaches for processing spatial data such as images. For sequential data, such as text, however, RNNs [6] often outperform CNNs by modeling contextual dependencies through the process of remembering and incorporating prior input values. Hybrid versions of these frameworks also exist, such as Convolutional Recurrent Neural Networks (CRNNs), which combine CNN feature extraction with RNN sequence modeling, enabling end-to-end image content analysis. [7]

More recently, Transformer-based models emerged as a dominant architecture, enabling modern Large Language Models (LLMs) and other advances in Deep Neural Networks. LLMs, in particular, have rapidly become a cornerstone of everyday life. Transformers are capable of capturing long-range dependencies and contextual relationships across an entire document. Transformer architectures also include non-textual adaptations such as Vision Transformers (ViT) [8] and TrOCR [9]. Transformers can often outperform RNNs in recognizing complex layouts. [10] Vision Transformers were introduced to apply the Transformer paradigm to image analysis. They provide the foundational component for building Vision Language Models (VLM), which are multimodal LLMs capable of generating output based on textual and visual content. The initial step in a VLM typically involves vision encoding using ViT (to represent image data using feature vectors), followed by language encoding (to represent the text such as the user query as feature vectors), followed by a projection and integration step (to align both of these embeddings within a common encoding space combined with a Transformer model), and finally the output step that generates the text-based result.

3.2. Existing Evaluation Approaches for Content Extraction Methods

With the development of these architectures, it became essential to establish and implement rigorous evaluation and scoring mechanisms to quantitatively and qualitatively assess the performance of each such framework, particularly under challenging conditions such as noisy inputs, complex layouts, as well as dense or overlapping content, in order to determine their robustness, reliability, accuracy, and suitability for real-world applications. There are multiple methods that assess these frameworks, with one prominent example being Recall-Oriented Understudy for Gisting Evaluation (ROUGE-1) [11]. ROUGE-1 is a metric for evaluating text summarization that measures single-word (uni-gram) overlap between a generated summary and one or more reference summaries. It uses Precision, Recall, and F1-Score to quantify performance. The scores range from 0 to 1, where 0 represents no matching of words and 1 represents perfect matching of words.

Another scoring technique used for recognition evaluation is the Bilingual Evaluation Understudy (BLEU) scoring [12]. BLEU is used for evaluating the quality of the extracted text. The BLEU scoring scale ranges from 0 to 1, where 0 represents the lowest quality and 1 the highest [13]. Some variants of BLEU scoring include CodeBLEU [14], which specifically targets the evaluation of program code extraction, and BLEURT [15], which is based on the Bidirectional Encoder Representation from Transformers (BERT) to mimic a more human-like evaluation approach for extractions. It uses the same scoring scale as the original BLEU. BLEU-4 [16] is primarily used for evaluating extractions, utilizing a comparison to human translations. BLEU-4 is named thus because it utilizes 4 words to calculate precision, using a scoring scale that once again ranges from 0 to 1. Another scoring approach is the "mean Average Precision (mAP)" [17], which evaluates both the localization and classification performance. While precision is employed to measure the accuracy of the model’s positive predictions, recall measures the completeness of the model’s predictions, and the F1-score represents the mean of precision and recall [18].

4. Diagram Analysis

4.1. Analysis Overview

Since this paper focuses on recent developments for diagram analysis, each paper we are surveying in this section builds upon an extensive foundation of substantial prior work. We fully acknowledge and appreciate this deep historical grounding of the cited works and encourage the reader to delve into these works for a deeper understanding of these techniques. The scientific works we are focusing on in our survey represent a cross-section of the state-of-the-art in image content analysis for the four specific technical illustration types we selected. The authors of these respective papers build upon the extensive prior work to explore new approaches and improvements that push the boundaries for analyzing diagrammatic images. These techniques are critical in numerous application areas, in particular as information extraction tools for AI workflows, multimodal document understanding, document digitization, and literary curation efforts.

The provided analysis examines each article for three key review criteria:

• inspiration, which covers the foundational ideas that influenced the study;
• techniques, where we explore specific methods and tools used in the research, along with their results; as well as
• future directions, which outline the areas the authors wish to enhance or explore further.

Our analysis is divided further into the structure of the image content being extracted, as different methods have been evaluated against flowcharts, block diagrams, electrical circuit schematics, and timing diagrams.

4.2. Flowchart Analysis

4.2.1. Review of Identified Papers

In 2020, the authors of [19] proposed using region-based segmentation to convert flowcharts into graphs, introducing a custom approach for precisely extracting 2D elements from flowcharts. They discussed prior attempts, such as [20], which explored flowchart extraction from digital images using a neural network, and [21], which developed Flow2Code, a mobile app that converted photographs of hand-drawn flowcharts into code.

While Flow2Code’s original text extraction was performed using offline sketch recognition and computer vision, to advance flowchart text extraction for increased precision and structure recovery, the authors of [19] proposed and evaluated their own technique. The authors followed a traditional preprocessing step using the Otsu method [22] for image thresholding. Feature extraction was achieved through region-based segmentation to separate text and non-text regions, also referred to as regions of interest (ROI) [23]. Morphological labeling was then used to extract text from corresponding regions, and shape identification was performed by analyzing contour vertices to determine flowchart shapes. The flowcharts were ultimately converted into graph-like structures composed of nodes, labels, shapes, and edges. A final plot was generated to visualize the graph. The Open Source Computer Vision Library (OpenCV) [24] and Python were used for implementation.

Their experimental evaluation involved a small dataset of only 25 different flowcharts, which were categorized and converted into graph representations. Each element of the flowcharts was parsed and examined. The approach successfully handled more complex parameters such as scaling, dashed or directed lines, and colored shapes, and achieved 90% accuracy during testing. The content extraction process effectively transformed flowcharts into structured graphs. This result demonstrates the effectiveness of the method in handling diverse graphical features. The authors noted that future work would address greater flowchart complexity and explore alternative extraction methods.

The authors of [25] created their own method of flowchart extraction called Graph Recognition Convolutional Neural Network (GRCNN) in 2020. It utilizes FlashFill [26], an algorithm famously used in Microsoft Excel, which takes a set of given input-output text pairs and determines the user-desired text transformation from those samples. This transformation can then be applied to other input text samples. In GRCNN, a CNN is utilized for image content extraction, with an edge network used to transform the feature vector from the CNN into an edge description of the given flowchart. Using this approach, the nodes of the flowchart are identified and also described.

A custom dataset containing 2490 flowchart images was used for the evaluation of GRCNN, with flowcharts ranging from 3 to 6 nodes and 0 to 2 decisions. These flowcharts were created using Graphviz [27]. An Intel i7 CPU and an NVIDIA GTX 1070 GPU (Graphical Processing Unit) were used for this evaluation. Sequence accuracy was used as the metric in this evaluation of GRCNN’s performance. The edge and sequence extraction scores were 94.1% and 90.6%, respectively. However, for node extraction, a relatively low accuracy score of 67.9% was achieved. Consequently, the overall graph extraction accuracy (comprised of edge, sequence, and node extraction) was only 66.4%. This indicates that significantly more research was needed to consistently and accurately extract the structure of a flowchart.

Another approach to flowchart extraction was published in 2020 in [28]. It aims to improve upon other flowchart extraction approaches, such as [29,30], by facilitating the extraction of hand-written flowcharts instead of programmatically generated flowcharts. Images are first acquired using a camera or scanner. It then utilizes the Otsu method to transform each input image via threshold-based binarization, followed by morphological operations, dilation and application of the Ramer-Douglas-Peucker method [31] for object and line detection, as well as beam search [32] for the hand-written text detection. For evaluating their approach, a small dataset of 20 flowcharts was used. The average word and character accuracy are 66.1% and 86.52%, respectively. While the resulting scores are relatively low, they nonetheless illustrated the potential of this technique and warranted further exploration. The authors themselves stated that their future work aimed to improve the accuracy of flowcharts containing loops and to create a more diverse and complex flowchart.

Another study conducted and published in 2020 utilized deep learning and OCR to convert flowcharts into graphs [33]. The authors referenced earlier works, such as [34], which employed dynamic programming for stroke organization within a Random Forest framework, and [35], which converts flowcharts into source code. They also discussed [20,36]. The first step in their presented approach involved image preprocessing to obtain the flowchart in a planar perspective. This is followed by applying OpenCV contour detection. For shape recognition, the authors first applied Tiny YOLOv3 [37] in conjunction with Darknet [38]—a C and CUDA-based deep neural network (DNN) framework commonly used for implementing YOLO. YOLOv3-SPP (YOLOv3-Spatial Pyramid Pooling) [39] was used for lines and arrow recognition, and Google Cloud’s Vision API was used for OCR-based text extraction. Graph Models (GM) were applied to represent the flowcharts. The final step converted this GM data into Flowchart Graphs (FGs). All experiments were implemented using Python on Ubuntu 18.04.

For the performance evaluation of this work, a new dataset comprising 161 images was created, consisting of 38 digital and 123 hand-drawn flowcharts. Using mAP, their presented method achieved an average precision of 99.82% for shape recognition and 88.14% for arrow and line detection. Although these results are promising, the authors emphasize that further improvements could be achieved by using larger datasets and more advanced frameworks. This approach nevertheless demonstrates significant potential for accurately extracting flowcharts. The authors further suggest that it would be worthwhile to develop a mobile application capable of capturing flowcharts and translating them directly into code.

In 2022, [40] proposed FlowChart to Code (FC2Code), a system that extracts text from flowcharts, converts it into pseudocode, and generates executable source code from it. Prior work, such as [36], introduced RAPTOR, a visual programming tool that helps students translate flowcharts into pseudo-code. The FC2Code system used a custom dataset of 320 flowcharts with corresponding source code, of which 50 were used for testing. Structure recognition was applied to convert flowcharts into pseudocode, while Bidirectional LSTM [41] and Graph Attention Network (GAT) models [42] extracted textual information. To maintain information integrity, Reversed Flowchart, GAT, and the TranX decoder [43] were utilized to convert pseudocode into executable code. Results were compared against LSTM and Transformer [44] baselines.

Performance was evaluated using the BLEU score, where the proposed method achieved 55.68% compared to 38.25% for LSTM and 44.05% for the Transformer baseline approach. As shown, this method significantly outperforms shows both LSTM and Transformer networks. Although effective, the authors note that further work is required to enhance precision and address challenges in integrating flowchart structure into pseudocode.

In 2023, the authors of [45] introduced a public flowchart dataset named FloCo and a method, FloCo-T5, that converts flowcharts into Python code. This method was evaluated using a dataset containing 11,884 flowcharts, including both digital and handwritten ones. Their presented approach is based on the work in [46], another flowchart dataset but not publicly available, in [21], which converted hand-written flowcharts into executable code, as well as the work in [47] – the seminal work that introduced Bidirectional Encoder Representations from Transformers (BERT) for NLP language representation. Closely related is the work in [48], which digitizes handwritten flowcharts using Faster R-CNN and OCR methods, but only supports hand-drawn images without text. A Hough transform (HT) [49] was employed for shape recognition, EasyOCR was used for text recognition, while Code-T5 [50] was employed for source code generation. Handwritten text detection was handled using CRAFT [51], and TrOCR was used for text recognition.

To support their evaluation efforts, data augmentation was performed to increase the size of the available dataset, and a BERT-like Code-T5 strategy was employed to enhance the structural and semantic understanding of code and flowcharts. OCR fine-tuning with positional encoding further improved text alignment. A single NVIDIA A6000 with 48GB was used for testing. The proposed method achieved a BLEU score of 21.4 and a CodeBLEU score of 34.6. Although a large dataset could be used for this performance evaluation, the authors note that extracting flowcharts into Python source code requires further study. Future efforts thus aim to reduce computational demands and design models better suited for flowchart-to-code translations.

In 2024, Darda and Jain introduced a system that extracts flowcharts and generates Python code [52]. It builds upon the work in [40,53]. This work was influenced by the preceding efforts in [54], which introduced and presented S-DistilBERT, along with a version of Flow2Code published in 2019 [55] used for processing pictures of handwritten flowcharts and converting them into executable code. Darda and Jain utilized EasyOCR and OpenCV for text extraction, and employed Llama 2 LLM [56] for Python code generation via an API-connected interface. Python code generation was successful for 75% of the flowcharts, and the method produced accurate source code, even when 62% of the flowcharts had added noise. However, the authors also note that improvements are needed for more accurate text extraction and to provide multilingual capabilities.

As shown by these works, the advancements in LLMs have enabled new techniques for flowchart information extraction, which prove to be quite effective across a range of different imagery. However, we can also observe that significantly more work is needed to improve accuracy and precision. The following works aim to address this drawback in the preceding techniques.

In 2025, our team conducted limited experiments to evaluate the performance of a multimodal VLM when tasked with describing a flowchart, aiming to assess the viability of directly utilizing VLMs for such tasks without preprocessing. In these experiments, we utilized Google Gemma-3-27B, and prompted it to describe flowcharts, such as the one shown in Figure 6, with a portion of the resulting description shown in Figure 7. We are highlighting a particularly obvious error produced by the VLM, falsely denoting a loop back to the Start node, rather than a branch to the End node that is actually shown in the flowchart. Such erroneous information would result in significant problems for downstream tasks that depend on the extracted data, and demonstrates that, despite recent advances in LLM capabilities, significant challenges persist.

Another study in 2023, [54] proposed a custom S-BERT model named S-DistilBERT, inspired by [57], which extracted flowcharts from online sources, and [58], which is a lightweight variant of BERT, [28,59] is also referenced. Their experiments utilized 50 handwritten flowcharts, each with manually created pseudocode and descriptions, for validation purposes. The DistilBERT model was used for preprocessing the data, and a third-party text OCR app was used for text recognition. This resulted in an overall accuracy of 75.59%. This work once again demonstrates that, compared to other proposed methods, Transformer-based approaches perform very well for these types of applications. The authors note that recursion in flowcharts may be looked at in the future.

Several studies published between 2024 and 2025, such as [60,61,62], all reference FlowLearn [63] to enhance dataset annotation and architectural performance. For example, the authors of [60] introduced TEXTFLOW and utilized the datasets FlowVQA [64] (2,272 flowcharts) and FlowchartQA [46] (1 million images) for their performance evaluation. The presented TEXTFLOW combines MERMAID for annotations, Graphviz for node and edge definition, and PlantUML [65] with Visual Language Models (VLMs) [66] such as Qwen2-VL [67] and GPT-4o for logic understanding. The system utilized both VLMs and LLMs to construct flowchart structures and was trained on four NVIDIA A100 GPUs. FlowLearn’s reported results demonstrated strong text extraction performance with an F1-score of 88.23%, with their suggested future work focused on reducing computational complexity. The approaches presented in [60,61,62] demonstrate similar results, overall.

Inspired by [53], the authors of [68] in 2024 proposed GenFlowchart, a flowchart extraction method influenced by earlier transformation algorithms such as the PAD approach [69], along with the technique presented in [70] that iterates through flowcharts with dialogs. Image preprocessing was conducted using grayscale conversion, followed by Pytesseract OCR [71] configured with the Page Segmentation Mode (PSM) [72] selected to be mode 4, and the OCR Engine Mode (OEM) configured for mode 3. Additionally, the Segment Anything Model (SAM) [73], and PyMuPDF [74] were integrated for text and shape recognition.

The study evaluated 550 flowcharts based on text extraction and the description of the flowchart, with scoring methods such as BERT-P, BERT-R, BERT-F1, and Cosine Similarity. Their presented approach achieved an overall accuracy of 86.49% using BERT-F1 for instruction-based prompting. Based on the previous methods discussed, it is evident that using generative AI can enhance the overall effectiveness of flowchart extraction.

In [75], the authors present FR-DETR (Flowchart Recognition Detection Transformer)–an end-to-end multitask network. Prior work had used text–graphic separation and OCR-based extraction [76], while [77] and [78] employed DETR for object recognition. FR-DETR, on the other hand, integrates CNNs for feature extraction and Transformers for encoding and decoding, with the obtained results further improved using a Feedforward Neural Network (FNN) [79]. Finally, a coarse-to-fine strategy is employed for detecting arrows and lines.

Using CLEF-IP [80] and a custom dataset of 1,000 flowcharts that were randomly divided into 800 training images and 200 testing images, FR-DETR achieved 98.7% precision and 98.1% recall on CLEF-IP, and 94% precision and 93.1% recall on the custom dataset. This once again illustrates the benefits of using transformer-based approaches for extraction when compared to other, more traditional, methods. FR-DETR’s computational requirements, however, preclude its use in mobile on-device applications. That area remains for future research efforts.

Another image content extraction method for hand-written flowcharts was introduced in 2022 by [59], building on the work shown in [81,82], among others, with the goal of improving upon their overall performance and accuracy. In their approach, each flowchart is represented in graph form using Instance GNN (InstGNN) [83]. Feature extraction is performed to simplify the data, and a GAT is used for learning and updating the nodes and edges of each flowchart. Training was conducted using the Deep Graph Library (DGL). Edge classification and node learning were conducted at the postprocessing stage, and symbols were verified using ResNet-18 [84]. A custom dataset called CASIA-OHFC, containing 2957 hand-drawn, annotated flowcharts, was used for testing. The Stroke Classification Accuracy (SCA) was used for evaluation, and the proposed method achieved a score of 76.44%. To further improve their approach, the authors suggest exploring various neural networks for feature extraction and the real-time identification of strokes and symbols.

A 2024 study introduced LS-DETR [85], targeting handwritten flowchart recognition. It builds upon [75] and earlier diagram recognition work published in [86] along with FLoCo-T5 [45]. [25,40,48,54] is also referenced. LS-DETR integrates CNN encoders and DETR decoders with DPText-DETR [87] for angled and polygonal text, and Enhanced Factorized Self-Attention (EFSA) for precise shape prediction. FLoCo-T5 was used to convert recognized flowcharts into source code. Using four NVIDIA A100 GPUs, the authors evaluated their approach using mAP on multiple datasets (ImageNet, ICDAR19 ArT V3-Flowchart, GFTotal-Text, and a custom 85-flowchart set), demonstrating improved accuracy over prior methods: With the use of multiple datasets and their proposed transformer-based algorithm, the Average Precision (AP) at 50% Intersection over Union (IoU) is 99.18%. The authors demonstrate that their approach can significantly improve both accuracy and precision compared to previous efforts. However, improvements are needed for enhancing handwritten flowchart-to-code translations.

Flowchart extraction has recently received increasing attention, with 2025 seeing the highest number of methods developed, as illustrated in Figure 8. This view was generated using Litmaps, showing the connectedness of the papers discussed in this section, as well as miscellaneous related works organized in the blue ring.

In 2025, Omasa et al. [61] focused on improving the arrow precision and noise reduction provided by GenFlowchart [68]. Their work expands upon GenFlowchart, along with [63], and also discusses their work in the context of FloCo-T5 [45] as one of the foundations and inspirations for their own technique. [25] is also referenced in their paper. Using OCR for text extraction and DAMO-YOLO [88] for object detection, they combined OCR and detection data to identify arrow start–end pairs, constructing graph structures and performing reasoning analysis with GPT-4o’s multimodal capabilities. On a set of 30 flowcharts, the method achieved a 88.9% OCR accuracy, as assessed and compared to results obtained via human evaluation. However, additional work is needed to enhance graph-based prompting, integrate confidence-aware reasoning, and refine evaluation methods for robust and generalizable real-world deployment.

Another VLM-based approach was published in 2025 by Soman et al. [62]. Their flowchart extraction method builds upon 2024’s FlowVQA [64] and FlowLearn [63], and realizes graph structures from images using a fine-tuned version of a Qwen2-VL VLM. To evaluate the performance of their approach, the authors utilized the FlowLearn dataset, with graph performance measured via Graph Edit Distance (GED) [89]. Additionally, they implemented a Retrieval-Augmented Generation (RAG) system and evaluated it using TeleRoBERTa [90] and bge-large [91]. Using the top-k retrieval metric performance evaluation, which spanned 1,586 images, the system achieved an accuracy of 71.91% using TeleRoBERTa. The authors discussed their plans to extend their approach by implementing JSON structures through RAG, along with expanding to other forms of technical illustrations, such as Unified Modeling Language (UML) diagrams.

The author of [92], published in 2025, leveraged LLMs for flowchart extraction using the C4 model dataset. Their work builds upon [52,93,94]. Here, they first pre-process images to remove textual and visual noise. They also implement the LLM-based Tesseract OCR framework for diagram enhancement. RAG is used to enable Tesseract OCR to utilize external documents, code, and system details to create diagrams that are accurate, up-to-date, and match the real system. Datasets were created using Mermaid, PlantUML, Structurizr [95], Graphviz, and Diagrams as Code. Their evaluation was conducted with a focus on knowledge extraction, standardization, toolchain support, flexibility, and ease of use. The code generation post-extraction demonstrated very high compilation success rates, reaching 100% for Mermaid flowcharts, for those flowcharts where content extraction succeeded prior to Mermaid code generation. Future work will focus on developing an Agentic AI to dynamically determine diagram types, enhancing usability and deployment through improved UI and integration, refining document structuring with flexible rules and additional LLM processing, strengthening diagram compilation with multi-DSL validation and feedback loops, and expanding the dataset to improve generalization and output quality.

Another 2025 study employed third-party APIs combined with LLMs [96]. It utilizes automated computing [97,98] involving API calls to online image processing services, followed by leveraging GPT-4 Vision for content extraction, and finally using AI-based code generation in multiple programming languages to turn flowcharts into code. Using GPT-4, the system achieved an Artificial Analysis Quality Index score of 77, and a 92% efficiency in human-evaluated code conversion. This proposed method demonstrates that with more capable LLMs, the task of accurately extracting information from flowcharts can be improved. It also demonstrates that this task can be built upon online services to offload the processing requirements from the actual devices, thereby widening the possible target platforms and application domains. Similar to many other surveyed papers, however, the authors note that additional work is needed to improve accuracy and incorporate support for handwritten diagrams.

Several papers published in 2025 demonstrated that multi-modal LLMs can compete with, and in some cases outperform, purpose-built image content analysis frameworks. One such 2025 study [99] incorporated LLMs for recognition and code generation, building upon earlier work on diagram digitization [100]. In their approach, preprocessing is conducted first to scramble the text in the diagrams due to copyright concerns. Manual cropping enables enhanced precision before OCR-based abbreviation recognition and GPT-4o LLM-driven code generation. Using the OPC UA and PROFINET datasets, with 15 and 80 diagrams, respectively, the model achieved 63% and 45% edge detection accuracy. The key observation from this study is that even when using an LLM for flowchart extraction, significant preprocessing work is required beforehand to improve the obtained content extraction results.

Finally, the work presented in [101] introduced the Dynamic Flow-to-Code Generator (DFCG), which extracts flowcharts and generates source code. This was influenced by the sketch-based flowchart detection research presented in [102] and code generation from flowcharts [103], as well as the works in [40,45]. Their presented method uses binarization for noise reduction, an approach called Faster Region-based CNN (Faster R-CNN) and Visual Geometry Group (VGG-16) [104] for shape detection, Region Proposal Network (RPN) to identify any objects in the feature map, Keras OCR with C-RNN for text extraction, Hough Transform for arrow and line detection, and finally a Graph Neural Network (GNN) [105] for graph construction and conversion into several programming languages. For evaluation, the authors used a dataset containing 775 digital and 100 handwritten flowcharts, achieving a 95.8% mAP, a 93.4% precision, a 94.3% recall, and a 93.8% for F1. By employing a multi-layered algorithm and a sufficiently large dataset, they demonstrated the high performance achievable by their approach, albeit at the expense of significant computational resources, without utilizing transformers. Future DFCG improvements include expanding language support, integrating LLMs for better code quality, enhancing handwriting and graph recognition, optimizing models for mobile use, and adding multimodal inputs like speech and sketch-based flowchart creation.

In this section, we surveyed a wide variety of image content analysis approaches for flowchart images. Table 1 summarizes these studies.

4.2.2. Flowchart Review Summary

Of the 19 methods reviewed, we observe that 8 reported performance metrics that demonstrate a strong ability to successfully conduct this task, while 11 of the surveyed papers reported results that we would deem insufficient for many real-world applications. Despite substantial progress in this domain, we observe that none of the techniques have demonstrated consistently high performance across all tested flowchart images. Thus, we conclude that further improvement is needed in areas such as dataset quality, arrow and line recognition, and comprehensive flowchart extraction.

4.3. Block Diagram Analysis

4.3.1. Review of Identified Papers

Our survey identified only very few research works specifically related to extracting content from Block Diagrams. We believe that this stems from the close relationship between flowcharts and block diagrams, as most works related to image content analysis can utilize the work already performed for flowcharts.

In 2022, BloSum [106] was published, which utilizes OCR and deep learning models for extracting block diagram content. It builds upon Chart-Text [107], which generates descriptions from chart-based images. It excels at chart type classification, achieving 99.72% classification accuracy. However, it is not well-adapted to producing suitable descriptions for different chart types. Instead, T5 [108] is utilized in BloSum for generating text sentences.

A custom dataset of complex Computerized Block Diagram (CBD) was created from search results for block diagrams. 96 images were selected and subsequently used for testing of BloSum. The dataset annotation was performed using the LabelImg tool. The authors of BloSum also tested their approach on additional datasets called FC_A, which contains 171 hand-drawn images, and FC_B, comprising 196 hand-drawn diagrams. The BloSum model also used Faster R-CNN with Feature Pyramid Network (FPN) [109], along with Inception-ResNet-V2 [110] for shape prediction. EasyOCR was used for text prediction, and a Hough Line Transformer (HLT) was used for arrow prediction. A new architecture called Triplet Generator (TG) connected the shapes and arrows in the form of triplets. Image captioning was performed by using the Show, Attend, and Tell (SAT) model. [111]. Finally, T5 and Bidirectional and Auto-Regressive Transformer (BART) [112] language models were both modified and utilized within BloSum for text extraction.

For CBD, BloSum with the T5 model, which was modified to utilize 770M model parameters overall, achieved the highest BLEU score of 42.18% and ROUGE-1 score of 80.78%, along with a low perplexity (PPL) score of 7.54. BloSum with the same T5 model also scored the highest BLEU score and ROUGE-1 scored 51.73% and 88.24% for FC_A, respectively. For FC_B, BloSum with the same T5 model also scored the highest for BLEU at 53.17% and ROUGE-1 at 89.56%. Although this method employs traditional extraction techniques in conjunction with transformers, it nonetheless demonstrates that significant work remains to be conducted to further improve the performance of block diagram extraction. One area the authors identified is the need to further explore alternative methods for extracting shapes and arrows, as well as improving support for more complex block diagrams.

In 2024, another framework for block diagram extraction was introduced called BlockNet [113]. This paper referenced BloSum [106], due to the similarity of block diagram extraction. The authors state that BloSum can suffer from hallucinations due to the use of language models in BloSum. They also point out the smaller data set size of only 463 images total as a contributing factor. BlockNet, by comparison, uses a dataset of 76K images. Other related work mentioned involved symbol detection using Faster R-CNN [114].

To create the dataset used by BlockNet, Mermaid code was utilized for block diagram creation and visualization, along with adding noise to the images to enhance the robustness of the resulting BlockNet approach. GPT-3.5 was used for the annotation of the block diagrams. The dataset has 83,394 English and Korean block diagrams notated as BD-EnKo dataset. The authors divide their technique into 3 key components: a local extractor, a global extractor, and an integrator. In the local extractor stage, YOLOv5 was utilized for object detection, trained on the aforementioned CBD, FC_A, and FC_B datasets. The Pororo OCR package was used due to its support of both English and Korean text. A custom algorithm called BlockSplit divides block diagrams into sub-diagrams and integrates the Triplet Generator to represent shapes, arrows, and their relationships to each other. In the global information extraction stage, the Swin Transformer (ST) [115] was employed as the visual decoder, and BART was used for text decoding. In the integrator stage, to minimize errors, GPT-4V was used to combine the local and global data and to output a summary of the given block diagram.

The BLEU score, ROUGE-1 score, and BLEURT score were 72.31%, 88.9%, and 41.0%, respectively, using the CBD dataset. This demonstrates that using a larger dataset and a more complex framework can achieve higher precision and accuracy. Nonetheless, the reliability of these approaches remains a significant challenge when applied to a wide range of diagram images of varying quality.

4.3.2. Block Diagram Review Summary

Table 2 shows the high-level comparison of the surveyed approaches. For both BloSum and BlockNet, the published results demonstrate the viability of the presented approaches, while also highlighting the potential for further performance improvements. Both methods indicate that they encountered significant errors at the end stage, making them unsuitable for everyday use.

4.4. Electrical Circuit Diagram Analysis

4.4.1. Review of Identified Papers

While flowcharts and block diagrams are seen as closely related, electrical circuit diagrams and schematics are significantly different. Image content extraction from electrical circuit diagrams often targets circuit simulators via netlist generation, and thus demands very high extraction accuracy. Netlists are files that utilize a text representation to describe an electrical circuit, including its components, their parameters, and the connections between them.

In 2020, the work in [116] describes a circuit diagram approach that expands upon the works of [117,118,119]. Their approach starts by first binarizing the image and performing noise reduction. This is followed by localization, to extract circuit components without any of the diagram’s wires after applying the Run Length Smoothing Algorithm (RLSA) [120]. The authors also generated their own dataset of 60 hand-drawn circuit diagrams, as they were unable to locate any publicly accessible datasets applicable to their research. A localization accuracy evaluation was performed, yielding an overall accuracy of 91.28% in extracting circuit components.

The work in [121], called Img2Sim and published in 2022, uses an Artificial Neural Network (ANN) [122] approach to convert circuit schematics into a netlist. This work builds upon a prior approach published in [123] that used deep learning to recognize circuit components, and [124], which involved identifying hand-drawn circuit images using OCR. The paper [125] is also referenced. However, while these methods primarily focus on passive components, Img2Sim also aims to support active components.

A dataset comprising 330 circuit diagrams was used for testing, utilizing YOLOv5 for element recognition, a Hough Transform for node detection, and OCR for text recognition, with the final results generated as a netlist. Their presented method detects circuit components with an accuracy of 98%, and generates correct netlists with an accuracy of 90%. This demonstrates that, while there is room for improvement in circuit diagram extraction from images, significantly more research is needed to accurately represent these diagrams using netlist files post-extraction, in order to support the intended downstream applications.

Another method, presented in 2022 [126], utilizes a DNN to generate netlists from hand-drawn circuit diagrams, and the CNN-based classification approach published in [125] to classify circuit components. Other works, such as [124,127,128,129] influenced their research and the published extraction approach. YOLOv5 is utilized for component recognition, adaptive thresholding, and bounding box extraction for terminal recognition, as well as k-means clustering for detecting intersections. The Hough Transform is applied for line detection. This approach was implemented using PyTorch and subsequently evaluated using a dataset comprised of 388 images. It achieved an average mAP score with 0.5 IoU of 99.19%. Their results indicate good extraction performance for simple circuit diagrams using passive components, but more research is needed to improve the accuracy of more complex diagrams involving active components and Integrated Circuit (IC) components. Additionally, these methods require further evaluation using larger datasets spanning a wider range of circuit diagram representations.

In 2022, [130] utilized machine learning methods to convert hand-drawn circuit diagrams into netlists. It builds on the work in [116] for component detection in hand-drawn circuit diagrams, as well as the work in [124]. A custom dataset containing hundreds of images of resistors, diodes, inductors, and op-amps was used for evaluation. These images were converted to grayscale to facilitate noise removal prior to binarization. Their presented scheme incorporates dilation and skeletonization [131], as well as OCR for text extraction, the use of Histogram of Gradients (HOG) [132] and Support Vector Machine (SVM) [133] for classifying the components using OpenCV, along with the identification of the resulting nodes. The method achieved an average precision of 94%. Even though this score is high, since their scheme focused only on simple passive components, their approach does not scale well for more complex circuit diagrams.

The authors of [134], published in 2021, developed a method that leverages augmented reality techniques to convert hand-drawn circuit diagrams into other representations using a smartphone. The author’s work expands upon [135,136], which used an SVM approach for circuit recognition. Feature extraction is performed using a custom algorithm that combines region proposal using SURF, Eigen, and FAST. Dilation and Hough Transform are also incorporated as part of the processing workflow. A capsule network architecture (CapsNet) [137] is used instead of a CNN due to limitations with CNNs, as discussed by the authors in their paper, primarily related to the significantly higher training dataset size required for CNNs compared to CapsNets. The final step in their approach is circuit simulation, which determines whether the extraction produces a viable netlist, concluding with the export of the netlist generated by the previous steps. With a custom dataset comprising 800 circuit diagrams used for evaluation, and utilizing the mAP scoring metric with an IoU threshold of 0.3, the resulting score for this approach is 93.64%.

Circuit diagram extraction has attracted increasing attention in recent years, driven by advancements in machine learning that enable more accurate recognition and interpretation of complex schematics, as shown in Figure 9. Using Litmaps, this analysis illustrates the relationships and interdependencies among the reviewed papers.

The authors of [138,139,140] each developed their own circuit extraction techniques that advance upon [129] by widening its scope, which focused exclusively on text extraction from hand-drawn diagrams. The techniques discussed in [138,141] also reference the research [127], which focuses on recognizing hand-drawn schematics. Published in 2022, the work in [138] presents a model that can analyze hand-drawn or computer-generated schematics specifically of power converter circuits, generate netlists from them, and simulate them in real-time. It is inspired by older works, such as [121,123,126,128,130,134]. For evaluating their presented technique, they developed their own dataset. The images underwent noise filtering to help build a more robust system. A variant of YOLO called You Only Look Once Representation (YOLOR) [142] was used to detect components. Hough Transform was used to detect lines. Edge detection is performed to improve the performance of the HT. Node recognition was conducted using a K-Means clustering technique [143]. The resulting netlist is then evaluated and simulated using PySpice. The technique’s assessed mAP0.5 score was 91.6%. This work’s biggest shortcoming is in diagrams involving surface-mount components in Printed Circuit Board (PCB) schematics. The authors also compared their performance obtained using PySpice with performance metrics from LTSpice, which confirmed the accuracy of the generated netlists for diagrams where extraction was conducted successfully.

In 2023, Bayer et al. published [140], which aimed to facilitate extracting component labels and junctions from circuit diagram extraction using keypoint extraction [144] and instance segmentation [145]. For evaluation, it utilized the Circuit Graph Hand-Draw (CGHD) dataset, containing 2,304 hand-drawn circuit diagrams. Additionally, their approach is inspired by [122,124,126]. The presented approach first separates the background from objects using binarization maps, which are then further refined using the LabelME annotation tool to clean up the coarse polygon masks. The image was then simplified by layering coarse polygons with binarization maps. Contour detection was employed to enhance the precision of the polygons, reducing ambiguity in the annotations, before proceeding to contour detection. Keypoint Generation precisely identifies connection points on electrical symbols after applying the morphological operator to reduce the impact of noise. A technique called Keypoint Port Assignment is then performed to link detected connection points on electrical symbols to their correct terminals, with the final step being the creation of electrical circuit graph representations.

The results obtained from applying this approach with the CGHD dataset demonstrate a mask accuracy of 94%. This demonstrates that traditional computer vision information extraction approaches can yield commendable results. However, they are often outperformed by modern ML-based approaches, as shown in this survey.

Closely related to [138], the authors of [139] in 2023 published a technique that combined OCR, structured pattern extraction with regular expressions, and shape matching using geometric methods, building upon their own prior work published by [124,140]. The aforementioned CGHD dataset was used for their testing. During the text detection stage, Faster R-CNN was used. For handwritten recognition, the approach relies on a combination of CNN and LSTM, as well as Regular expressions (RegEx) for text label identification. The Euclidean distance was the metric employed by the authors for text symbol assignment, and prefix resolution was used for unit of measurement resolution. The work demonstrated promising results. It excels at resistors, capacitors, and voltage labels. However, subscript annotations and negation lines are not yet supported, and additional work is needed to address these and related capabilities. More work is also needed for the normalization of electrical values. Lastly, a consistency evaluation must be conducted to confirm the integrity of the graph’s generated output representation.

This 2024 research paper [146] proposed another technique for circuit diagram extraction, influenced by earlier techniques such as [147], which was created to help to recognize faulty analog circuits using neural networks, as well as [126,128,148]. Following the initial image acquisition, they first convert the image into a grayscale representation. YOLOv8 is leveraged to perform text recognition with high accuracy, whereas a CNN and Line Detection are used during image segmentation to classify the structure of the circuit components and their corresponding value labels. The resulting text representation of the circuit diagram is used in a subsequent analysis stage to evaluate the accuracy of current and voltage levels at each node present in the circuit. 50 diagrams were used for testing, performed on a NVIDIA RTX 4070 GPU, and implemented using Python. This evaluation yielded an 84% success rate, with 42 out of 50 images extracted without errors. However, a more comprehensive evaluation using a significantly larger dataset is needed to properly assess this technique in comparison to others in this domain.

In 2024, [149] presented another technique that aims to convert schematics into netlists, based on discussions from [121,126,128,130]. Line detection was performed by using a Lightweight Convolutional Neural Network (L-CNN) [150]. EasyOCR was employed with a detection model called Character-Region Awareness For Text Detection (CRAFT). This model was fine-tuned from a model checkpoint of both SynthText and ICDAR 2015. A custom dataset containing 124 circuit diagrams was used in their performance testing to evaluate symbol detection. Although the obtained results were promising for component and line recognition, significant improvements are needed, particularly in terms of robustness for varying symbol orientations, and detecting small text labels such as pin numbers used on integrated circuit components.

A similar effort, published in 2024 in [151], aimed to address the process of extracting information from digitized circuit diagram images by using a custom OCR model, improving on [152]. Here, Otsu’s method was used for binarization and skeletonization to gain a specific contour size of 1 pixel wide for the circuit. A Progressive Probabilistic Hough Transform (PPHT) [153] was then employed for recognizing lines in the circuit, and Tesseract OCR was used to read the text of each symbol. This approach was then implemented as a software solution and evaluated using a custom dataset of 15 schematics, with 3 being hand-drawn and 12 obtained from various publishers. Their testing showed that 166 out of 194 symbols in the circuit were correctly identified, resulting in an accuracy of 85.6%.

In 2025, the work in [141] evaluated circuit diagram recognition using a graph structure. This was inspired by efforts on the automatic generation of detection rules for the identification of a circuit published in [154,155], which both leverage machine learning to recognize circuit components, as well as the work published in [127]. Otsu’s method is used for the preprocessing stage to perform noise reduction. It then utilizes ResNet-101 [156] and VGG for circuit retrieval. Their work used a more optimized structured graph for the circuit diagrams. GED was employed as their graph matching measurement [157], while Intersection of Union (IoU) and mAP were used for object detection metrics. For the evaluation of this technique, a dataset of 227 images was employed together with GAM-YOLO [158]. It resulted in a mAP score of 91.4%. For circuit diagram extraction, the results show significant promise compared to previous methods. Its high accuracy enables a variety of downstream tasks to be performed successfully using the circuit diagram’s graph structure representation.

Additionally, a 2025 paper [159] discussed using DL for circuit diagram extraction, referencing [116,121,126,129,130] as inspirations for their work. The Kaggle dataset [160] was used for testing, with additional complex circuit symbols, such as operational amplifiers (OP-AMPs), included. The authors developed their own CNN Multi-Layer Model. The method scored a 96% test accuracy using mAP, which is significantly better than prior approaches. The authors also created a web application that allowed users to upload images and generate netlists. This work demonstrates notable improvements in the automated extraction and interpretation of circuit diagrams.

Finally, building on [151], the authors of [161] conducted an experiment that individually extracted each layer of a circuit diagram, allowing for a more accurate extraction. It aims to improve upon prior efforts such as [151] for better support of digitizing circuit diagrams. It also incorporates efforts from [162], which specifically aims to improve the detection of small objects in circuit diagrams. YOLOv7 was used for element layer detection within a circuit diagram, PaddleOCR [163] and PaddleClas image classification [164] were used for text layer extraction. The author used a connection relationship algorithm to understand the relationship between each diagram. Their framework was implemented using Python 3.9, PyTorch 1.13, and OpenCV to run on a host with Ubuntu 20.04 LTS and a single NVIDIA RTX 3090 GPU. Using a dataset containing 5,000 diagrams and an Element Feature Library (EFL), this method achieved an F1-score of 99.6% with diagrams featuring various backgrounds. This highly promising result shows the significant advances made in this research domain in recent years.

4.4.2. Electrical Circuit Diagram Review Summary

Table 3 contains the 14 schematic diagram extraction papers that were discussed in this part of our survey. It is our estimation that 10 out of 14 methods are suitable for various application domains, whereas the remaining 4 papers demonstrate earlier efforts that paved the way for later significant advances in the accurate and detailed extraction of electrical circuit diagrams from images. All of these efforts, however, could benefit from further evaluation using larger benchmark datasets to better compare their individual performance across the variety of algorithms and ML techniques proposed and studied over the years for this application domain.

4.5. Timing Diagram Analysis

4.5.1. Review of Identified Papers

A category of technical illustrations that stands on its own is timing diagrams. Unlike many other diagram types, the importance of line positioning is significantly elevated, and thus it mandates highly accurate extraction to accurately represent signal change timing.

A study published in 2023 developed a method named TD-Magic [165], using object detection for rising and falling edge detection, along with OCR to detect text labels. It also incorporated the use of image processing techniques to extract synchronization indications. In addition to influences from [166], its approach to text extraction from timing diagrams was inspired by work from the medical domain [167], focusing on Electrocardiogram (ECG) signal extraction. Feature recognition was performed using a Signal-Edge Detector (SED) [168] and YOLOv5 for edge recognition. PaddleOCR was used for text detection. The Line-and-Arrow-Detection (LAD) method was employed for identifying arrows and lines in the timing diagrams. Semantic Interpretation (SEI) [169] was employed to create annotations for each aspect of the timing diagram using Strict Partial Order (SPO) [170]. For experimentation and evaluation of their approach, the authors also utilized Formal Methods (FM). The dataset for evaluation included 29 timing diagrams from various sources. Out of the 29 timing diagrams, 23 were effectively obtained at the template stage. However, only 15 of the test cases could be processed successfully and accurately. Using the mAP scoring metric, an overall score of 76.7% was achieved. Hence, significantly more work is needed to make this approach more robust and performant. Furthermore, a larger dataset is needed for the evaluation to be more informative and applicable to different downstream application domains.

Another timing diagram extraction technique was introduced in 2025, called TD-Interpreter [171]. While it was inspired by TD-Magic [165], the only aspect they have in common is that TD-Magic and TD-Interpreter both read timing diagrams. Another inspiration for this work came from LLaVA–a Visual Language Model (VLM) [172]. TD-Interpreter utilizes a ViT to process images and extract visual features. The LLaVA VLM processes the extracted information, performing reasoning and generating responses. Low-rank-adaptation (LoRA) [173] was employed for fine-tuning the layers of the proposed algorithm. A custom dataset was created for testing using caption-based data. It is comprised of 4942 timing diagram images. Additionally, for reasoning-based testing, 5292 timing diagram images were used. The evaluations were conducted on a system with 8 NVIDIA A800 GPUs, and achieved a BLEU-4 score of 95.9%, and ROUGE-1 score of 96.7%. Thus, the authors demonstrated that, with the use of transformers and a large dataset, along with access to sufficiently powerful computational hardware, it is feasible to achieve highly precise and accurate timing diagram extraction.

4.5.2. Timing Diagram Review Summary

Table 4 shows the comparison of TD-Magic and TD-Interpreter. The applicability of TD-Magic is limited by its low score achieved over a comparatively small dataset. However, it did succeed on several test cases and thus warrants further research to improve its performance, especially for use cases with limited computational resources such as in edge devices. The TD-Interpreter, on the other hand, employs a significantly more advanced VLM-driven approach that includes reasoning capabilities and was tested on a much larger dataset. Thus, among the two, TD-Interpreter is a more promising approach that, given the required computational resources, will be able to achieve significantly better results for downstream applications depending on accurate timing diagram extraction.

4.6. Comparative Observations and Summary Across Illustration Types

Based on the reviewed research papers across all four technical diagram types, Transformer-based approaches consistently demonstrate strong performance in terms of accuracy, adaptability, and robustness, regardless of diagram type. While these Transformer-based approaches are computationally expensive, more lightweight models are available for application scenarios where efficiency outweighs accuracy, such as in edge computing or mobile applications. Object detection frameworks such as YOLO and LLMs such as the various GPT models also offer reliable performance across diverse scenarios. Traditional extraction methods remain useful for specific tasks, but combining them with modern techniques generally yields the most effective results across a wide range of situations.

It is important to note that different diagram types attract varying levels of research attention, which influences how rapidly performance improvements can be achieved. Flowcharts are the most extensively examined type of technical diagram, which has led to more advanced, fine-tuned, and higher-accuracy extraction techniques, along with publicly accessible benchmark datasets. Circuit diagrams have also received considerable research focus. However, progress in this area remains constrained by the reliance on small, domain-specific, custom-built datasets. In contrast, timing diagrams and block diagrams have received relatively limited attention, leading to slower methodological advancements and lower overall extraction performance.

A notable issue identified in the reviewed research is the scarcity of publicly available datasets that can be reliably used for testing and evaluating extraction methods. Most studies relied on custom datasets tailored to their specific needs, which may introduce bias and limit the generalizability of their results. The availability of a standard public dataset could enable more robust and applicable benchmarking, supporting fairness in comparing various image content analysis approaches.

5. Conclusions

With the rapid proliferation of LLMs and Agentic AI in our everyday lives comes a resurgence in the need for multimodal data access to inform queries and produce accurate responses. As a result, we see growing research activities in image content analysis and extraction for complex technical illustrations, such as flowcharts, block diagrams, schematics, and timing diagrams—illustrations where a significant portion of the information is represented not just by text, but by the contained shapes, their styling, and their relationships. To cope with increasingly complex diagrams and the need to reduce extraction errors, the techniques utilized for extracting information from these technical illustrations must also evolve. The rise of AI-powered techniques resulted in a number of highly promising image content analysis techniques, achieving higher accuracy and richer contextual representations than ever before, across a variety of diagram types.

Our survey found that the majority of recent research activities focused on flowcharts, while circuit schematics represent the second-most studied diagram type. In contrast, block diagrams and timing diagrams have received only limited interest, with only two studies dedicated to each of these categories in recent years.

While the advancements made so far are promising, significant work remains as future research avenues to optimize extraction methods for various diagram types, reduce extraction errors, and improve the cost and complexity of these algorithms. Reducing complexity, in particular, is crucial for accelerating information extraction in an era where an ever-increasing amount of multimodal information is being generated. Similarly, for applications where accuracy is paramount, further research is necessary to enhance information extraction in terms of both completeness and accuracy. This is vital for reducing hallucinations and downstream generative errors in Agentic AI systems. Finally, we also observed that the significant lack of benchmark datasets severely impedes objective comparisons across the various techniques. While several research works utilize larger datasets with thousands of images, others use much smaller datasets, sometimes comprised of only a few dozen images. Thus, there is a strong need to establish and publicly release benchmark datasets that span the full range of technical diagram types, with sufficient representation in terms of complexity, color usage, multilingual aspects, diagram correctness, and other features.

This survey aims to provide both a cross-section of research work related to information extraction from technical diagrams, as well as inspire researchers to pursue the various open research questions in this field, in an effort to advance the accuracy, fidelity, and efficiency of these techniques, because there are numerous applications that would be enabled by these capabilities.