From OCR to VLM: A Survey and Position on Document Understanding Architectures

1. Introduction

Document understanding sits at a peculiar intersection of human cognitive ease and machine computational difficulty. A human can extract information from structured documents, such as an insurance form: recognizing amounts, understanding checkbox choices, and validating that totals match sums. These acts require sophisticated visual processing, structural reasoning, semantic interpretation, and knowledge integration. For machines, each represents a distinct challenge.

The field has evolved through distinct paradigms. Early systems decomposed the problem into separate stages: OCR converted pixels to characters, layout analysis identified structure, rule-based extraction mapped text to fields, and application logic validated results. This modular approach offered debuggability but suffered from error propagation: mistakes in character recognition cascaded through subsequent stages.

Modern VLMs challenge this paradigm by learning end-to-end mappings from document images to structured outputs or natural language responses. A single model might simultaneously perform character recognition, layout understanding, information extraction, and reasoning. This integration can improve overall performance but introduces new complexities: which architecture for which task? How do we balance OCR precision against reasoning capability? When should we use specialized models versus general-purpose systems?

The landscape has fragmented substantially as of 2024–2025, with practitioners facing a complex tradeoff space. Traditional OCR remains the fastest and most cost-effective option for simple text extraction. Document-specific VLMs like Donut [Document understanding Transformer; [1]], Pix2Struct [2], and TrOCR [Transformer-based Optical Character Recognition; [3]] offer end-to-end structured extraction with strong OCR precision. General-purpose models like Generative Pre-trained Transformer 4 Vision (GPT-4V), Claude Sonnet 4.5, and Gemini 3 Pro handle reasoning and validation but at higher cost and latency. Recent innovations suggest these paradigms may be converging. DocVLM [4] demonstrates that hybrid approaches can achieve both efficiency (80% fewer tokens) and accuracy (86.6% on DocVQA [5]). This technological evolution is driving rapid enterprise adoption: the intelligent document processing (IDP) market has reached $7.89 billion with projected 30.1% annual growth.

Existing comparisons focus narrowly on benchmark accuracies, leaving practitioners without guidance for real-world deployment. A model achieving 95% on receipt parsing might fail catastrophically on insurance forms with checkboxes. Another model excelling at visual question answering might miss small text in dense tables. Raw accuracy numbers obscure the capability dimensions that actually determine fitness for purpose.

This paper combines a survey of the document understanding landscape with a position on how practitioners should navigate it. We make three contributions. First, we survey the technical evolution from CNNs through transformers to modern VLMs, identifying two distinct architectural paths and explaining why they excel at different tasks. Second, we propose a five-dimensional capability framework that replaces discrete model categories with continuous characterization, arguing that this shift in perspective is necessary for principled model selection. Third, we take the position that architectural tradeoffs between precision-optimized and reasoning-optimized paths are fundamental rather than transient, and provide decision frameworks grounded in this claim for mapping application requirements to architectural strengths.

2. Background: The Stages of Document Understanding

Understanding a document requires progression from low-level perception to high-level cognition. We conceptualize this as four distinct capabilities, each building upon the previous. Figure 1 illustrates this layered progression.

At the foundation lies character recognition: converting visual patterns into text, handling varied fonts, languages, and degraded quality. Above this, structural understanding captures spatial relationships—which text belongs to which table cell, reading order, hierarchical sections—where models like LayoutLM [6] incorporate explicit positional embeddings. Information extraction maps recognized text to semantic meaning: amounts, checkbox choices, field-value associations. At the highest level, document intelligence encompasses reasoning, validation, and question answering that require integrating information and applying domain knowledge.

Traditional approaches implement each stage with separate models, offering debuggability but suffering from error propagation: character recognition mistakes cascade through subsequent stages. Modern VLMs challenge this by learning direct mappings from images to structured outputs, jointly optimizing all stages but operating as less interpretable systems.

3. A Framework for Characterizing Model Capabilities

Traditional categorizations (e.g., “OCR model,” “layout model,” “VLM”) obscure more than they reveal. They suggest discrete bins when capabilities exist on continuous spectra. We propose a more nuanced view recognizing five fundamental dimensions.

3.1. Text Recognition Ability

This dimension spans from basic (clean printed English) through intermediate (multilingual, rotated, varied fonts) to advanced (handwriting, degraded images, historical documents). Traditional OCR systems like Tesseract [7] excel at clean printed text but struggle with handwriting. Modern VLMs trained on diverse datasets show improved robustness but sometimes sacrifice precision where specialized models perform better.

The critical question: what range of text types must you handle reliably? If your documents contain only printed invoices, specialized OCR suffices. If they include handwritten forms and historical documents, you need broader capabilities that VLMs provide.

3.2. Structural Understanding

This dimension measures topological understanding: the ability to recognize relationships and organization independent of precise geometric coordinates. It encompasses three key aspects. First, reading order determines the sequence in which text should be read. This includes navigating multi-column layouts, traversing table cells left-to-right and top-to-bottom, understanding field sequences in forms, parsing mathematical expressions, and handling mixed content with embedded figures. Second, containment hierarchies capture nested structure: sections contain paragraphs, paragraphs contain lines, and lines contain tokens. Third, relational structures identify associations between elements: which value belongs to which label in forms, how checkboxes connect to their labels, and which cells belong to which table rows and columns.

This ranges from basic (plain text with reading order) through intermediate (tables, multi-column layouts) to advanced (complex hierarchical documents, charts, figures). Layout-focused models like LayoutLM [6] and Pix2Struct [2] incorporate explicit structural representations through positional embeddings and visual features. General VLMs rely on learned spatial reasoning without explicit structure modeling.

For simple forms, learned reasoning may suffice. For complex tables spanning pages with intricate cell relationships, explicit structure helps. The key question: how much structural complexity appears in your documents?

3.3. Output Flexibility

This progresses from fixed (predefined classification labels) through semi-structured (Markdown, JSON with fixed schemas) to fully flexible (arbitrary formats via natural language specification). Encoder-only models output fixed labels (fast but inflexible). Decoder-based models generate structured text in predefined formats. General VLMs offer maximum flexibility through natural language prompting.

This dimension directly impacts adaptation ease. Fixed outputs require retraining for new schemas. Flexible generation adapts via prompt changes, though perhaps less optimally than fine-tuned alternatives.

3.4. Spatial Understanding

This dimension measures geometric understanding: the ability to precisely locate and encode spatial positions of document elements. It encompasses absolute and relative coordinate systems, bounding box and polygon prediction, and the ability to precisely highlight the source location of a value or answer. This is fundamentally about geometry—knowing the exact pixel coordinates or spatial regions where information appears.

Document-specific models often provide precise bounding boxes, enabling downstream systems to highlight source regions. General VLMs typically operate at more abstract semantic levels, understanding “the amount in box 7” without necessarily providing pixel coordinates.

This matters when you need extraction validation by showing users where information came from, or when downstream systems require precise localization for highlighting or verification.

Structural and spatial understanding are related but distinct: a model might correctly determine reading order with fuzzy bounding boxes (high structural, low spatial), or produce accurate word-level coordinates without understanding table relationships (high spatial, low structural). The distinction matters because some applications require precise coordinates (UI highlighting) while others need correct relationships (structured extraction).

3.5. Task Adaptability

This ranges from single-task (fine-tuned for specific document types) through few-shot (adapting with examples) to zero-shot (handling novel tasks via instructions). Traditional OCR offers no task adaptability: it recognizes characters regardless of context. Layout models support fine-tuning with labeled data. General VLMs excel at zero-shot adaptation through natural language instructions.

This dimension determines deployment effort for new document types. Single-task models require expensive retraining. Zero-shot models adapt via prompting, though perhaps suboptimally compared to specialized alternatives.

Output flexibility and task adaptability tend to correlate but are not identical: some models emit flexible formats but require fine-tuning for new tasks, while others adapt zero-shot with more constrained outputs.

These dimensions reveal fundamental tradeoffs—no single model excels everywhere. Understanding your application’s position along these dimensions guides selection. Figure 2 illustrates how three model archetypes score differently.

4. The Architectural Evolution of Document Understanding

To understand why models occupy different positions in this capability space, we must trace their architectural evolution through three major transitions: CNNs for visual processing, transformers for sequence modeling, and vision-language pre-training for multimodal understanding. Figure 3 illustrates this lineage.

4.1. The CNN+LSTM Era: Neural OCR

Before neural networks, OCR systems relied on rule-based approaches: hand-crafted feature extractors, template matching, and statistical classifiers. Systems like early Tesseract (versions 1–3) used adaptive thresholding, connected component analysis, and character-level classifiers trained on extracted features. These approaches worked well for clean, standardized documents but struggled with font variation, noise, and complex layouts. Each new document type often required manual tuning of parameters and features.

Deep learning adoption in OCR (2015–2020) replaced hand-crafted features with learned representations. Modern neural OCR systems follow a two-stage pipeline: detection and recognition. A CNN performs pixel-level segmentation to locate text regions, outputting bounding boxes. For each region, another network extracts visual features through convolution, models temporal dependencies via bidirectional Long Short-Term Memory networks (LSTMs), and decodes characters through Connectionist Temporal Classification (CTC). This architecture powers systems like Tesseract v4+, CRNN, EasyOCR, and PaddleOCR.

This architecture excels at speed (50–100ms per page) and handles rotated text well through learned rotational invariance. It generalizes far better than rule-based systems, handling font variations and moderate noise without manual tuning. But it has fundamental limits stemming from its design. The two stages operate independently, preventing error correction between detection and recognition. There is no language understanding: “INVOICE” is merely seven characters with no semantic meaning. Checkboxes, signatures, and other non-textual elements are invisible to the system, which only recognizes text.

These limitations are not bugs but inherent consequences of the architecture. CNNs excel at local pattern recognition but lack mechanisms for long-range dependencies or semantic understanding. The two-stage pipeline optimizes each component independently rather than the full task.

4.2. The Transformer Revolution

The transformer architecture [8] brought self-attention mechanisms enabling each element to attend to all others simultaneously. Three architectural patterns emerged that persist in modern document systems: encoder-only models [BERT; [9]] use bidirectional attention for fast classification but cannot generate flexible output; decoder-only models [GPT; [10]] use causal attention for flexible text generation but lack bidirectional context; encoder-decoder models [T5; [11]] combine both, balancing understanding with generation through cross-attention.

4.3. Vision Transformers: Bridging Vision and Language

The Vision Transformer [ViT; [12]], introduced in 2020, adapted transformers to images with a deceptively simple idea: treat images as sequences of visual tokens. The process: divide the image into fixed-size patches (e.g., 16×16 pixels), flatten each patch and project to an embedding dimension, add position encodings, feed through standard transformer layers.

This bridged vision and language: images and text could now be processed by the same architecture. But ViT alone does not solve document understanding. It is trained for image classification, learning to recognize objects and scenes, not to read text or understand document structure.

From ViT, document understanding evolved along two distinct paths, each optimizing for different objectives and making different tradeoffs. Figure 4 illustrates the architectural distinction between these two approaches.

4.4. Path One: High-Resolution ViT to Document VLMs

The first evolutionary path optimizes for pixel-level precision. Models like Donut [1], Pix2Struct [2], Nougat [13], and TrOCR [3] use ViT encoders at high resolution, extracting dense patch embeddings that preserve fine-grained visual details. TrOCR represents an early transformer-based approach using pre-trained image and text transformers for end-to-end text recognition, achieving state-of-the-art performance on printed, handwritten, and scene text. The OCR-free Donut model advanced this further by eliminating the need for separate OCR pipelines entirely: no separate text encoder exists. An autoregressive decoder generates output text token by token, attending to visual embeddings through cross-attention.

Training occurs on synthetic documents with OCR-like tasks: reading text from diverse layouts, recognizing table structures, parsing mathematical formulas. The model never sees explicit OCR annotations. It learns end-to-end from (image, structured text) pairs, discovering how to map visual patterns to characters through gradient descent.

This architecture delivers excellent OCR accuracy because it optimizes directly for character-level precision. It handles complex layouts and tables well by learning to attend to relevant image regions while generating structured output. It can output various structured formats (Markdown, LaTeX, JSON) by learning different generation patterns. And it has no external OCR dependency: the vision encoder itself learns character recognition.

However, it shows limited reasoning capabilities. The model excels at reading what is visually present but struggles with validation, cross-field reasoning, and question answering requiring inference. It requires substantial training data for new domains because it learns through pattern matching rather than conceptual understanding.

4.5. Path Two: CLIP to General VLMs

The second evolutionary path optimizes for semantic reasoning. Models like GPT-4V, Claude, Qwen-VL, Gemini 3 Pro, and LLaVA [14] use CLIP-style encoders [15] that learn from internet-scale image-text pairs. CLIP pre-training matches images with captions through contrastive learning: images of dogs should align with “dog,” beach scenes with “beach” and related concepts like “ocean,” “sand,” “vacation.”

This trades pixel-level detail for conceptual understanding: learning what things mean rather than precisely what they look like. The vision encoder learns semantic features aligned with language concepts, not low-level visual patterns optimized for character recognition.

The architecture combines a CLIP/SigLIP vision encoder with a large language model decoder (100B+ parameters). The decoder attends to image tokens via cross-attention, integrating visual and linguistic information. Adapter modules (Q-Former, Perceiver, Resampler) may compress visual information to reduce computational cost while preserving semantic content.

Training proceeds through multiple stages of increasing complexity. First, vision-language alignment on image-caption pairs teaches basic correspondences between visual and linguistic concepts. Then instruction tuning with diverse visual tasks (Visual Question Answering [VQA], captioning, OCR, reasoning) teaches task-following behavior. Some commercial VLMs (like GPT-4V, Claude, Gemini) add a final alignment stage using Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO) to align model outputs with human preferences for helpfulness and accuracy.

This architecture delivers superior reasoning and comprehension. It validates consistency through logical inference, answers questions requiring multi-step reasoning, and integrates world knowledge: “Is this a valid US ZIP code?” It handles novel document types through zero-shot adaptation: describe the task in natural language and the model adapts, though perhaps not optimally.

However, it may miss fine-grained visual details because CLIP prioritizes semantic alignment over pixel precision, not necessarily optimized for reading 6-point font or distinguishing subtle checkbox marks. It also incurs higher computational cost and slower inference due to the large language model backbone.

4.6. The Convergence Question

We observe these evolutionary paths gradually converging. Document VLMs add reasoning capabilities by incorporating stronger language model components to improve semantic understanding beyond pure extraction. General VLMs improve OCR precision: GPT-4V to GPT-4 Turbo shows better handling of fine-grained details, Claude Sonnet 3.5 to 4.5 improves document understanding, achieving 0.145 overall edit distance on OmniDocBench OCR benchmarks as of late 2024.

Hybrid models explicitly attempt both strengths simultaneously. PaLI combines multiple vision encoders to capture both pixel-level details and semantic features. Qwen-VL 1.5 uses adapter modules to fuse different visual representations. A particularly promising recent approach is DocVLM [4], which integrates an OCR-based modality into existing VLMs through learned queries. DocVLM improves DocVQA accuracy from 56.0% to 86.6% with InternVL2 while using 80% fewer image tokens with LLaVA-OneVision, demonstrating that efficiency and accuracy improvements can coexist. The method employs an OCR encoder to capture textual content and layout, compressing these into a compact set of learned queries incorporated into the VLM.

The challenge remains complexity: each additional encoder increases model size, training cost, and inference latency. Whether unified models will dominate or coexist with specialized alternatives remains an open question for production systems.

Alongside convergence, efficiency-optimized variants compress these architectures for throughput and cost. Models like DeepSeek-OCR (3B parameters, 570M active, processing 200,000 pages/day on a single A100) employ smaller vision encoders, token compression, and knowledge distillation. These variants bridge the gap between classical OCR (fast but limited) and full-scale VLMs (capable but expensive), serving high-volume deployments where cost and latency matter more than advanced reasoning.

5. System-Level Architectures: Pipelines, End-to-End, and Retrieval-Augmented Systems

Beyond individual model architectures, document understanding systems can be organized at a higher level into three distinct patterns: end-to-end models, pipelines, and retrieval-augmented systems. These represent different approaches to composing models and processing workflows, each with distinct tradeoffs. Figure 5 illustrates these three patterns.

End-to-end systems use a single VLM to map directly from document images to structured outputs, enabling global optimization but operating as a black box where failure modes are harder to debug.

Pipeline architectures decompose understanding into sequential stages (OCR → layout → extraction → validation), offering modular debuggability but suffering from error propagation between stages.

Retrieval-augmented systems shift from extracting everything to retrieving what matters: indexing document content, then retrieving relevant pages based on queries before applying VLM reasoning. Models like ColPali [16] use late-interaction embeddings for efficient document retrieval, dramatically reducing computational cost for large document collections.

6. Practical Considerations for Model Selection

With architectural understanding established, we can now address the practical selection problem. The choice depends on four primary factors that interact in complex ways.

6.1. Task Requirements

Task requirements fall along a spectrum from pure extraction to complex reasoning. At the extraction end, you only need to extract field values from structured forms with consistent layouts. Here, fine-tuned layout models offer optimal accuracy per dollar. They require labeled training data but achieve 90%+ accuracy with modest compute once trained.

Moving toward reasoning, you need validation: checking that amounts sum correctly, dates are valid, mutually exclusive options are not both selected. Here general VLMs become necessary. The same reasoning capability that enables flexible question answering also enables consistency checking. The model can understand that if checkbox A is marked, checkbox B (mutually exclusive) should not be.

Understanding where your task sits on this spectrum guides architectural choice. Pure extraction does not require expensive general VLMs. Complex reasoning does not work well with simple extraction models.

6.2. Document Characteristics

Document characteristics profoundly impact which architectures succeed. Clean printed text with simple layouts? Traditional OCR may suffice at 50ms per page, achieving 95%+ accuracy at minimal cost. Such documents present no challenges requiring advanced capabilities.

Complex tables spanning pages? Document VLMs handle structure better through learned attention patterns that can track cell relationships across columns and rows. Traditional OCR provides character sequences but no structural understanding.

Non-textual elements like checkboxes, signatures, or stamps? Here the architectural differences become stark. Our experience with processing structured documents across a wide range of industry sectors suggests that checkbox detection differentiates models dramatically. Traditional OCR systems achieve 50–60% accuracy on checkboxes because they fundamentally lack visual pattern recognition for non-textual elements: there are no characters to recognize. Document VLMs reach 80% by learning checkbox visual patterns (filled vs empty boxes) during training on synthetic forms. General VLMs achieve 90% by combining visual understanding of checkbox states with semantic reasoning about associated questions, enabling them to resolve ambiguities through context.

This example illustrates a broader principle: architectural capabilities must match document characteristics. OCR systems cannot handle non-textual elements regardless of how well we tune them: the architecture simply lacks the capability. Understanding what your documents contain guides what architectures can possibly work.

6.3. Operational Dimensions for Model Selection

Operational constraints often determine feasibility more than capabilities. We formalize these as operational dimensions parallel to capability dimensions, enabling systematic comparison across models. Figure 6 illustrates how different model archetypes occupy different regions of the cost-latency tradeoff space.

Four operational dimensions determine feasibility (Table 1). Inference cost ranges from free (open-source weights requiring self-hosted infrastructure) to $0.01–0.10 per page for API calls; high-volume deployments (>100K pages/month) favor self-hosting, while variable workloads benefit from pay-per-use APIs. Latency spans orders of magnitude—50ms for OCR, 500ms–1s for document VLMs, 1–5s+ for general VLMs—directly constraining whether real-time or batch processing is feasible. Adaptation effort ranges from zero-shot prompting (general VLMs) to fine-tuning with labeled data (specialized models), trading deployment speed for task-specific accuracy. Privacy and deployment constraints may eliminate API-based solutions entirely due to regulatory requirements (HIPAA, GDPR), regardless of performance advantages.

6.4. Stakeholder Priorities

Stakeholder priorities often conflict: engineering teams prioritize speed and infrastructure costs, business stakeholders prioritize accuracy and time-to-value, compliance teams prioritize explainability and privacy. Making these tradeoffs explicit through weighted scorecards (e.g., accuracy 40%, cost 30%, latency 20%, privacy 10%) prevents decision paralysis and documents rationale.

6.5. Common Decision Patterns

Despite complexity, common patterns emerge from these constraints:

High-volume, fixed document types (processing millions of invoices monthly): Fine-tuned layout models offer the best accuracy-cost ratio after initial training investment. The high upfront cost amortizes across millions of documents.

Low-volume, diverse types (law firm processing varied contracts): General VLM APIs provide zero-shot flexibility without training overhead. Cost per document is higher but total cost is lower without training expenses.

Critical accuracy with mixed types (healthcare records requiring 98%+ accuracy): Hybrid approaches combining OCR precision with VLM reasoning work best. Use OCR for printed text where it excels, VLM for validation and reasoning where it excels. Recent methods like DocVLM [4] demonstrate that such hybrid approaches can achieve high accuracy while maintaining efficiency.

Real-time extraction with simple layouts (mobile check deposit capturing account numbers): Traditional OCR’s millisecond latency remains unmatched. Accuracy suffices for simple structured fields.

Multipage document processing (analyzing research papers, legal contracts, comprehensive reports): Models with extended context windows (Gemini 3 Pro’s 1M tokens) or token-efficient architectures (DocVLM’s compression approach) enable processing entire documents. Evaluate using benchmarks like Multipage Document Visual Question Answering (MP-DocVQA) and Document Understanding Dataset and Evaluation (DUDE) that specifically test cross-page reasoning capabilities.

7. Discussion: Why Tradeoffs Persist

A natural question is whether continued scaling and architectural innovation will eventually produce a single model that dominates across all dimensions. We argue this is unlikely for fundamental reasons.

The tension between pixel precision and semantic reasoning stems from competing optimization objectives, not insufficient scale. High-resolution visual processing demands dense patch embeddings and fine-grained attention—computationally expensive operations that scale quadratically with resolution. Semantic reasoning demands large language model backbones trained on diverse corpora—expensive in parameters and inference cost. Combining both means paying both costs simultaneously. Innovations like DocVLM [4] show that hybrid approaches can mitigate this tension (80% fewer tokens while improving accuracy), but they add architectural complexity rather than eliminating the underlying tradeoff.

We therefore argue that the document understanding landscape will remain stratified. Classical OCR will persist for high-volume, latency-sensitive extraction of clean text. Document-specific VLMs will serve precision-critical applications where reasoning is secondary. General VLMs will dominate where flexibility, reasoning, and zero-shot adaptation outweigh cost concerns. Efficiency-optimized variants and retrieval-augmented architectures will fill the gaps between these niches, each occupying a different region of the capability–cost space.

The practical implication is that model selection cannot be reduced to “choose the most capable model.” It requires understanding the dimensional tradeoff space we have outlined and making deliberate choices aligned with application requirements. We believe the five-dimensional framework proposed here provides a more principled basis for these decisions than benchmark leaderboards or marketing claims.

8. Conclusions

This paper has surveyed the evolution of document understanding from OCR to VLMs and argued that the field’s architectural divergence into precision-optimized and reasoning-optimized paths reflects fundamental tradeoffs rather than transient limitations. Our five-dimensional capability framework—text recognition, structural understanding, output flexibility, spatial awareness, and task adaptability—provides a principled vocabulary for characterizing models beyond discrete categories.

We have taken the position that convergence between architectural paths, while progressing, will not eliminate the need for deliberate model selection. The competing demands of pixel precision and semantic reasoning impose costs that cannot be simultaneously minimized. Practitioners are better served by dimensional capability mapping—explicitly characterizing requirements and matching them to architectural strengths—than by chasing benchmark numbers or defaulting to the largest available model.

As the field continues to mature, we expect the tradeoff space itself to shift: efficiency innovations will lower costs, hybrid architectures will expand the Pareto frontier, and retrieval-augmented approaches will redefine what “processing a document” means. But the need for principled navigation of architectural tradeoffs will persist, and we hope the frameworks presented here contribute to that effort.