Preprints.org logo
Preprint
Essay

This version is not peer-reviewed.

From the Encyclopaedia to the Vector: The Corpus Pipeline as Epistemic Act in Psychoanalytic AI

  † PhD in Psychoanalysis (Universidade Veiga de Almeida) and MSc in Engineering (COPPE/UFRJ). Founder and researcher at TMU-LAB (The Machine Unconscious Lab), dedicated to the intersection of psychoanalysis and artificial intelligence.

Submitted:

22 June 2026

Posted:

24 June 2026

You are already at the latest version

Abstract
This essay argues that, in artificial intelligence systems applied to psychoanalytic domains, the quality of the knowledge-acquisition pipeline can weigh as heavily as the language model in determining the quality of the system, and sometimes more. It presents a four-stage pipeline that transforms publicly accessible academic encyclopaedias into a structured vector database: standardised crawling with traceability, hierarchical chunking that preserves the semantic structure of the original text, enrichment through knowledge graphs and fragmented LLM extraction, and local multilingual vectorisation. It documents the Terminology Guard as a protection layer against the terminological poisoning that automatic translation models produce in specialised domains. It proposes four concepts: the pipeline as epistemic act, hierarchical chunking as preservation of meaning, the Terminology Guard as defence against terminological epistemicide, and traceability as the ethical condition of citation.
Keywords: 
corpus pipeline
;  
knowledge acquisition
;  
RAG
;  
hierarchical chunking
;  
terminology guard
;  
terminological poisoning
;  
traceability
;  
AI and psychoanalysis
;  
epistemology of acquisition
Subject: 
Arts and Humanities  -   Philosophy

The Sources: Four Textual Corpora and One Relational Layer

The choice of sources is not arbitrary and deserves justification that is technical as much as epistemological. Each encyclopaedia contributes a kind of knowledge that the others cannot supply, and the absence of any one would create gaps that the system could not compensate with the rest.
The Stanford Encyclopedia of Philosophy is possibly the most rigorous philosophical encyclopaedia available publicly. Its entries are complete academic articles, with an argumentative density comparable to peer-reviewed journals, written by recognised specialists and submitted to a rigorous editorial process. Maintained as a freely available project by Stanford University since 1995, it now holds more than 1,800 entries covering Western philosophy from the Pre-Socratics to the twenty-first century. For a system that researches psychoanalysis, the SEP serves three irreplaceable functions. The first is to provide rigorous conceptual definitions of the philosophers Lacan read, cited, and transformed: Heidegger, Hegel, Husserl, Saussure, Kant, Bentham. Lacan’s reception of Hegel, for instance, is not direct: it passes through Alexandre Kojève, whose particular reading of the Phenomenology of Spirit, centred on desire and recognition, is the Hegel that Lacan knew between 1933 and 1939. A system that does not know what Kojève did with Hegel cannot explain what Lacan did with Kojève. The SEP provides entries on Kojève, on the French reception of Hegel, on the Hegelian dialectic, composing the intellectual context without which Lacanian teaching remains opaque. The second function is to provide perspectives that the continental psychoanalytic tradition frequently ignores: Anglo-American analytic philosophy, the logic of Frege and Russell, the philosophy of language. The third is to offer conceptual genealogies that no psychoanalytic dictionary covers, because psychoanalysis is not made of psychoanalysis alone.
The Internet Encyclopedia of Philosophy operates in a similar format, with entries written by specialists but oriented towards accessibility and didactic clarity. Where the SEP tends towards the long and dense entry, the IEP tends towards the more accessible point of entry, useful for readers at different stages of formation. For the corpus, the IEP fulfils specific functions: it covers authors that the SEP, by its analytic orientation, under-represents, such as Merleau-Ponty, Bataille, Levinas, Fanon, philosophers relevant to the psychoanalytic field whose SEP entries are scarcer or more technical than is needed. The IEP also provides a more didactic treatment of complex concepts, useful when the system has to answer researchers in training: an IEP entry on the Hegelian dialectic, being less technical than the SEP’s, may be more useful to someone entering the field than the in-depth discussion the SEP offers. Far from being redundancy, this gradation of complexity among sources lets the system adjust the resolution of the answer to the level of the question.
NoSubject.com is a wiki dedicated to Lacanian terminology, possibly the most complete online resource on the theme in English. Its terminological coverage includes matemes, the graph of desire, the signifying chain, suture, retroactive punctuation, the subject supposed to know, concepts that the general philosophical encyclopaedias do not cover with the necessary specificity. For each term, it indicates the Seminars that are the primary source, the equivalents in different translation traditions, and the connections with other concepts of the field. The collaborative nature of NoSubject introduces a variability of quality that the pipeline treats as a declared limitation: where there is conflict with the primary corpus, Lacan’s own texts prevail.
Wikipedia and Wikidata fulfil complementary functions. Wikipedia provides biographical and contextual information of encyclopaedic reach for hundreds of authors and movements. Wikidata, as the structured database of the Wikimedia Foundation, offers typed relations with formal properties: P106 for occupations, P737 for influences received, P135 for intellectual movement, P800 for notable works. Wikidata records influence in one direction, through the property P737, “influenced by”; the inverse, whom a thinker influenced, is obtained by querying which items point to the entity through P737, since the former inverse property was deprecated. From these typed relations one can build graphs of intellectual influence with direction, something no narrative text offers with the same structure. Wikidata says that Lacan was influenced by Hegel. The human researcher qualifies: what kind of intellectual operation? Torsion? Appropriation? Subversion? The structured datum provides the factual skeleton upon which specialist curation operates.

Crawling: From the Page to the Traceable Document

Before describing the technical operation of crawling, a prior decision needs to be named: the exclusive choice of publicly accessible sources. The four sources are publicly accessible and free to consult, though public access is not the same as an open licence: the Internet Encyclopedia of Philosophy and the Stanford Encyclopedia of Philosophy retain copyright over their entries, and their archives and mirrors are fixed editions provided for access and citation rather than a general licence to redistribute. That choice is not pragmatic, even though pragmatism sustains it. It is an epistemological position: the system I built should not depend on paid databases whose access is restricted to institutionally affiliated researchers. The peripheral researcher, with no university affiliation that grants access to proprietary databases, should be able to build an equivalent system with the same sources. The option for public access limits the corpus, excluding high-quality commercial databases, yet it democratises the possibility of replication. And in psychoanalysis, where training happens outside the universities as much as within them, that democratisation is a condition of coherence.
Because public access and an open licence are distinct, the corpus records, for each source, the edition or capture date, the method of acquisition, and the terms that apply. The processing is internal: the pipeline stores structured representations for retrieval, and the system shows the user short fragments together with a link back to the original entry, so that attribution and verification are preserved rather than the source being republished in full. Entries are refreshed when a new edition of a mirror becomes available, and can be removed on request or when terms change. Where a source forbids redistribution, the system treats it as a pointer to be cited and consulted at origin, not as content to be served. This is the same ethics of the source that the Terminology Guard expresses at the level of words, now at the level of rights.
The encyclopaedias are accessible over the web, but direct crawling presents problems that make systematic use unfeasible. Rate limits imposed by the servers make the processing of thousands of entries impractical. The dependence on connectivity makes the pipeline fragile. And the volume of requests may violate terms of use. Where a source provides an authorised archive or mirror, the pipeline works from that fixed local edition: the SEP, for instance, makes a complete archive available periodically for academic research, and that archive is the edition the pipeline processes. Other sources are accessed through source-specific mechanisms consistent with their terms and are not mirrored without permission. NoSubject is a case in point: its terms permit only a temporary personal-use copy for viewing and forbid copying, modifying, transferring, or mirroring its content to another server, so the pipeline does not mirror it; it reads the entries through a terms-compatible access mechanism and stores only the derived structured representations and short attributed fragments. With this arrangement the processing is fast, offline wherever an authorised archive exists, repeatable, and consistent with the terms of each source.
The architecture I developed separates the generic engine from the source-specific adapters. When a new encyclopaedia is added to the corpus, only the specific adapter needs to be written. The generic engine provides three shared components: the source registrar, which keeps centralised metadata about each encyclopaedia; the generator of deterministic identifiers, which computes a UUID v5 from the file path in the local mirror, guaranteeing idempotency; and the HTML processor, which converts each page into clean Markdown, preserving the main text with its heading structure and removing navigation, footers, scripts, and noise. Each specific crawler implements only two components: the file-discovery logic and the CSS content selectors. They all produce the same standardised output structure, with uniform fields: deterministic identifier, local path, reconstructed web URL, title, content, and source metadata. That standardisation is what lets the downstream stages operate uniformly over heterogeneous sources.
A design principle that pervades the whole pipeline is complete traceability: each vectorised chunk can be traced back to the original entry, and each citation the system produces can be verified by the researcher with a single click. That traceability is not a comfort feature: it is a condition of academic trust with direct ethical implications. A system that cites without allowing verification invites hallucination. The researcher who cannot verify learns not to verify, and the system that learns that it is not verified learns to be less precise. Traceability closes that loop: the researcher who verifies regularly discovers the errors, and the system that is verified regularly is kept honest. I propose traceability as the ethical condition of citation in AI systems, rather than as an accessory technical feature.

Hierarchical Chunking: To Preserve the Structure Is to Preserve the Meaning

The Failure of Conventional Chunking

Conventional chunking treats the text as a linear sequence of characters. It divides it into blocks of N tokens with an overlap of M tokens. It works reasonably for generic text with little structure. It fails systematically for academic text with strong hierarchical structure. A Stanford Encyclopedia entry on Heidegger has clearly demarcated sections: life and formation, influences, phenomenology of Dasein, being-towards-death, the turn, reception and influence, bibliographies. These sections are not arbitrary divisions: they are semantic units that carry different kinds of information about the same author.
I ask the reader to imagine the following situation. A chunk that crosses the boundary between “phenomenology of Dasein” and “being-towards-death” contains information from two different sections with no indication that a transition occurred. The vector system treats that chunk as a semantic unit and produces an embedding that mixes two contexts. A search for “Heidegger Dasein” may return that hybrid chunk, which is technically relevant, since it contains the word “Dasein,” and semantically imprecise, since it mixes two distinct aspects of the thought. The problem is aggravated by the fact that the position of a statement within a section carries information that the flat text does not carry: the sentence “Lacan was influenced by Hegel” has a different weight in the section “Biography” and in the section “Key Concepts.” Conventional chunking erases that difference.
In informal development comparisons, rather than in a controlled study, the difference between fixed-size chunks, 512 tokens with an overlap of 50, and hierarchical chunks with semantic classification was large and consistent: fewer irrelevant chunks returned, a higher proportion of relevant chunks, and the near-disappearance of temporal contradictions, the chunks that would mix definitions from different periods of the same concept. I report these as formative observations from building the system, without a pre-registered protocol, a gold set, or independent raters; the section on limitations returns to their status. What they reflect is a qualitative change: the system moved from retrieving text that contains the term to retrieving text that has the semantic function adequate to the query.

SmartHtmlProcessor: Chunking Inspired by TreeRAG

The SmartHtmlProcessor implements hierarchical chunking in the spirit of the tree-structured and structure-aware approaches that the recent literature describes, such as TreeRAG (Tao et al., 2025) and BookRAG (Wang, Zhou, et al., 2025). The fundamental principle is to replace division by fixed size with division at the natural boundaries of the document, the H1, H2, and H3 headings, propagating the complete hierarchical context to each resulting chunk. The operation can be described in three steps: first, the HTML is parsed to extract the heading structure; second, the text is divided at the heading boundaries, creating chunks that correspond to complete sections; third, each chunk receives as metadata the complete hierarchical context, H1 as the entry, H2 as the section, H3 as the subsection.
The result is that each chunk produced knows where it stands in the structure of the document. A chunk on the Hegelian dialectic knows that it belongs to the entry “Hegel” (H1), to the section “Phenomenology of Spirit” (H2), to the subsection “Master-Slave Dialectic” (H3). That hierarchical context is stored together with the vector embedding and can be used as a filter in the search: when the system needs biographical context about Hegel, it searches in chunks classified as biography; when it needs the definition of “dialectic,” it searches in chunks classified as concept. The filter by section type transforms the vector search, which was retrieval of similar text, into retrieval of knowledge of the adequate kind.
Semantic classification operates by detecting patterns in the headings and is multilingual by necessity: the corpus includes entries in English, Portuguese, French, and Spanish. “Biography,” “Vie,” “Biografía,” “Vida e obra” are all classified as biography. “Key Concepts,” “Pensée principale,” “Pensamiento,” “Teoria” are classified as concept. The classifier uses regex with word-boundary matching to avoid false positives, and bibliographical chunks are marked with a flag that excludes them from the main semantic search, because lists of references produce low-quality embeddings for conceptual searches, although they are preserved for specific queries of the bibliographical kind. The effect is to constrain retrieval towards conceptual passages rather than to guarantee that every retrieved chunk is a concept.
I confess that the implementation of hierarchical chunking tripled the development time compared with conventional chunking. The gain, however, was transformative: the system came to distinguish between “Lacan on the Real in 1959” and “Lacan on the Real in 1973” because the chunks carried temporal and contextual information that flat chunking destroyed. It is no exaggeration to say that hierarchical chunking was the technical decision that turned the system from a sophisticated database into something that deserves the name of a knowledge base.

Enrichment: From Text to Structured Knowledge

Enrichment through Wikidata is the stage that turns encyclopaedic data into relational knowledge. For each author identified in the corpus, the pipeline queries the Wikidata API and extracts standardised properties: occupations, influences received, authors influenced, intellectual movement, notable works, dates, and nationality. The values are resolved multilingually, with a preference for the Portuguese label and a fallback to English. The asymmetry of influence is valuable here: Wikidata records who influenced a thinker through the property P737, and the reverse direction, whom a thinker influenced, is recovered by querying which entities point back through P737. From these typed relations one builds graphs of intellectual influence with direction, rather than mere lists of associated names. The genealogical graph of influences, fed by these typed relations, lets the system map where Lacan took his concepts from and how he transformed them, distinguishing the factual skeleton of the relation, given by Wikidata, from the qualification of the kind of intellectual operation, made by the human researcher. The integration uses the Wikidata entity identifiers as foreign keys, which lets text chunks from different sources be connected to the same author profile in the relational graph, without duplication and with idempotency.
For authors and concepts that demand more than relational data, I implemented fragmented LLM extraction. The central principle is the division of tasks: instead of sending the complete entry to the model with the instruction to “extract everything,” I divide the extraction into seven smaller and specialised prompts that run in parallel, each responsible for a specific kind of information. The identity step extracts full name, dates, and pseudonyms. The classification step extracts occupations with adequate granularity: “phenomenological philosopher,” “Lacanian psychoanalyst,” distinctions that the generic type “philosopher” does not capture. The concepts step operates in two cycles, first identifying names, then defining each concept with a short definition, a full definition, and an importance index. The relations step extracts intellectual allies, antagonists, influences received, and disciples. The style step extracts notes on terminological conventions and prohibitions: for Freud, the prohibition on using “cathexis,” Strachey’s translation error, and the recommendation to use “investment” or the German “Besetzung”; for Lacan, the prohibition on translating “jouissance” without a note and the recommendation to preserve “objet petit a” untranslated. These notes feed directly into the Terminology Guard.
Not every extraction requires a language model. For key concepts and predictable textual patterns, I implemented regex-based heuristics that extract information at no API cost and with millisecond speed. The heuristics identify terms introduced by the author (“introduced the concept of,” “coined the term,” “developed the theory of”), terms in quotation marks or italics that are candidates for technical concepts, and patterns of antagonistic relation (“broke with,” “opposed the view that,” “was critical of”). In development, the heuristics recovered most of the key concepts that the LLM would identify in the same text, at no API cost and with negligible latency, leaving the concepts introduced without the conventional patterns to be captured by the LLM. I report this proportion as a formative observation rather than as a measured benchmark. The heuristics and the LLM are complementary: the heuristics cover the predictable, the LLM covers what resists prediction.

The Terminology Guard: Against Terminological Poisoning

The most critical dimension of the pipeline is the Terminology Guard, a layer of terminological protection that intercepts automatic translations before they enter the system’s cache. The problem the Guard resolves is cache poisoning: the automatic translation models, including Meta’s 3.3-billion-parameter NLLB (NLLB Team, 2022), produce translations that are correct for everyday use and incorrect for technical psychoanalytic use.
Three examples make the problem concrete. The NLLB translates “foreclosure” as “exclusão” when the correct rendering is “foraclusão.” The difference is not merely terminological: it is clinical. “Foraclusão” is the Lacanian concept that distinguishes the psychotic structure from the neurotic one; “exclusão” is a word from the common vocabulary that erases the distinction the concept carries, a distinction that participates in the Lacanian structural account of psychosis and that a careful clinical reading depends on. The second example: the NLLB translates “repression” as “repressão” when the correct rendering is “recalque.” Freudian “repression,” Verdrängung, is a mechanism of the psychic apparatus that operates upon unconscious representations; “repressão” in the current sense designates a political or social operation. When the system’s cache records “repressão” where it should record “recalque,” every future search for that concept returns contaminated results. The third example is the most insidious: the NLLB translates “recalque” from Portuguese into English as “recalculating,” using the technical-geographical sense of the term rather than the psychoanalytic one. A translation error in the cache is worse than an occasional translation error: it is a systematic and silent error.
The Guard currently maintains full mappings across Portuguese, English, and French, with their correct equivalents and the common incorrect variants, while preserving a curated set of German source terms, such as Trieb, Verdrängung, and Besetzung, as canonical lemmas rather than treating German as a fully supported translation direction. The intervention happens at two moments: in the input translation, when the researcher’s query is translated into other languages for cross-language search, and in the output translation, when chunks in French or German are translated for presentation. At both moments, the technical terms are replaced by the correct equivalents before the text reaches the translation model. In internal development tests with the 3.3B NLLB variant, on the FR→PT and PT→EN pairs, the guard markedly improved the rendering of psychoanalytic terms in both directions; I report this as a formative observation rather than as a benchmark, since the results published by the NLLB authors concern general translation quality across many language pairs (NLLB Team, 2022) and not this terminological layer.
The list of 57 terms was built from fifteen years of clinical and teaching experience in psychoanalysis, identifying the most frequent and most consequential errors in the reception of the theory. The untouchable terms, which must never be translated, include jouissance, objet petit a, sinthome, lalangue, parlêtre, Écrits. The terms with a specific translation include Trieb (pulsão, not instinto), Verdrängung (recalque, not repressão), Besetzung (investimento, not catexia), forclusion (foraclusão, not exclusão). The curation of that list is an example of what Siriwardhana et al. (2023, p. 1) would call domain adaptation at the level of the infrastructure: the list cannot responsibly be delegated to AI alone; it requires expert curation and continuing review, the kind that decades of familiarity with the terminology and with the debates on translation make possible. The human makes the list once, and revisits it; the system applies it systematically.
I propose the concept of terminological epistemicide to designate that destruction of the technical sense by automatic translation, taking up Santos (2014, p. 92) in a dimension the author did not anticipate: the destruction of forms of knowledge by the translating machine. The Terminology Guard is the operationalised defence against that epistemicide.
A late discovery in development revealed a vulnerability that the apparent robustness of the pipeline concealed. The skills, the instruction blocks that guide the behaviour of the system in different contexts of use, were loaded without validation: if a file was corrupted, badly formatted, or incompatible with the active mode of operation, the pipeline did not detect it. It injected corrupted text into the prompt and the language model did what it could with defective material. The quality degraded without anyone knowing why.
The solution transposes to the field of the skills the same logic that the Terminology Guard applies to the terms: each skill declares, in a structured manifest, the modes of operation in which it is valid, the triggers that must be present to activate it, the languages in which it is available, the token limit it may consume, and the security level it demands. The system validates the manifest before loading the skill. If the skill declares that it operates only in reflective mode and the system is in holding mode, the skill is not loaded. If the skill exceeds the declared token limit, the system issues a warning. If the skill has no manifest, it is loaded with a warning of unverified compatibility.
The analogy with the formation of the analyst is direct. An analyst in training does not operate techniques they have not mastered: they do not make transferential interpretation before having been supervised in that practice, they do not see children before having specific training, they do not work with psychosis before having experience with the structure. The skill manifest is the computational equivalent of that formative restriction: it declares what the skill can do, under what conditions it may operate, and what lies beyond its reach. The system that respects those declarations is safer than the system that loads everything indiscriminately.

Vectorisation: From Text to Semantic Space

The last stage of the pipeline is the transformation of the enriched chunks into semantic vectors. The model used is BGE-M3 (Chen et al., 2024b), which operates in 1,024 dimensions and natively supports multilingualism. The choice is deliberate: in psychoanalytic domains, where the fundamental texts are in German, French, and Portuguese, a model that processes the three languages in the same vector space allows cross-language search without prior translation. The researcher who asks in Portuguese finds the relevant passage of a text in French, and the semantic proximity is computed in the vector space, with no intermediate translation. The difference between “objet a as das Ding” in Seminar VII and “objet a as surplus-jouissance” in Seminar XX, a subtle distinction that depends on theoretical context, has more room to be separated in a 1,024-dimension multilingual space than in the 384 dimensions of the MiniLM model used in the first version of the system, although dimensionality alone does not guarantee that the distinction is captured.
BGE-M3 supports three modes of representation, dense, sparse, and multi-vector retrieval, the latter being ColBERT-style, which can be combined for different kinds of search: dense mode for semantic search, sparse mode for keyword matching, combined through Reciprocal Rank Fusion (Cormack et al., 2009, p. 758) for hybrid search. That combination is particularly relevant for the psychoanalytic corpus: a search for “jouissance Lacan” has to find both chunks that discuss the concept semantically, without mentioning the word, and chunks that contain the specific technical term. The dense mode finds the former; the sparse mode finds the latter; the fusion brings both together without one cancelling the other. Liu et al. (2024) showed that language models tend to under-use information positioned in the middle of long contexts, the so-called “lost in the middle” problem. Hybrid search does not by itself solve that positional problem, which concerns how the model uses the context it receives; what it does is diversify the retrieval strategies and the origins of the retrieved chunks, reducing reliance on any single ranking.
The model runs locally, with no calls to external APIs, which means no external API cost, since the computational, energy, and maintenance costs remain, together with privacy and a high degree of reproducibility: under a fixed model, fixed weights, and a fixed library, configuration, and computing environment, the same text yields the same embeddings. The vector store is LanceDB, file-based and local. The choice over alternatives such as Chroma, Qdrant, or Pinecone reflects the nature of the system as a personal research platform: there is no network latency, no hosting cost, no dependence on an external service. The schema of each record captures the dimensions of structured knowledge that distinguish the pipeline from conventional chunking: deterministic identifier, source path, title and section, complete hierarchical context H1/H2/H3, semantic classification, text in Markdown, and the 1,024-dimension vector.
The idempotency of the pipeline is an aspect that seems minor but is a condition of iterative development. The same source file always produces the same UUID v5, because it is computed deterministically from the file path; the same chunks, because the SmartHtmlProcessor is deterministic for the same version of the HTML; and the same embeddings, because the local model does not change between runs. This means the pipeline can be re-run as many times as necessary without creating duplicates. Idempotency also allows partial updates: when a new entry is added to the local mirror, only that entry is processed; when an existing entry is updated, the pipeline detects the change and offers the option of selective reprocessing. That granularity of control is impossible with pipelines that do not guarantee end-to-end determinism.

What the Pipeline Produces: Numbers and Scale

The result of the pipeline is a corpus of 257,130 chunks in 1,024-dimension vectors, drawn from four textual corpus sources, the Stanford Encyclopedia of Philosophy, the Internet Encyclopedia of Philosophy, NoSubject, and Wikipedia, with Wikidata as a relational enrichment source, classified into six semantic categories, traceable to the source URL, and protected by 57 terms of the Terminology Guard. The Stanford Encyclopedia contributes approximately 1,800 entries with an average of 15,000 words per entry, around 27 million words of rigorous philosophy. The Internet Encyclopedia contributes approximately 700 entries. NoSubject contributes approximately 500 entries on Lacanian terminology. Wikipedia adds biographical and contextual entries for hundreds of authors and movements. Wikidata provides structured relational data on hundreds of authors. These counts reflect the state of each source at the capture date recorded for the corpus; the live encyclopaedias continue to grow beyond the figures recorded here.
The total processing time is approximately two hours on consumer hardware. The total storage is below 5GB, modest considering that it covers a broad portion of the philosophical and psychoanalytic material relevant to the project, in four languages. The scale is modest if compared with the billions of tokens of the language models. The density is another matter: each chunk knows what it is, where it came from, in what context it stood, and to which kind of knowledge it belongs.
These numbers say something that goes beyond scale. 257,130 structured chunks represent a corpus that no individual researcher could master in a career, and that the system can search in milliseconds. The difference between having that corpus as books on the shelves, accessible but not systematically searchable, and having it as a structured vector base, accessible and searchable by semantics, by section type, by language, by author, is the difference between possibility and practicality. The pipeline does not produce new knowledge: it turns existing knowledge into knowledge accessible at the resolution adequate to each kind of query.
Ram et al. (2023, p. 1316) argue that the integration of external context at inference time is the central operation of RAG systems. That formulation is precise, but incomplete: it describes what the system does with the knowledge without describing how the knowledge became available in the adequate form. The acquisition pipeline is the operation that makes possible the integration that Ram et al. describe. Without chunks that know what they are, the integration is blind; with structured chunks, the integration is informed.
The acquisition pipeline is not an autonomous system: it is infrastructure that feeds systems with distinct operational logics. A system of academic production consumes the pipeline for search with section-type filters, using the hierarchical context H1/H2/H3 to contextualise citations: when it recovers a chunk that says that Kojève defines Hegelian desire as the desire for recognition, it knows, by the metadata, that this information comes from the “Kojève” entry of the SEP, in the section “Key Concepts,” subsection “Desire and Recognition,” and it can trace back to the original URL for verification. Statements with no identified chunk are flagged as potential hallucinations. A system of clinical listening consumes the pipeline for a completely different function: the naming of affects. When a person describes a suffering without managing to name it, the semantic search in the pipeline with a conceptual-type filter recovers concepts that describe similar experiences, and the semantic classification constrains retrieval towards passages classified as conceptual, those that name experiences, rather than biographical data about those who described them. A system of memory consumes the structured profiles produced by the fragmented extraction and by the Wikidata enrichment, feeding biographical, relational, semantic, and stylistic layers. The pipeline is the entry point of the whole infrastructure: without it the layers would be empty, and without the layers the systems could not operate at the resolution the use cases demand. That one and the same pipeline feeds systems so different, one that writes, one that listens, and one that remembers, is the strongest evidence that the epistemic act of ingestion is prior to and independent of the act of use.

Epistemology of Acquisition

The main contribution of this essay is an argument: in specialised domains, the quality of the ingestion pipeline can weigh as heavily as the choice of language model, and sometimes more. A modest model supported by a carefully structured pipeline may outperform a stronger model supplied with context-poor fixed-size chunks; the controlled ablation required to establish that comparison remains future work. I present this as a design argument grounded in months of iteration, in which improvements in the pipeline produced gains that changes of model did not, rather than as a controlled result; a proper test would hold the pipeline fixed while varying the model, and the model fixed while varying the pipeline. The design consequence inverts the common priorities of the field: before choosing the language model, invest in the pipeline; before optimising the search, optimise the ingestion. Chen et al. (2024a) showed that RAG performance depends on several capacities that remain fragile, including noise robustness, negative rejection, and information integration; and the segmentation and quality of the chunks, in turn, affect both retrieval and the answer (Wang, Gao, et al., 2025). The quality of retrieval is, in this sense, limited by the quality of ingestion. The bottleneck of the bottleneck is the pipeline.
What does it mean to say that the pipeline is an epistemic act? It means that each technical decision of the pipeline embodies a position on what counts as knowledge. The decision to divide the text by hierarchical structure, rather than by fixed size, is a position on the relation between structure and meaning: to preserve the first is a condition of preserving the second. The decision to classify each chunk semantically is a position on taxonomy: the kind of information a passage carries, whether it is biography, concept, reception, or bibliography, affects what one can do with it. The decision to keep complete traceability is a position on verifiability: the knowledge that cannot be verified slides into belief. And the decision to protect the terminology through the Terminology Guard is a position on the irreducible specificity of fields of knowledge: there are terms whose technical sense admits no generic translation, and the machine that translates generically destroys the meaning it should preserve.
If the question is whether we are building a knowledge machine or a sophisticated database, the answer depends on what one understands by knowledge. If knowledge is stored information, we have a database. If knowledge is information that is structured, contextualised, relational, and traceable, organised in a way that allows retrieval with understanding, then the pipeline produces something that deserves the name of knowledge. The chunks are not passages of text: they are units of meaning with identity, context, type, and relations.
I hesitated before using the word “knowledge” to describe what the pipeline produces, because I know that epistemology has reasons to reserve it for the subject who knows. The machine does not know. The pipeline does not know. What the pipeline does is organise the material so that the human researcher who interacts with the system can know better, distinguish with more rigour, trace with more precision. The pipeline is an instrument of knowledge, not a subject of knowledge, and that distinction is what prevents the technical contribution from sliding into fetish. The Lacanian equivalent would be to say that the pipeline inscribes the Real of the corpus, the texts in their raw and dispersed materiality, into the Symbolic of the system as structured knowledge. Semantic classification is the process of symbolisation. The chunking hierarchy is the structural inscription: each chunk knows its place in the symbolic chain of the document. And the Terminology Guard is what prevents the process of symbolisation from introducing errors that would contaminate the whole chain.
An important limitation needs to be named with clarity: the pipeline depends on publicly accessible web sources, and the primary works, Lacan’s Seminars, Freud’s Complete Works, the Écrits, philosophers’ texts in critical editions, are not freely available on the open web and cannot be ingested by crawling. These works are added to the corpus by a separate process of digitisation and curation maintained since 2008 with more than 1,100 documents. The encyclopaedia pipeline covers the secondary and tertiary sources that provide the context for the primary sources to be well used. It is like the difference between having the books and having the shelves, the indexes, and the catalogues: the pipeline builds the infrastructure of contextualisation without which the primary texts would be islands with no bridges.
Diderot framed the aim of the Encyclopédie as ensuring that the work of past centuries would not be useless to the centuries to come (Diderot, 1755). The pipeline I present is, on a modest scale and without the totalising ambition of eighteenth-century encyclopedism, an attempt to make the work of the academic encyclopaedias of the twenty-first century useful to an artificial-intelligence system that researches, writes, and listens. The raw material is public. The transformation is the contribution. And that transformation, from encyclopaedia to vector, from text to structured knowledge, from data to knowledge with type and context, is the epistemic act that sustains any AI system in specialised domains.
I confess that there is a disquiet this essay cannot resolve and that perhaps it should not try to. At the end of eighteen years of digitisation and two years of intensive development of the pipeline, the corpus exists: 257,130 chunks with identity, context, type, traceability, terminological protection. Yet each chunk is a fragment, and the relation between fragment and totality is the question psychoanalysis has posed ever since Freud: the meaning is not in the isolated fragment, it is in the articulation among fragments, in the chain, in the gaps between links. The pipeline structures the fragments and declares the possible relations. The human researcher operates the articulations and listens to the gaps. And it is in that interval, between what the machine organises and what the subject articulates, that knowledge happens.

Limitations

This is a conceptual and design essay, and its claims should be read as such. The pipeline exists and operates, and the corpus it produced is real, but the comparative figures reported here, on noise, precision, temporal contradictions, heuristic coverage, and terminological improvement, are formative observations from building the system rather than the results of a controlled evaluation. They were produced without a pre-registered protocol, a gold set, independently trained raters, inter-rater agreement, fixed baselines, recorded model versions and parameters, or confidence intervals, and the comparative thesis that the pipeline can weigh as heavily as the model is presented as an argument rather than as a measured ablation. A proper evaluation would hold the pipeline constant while varying the model, and the model constant while varying the pipeline, and would report metrics, hardware, and uncertainty. The corpus also rests on copyrighted sources consulted under their own terms, which constrains redistribution and shapes what the system may serve rather than cite. These are the natural next steps for turning the design argument of this essay into a documented technical result.

Author Contributions

H.A.R.B. is the sole author and is responsible for the conceptualisation, the theoretical analysis, the design of the pipeline, and the writing of this manuscript.

Funding

This research received no external funding.

Data Availability Statement

The four textual corpus sources described, the Stanford Encyclopedia of Philosophy, the Internet Encyclopedia of Philosophy, NoSubject, and Wikipedia, together with the Wikidata relational layer, are publicly accessible under their respective terms. The derived indices, prompt libraries, and the curated primary corpus contain proprietary system-development material, and the primary texts retain their respective copyrights; they are therefore not publicly deposited. The pipeline architecture, the schema of the vector records, and the Terminology Guard list are available from the author on reasonable request.

Ethics Statement

This manuscript reports no study involving human participants and uses no identifiable clinical material. It describes the design of research software for the acquisition and structuring of a documentary corpus assembled from publicly accessible sources under their respective terms.

Acknowledgments

The author thanks Véronique Donard for the prior collaborative work on the foundations of an ethical and decolonial AI, on which several of the concepts mobilised here build, and for her scientific supervision of the broader research within which the PhDSapiens project took shape.
Disclosure of AI Use: In preparing this manuscript the author used large language models during 2026 to assist with translation from Portuguese into English, drafting, language revision, and the organisation of references. All conceptual claims, interpretations, and final decisions remain the author’s own. The author reviewed and corrected the entire manuscript, verified the cited sources, and accepts full responsibility for the final text. No AI tool meets the criteria for authorship, and none is listed as an author. The system discussed in the essay (the corpus acquisition pipeline of PhDSapiens) is an object of the research, distinct from any tool used in manuscript preparation.

Conflicts of Interest

The author is the founder of TMU-LAB and the designer and developer of the corpus pipeline, part of the PhDSapiens project, discussed in this essay. The author declares no financial conflicts of interest.

References

  1. Chen, J.; Lin, H.; Han, X.; Sun, L. Benchmarking large language models in retrieval-augmented generation. Proceedings of the AAAI Conference on Artificial Intelligence 2024a, 38(16), 17754–17762. [Google Scholar]
  2. Chen, J.; Xiao, S.; Zhang, P.; Luo, K.; Lian, D.; Liu, Z. BGE M3-Embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv 2024b, arXiv:2402.03216. [Google Scholar]
  3. Cormack, G. V.; Clarke, C. L. A.; Büttcher, S. Reciprocal rank fusion outperforms Condorcet and individual rank learning methods. In Proceedings of the 32nd International ACM SIGIR Conference; 2009; pp. 758–759. [Google Scholar]
  4. de Moura Junior, P. R.; Lelong, J.; Blangero, A. Adaptive chunking: Optimizing chunking-method selection for RAG. In Proceedings of the 15th Language Resources and Evaluation Conference (LREC 2026); 2026. [Google Scholar]
  5. Diderot, D. Encyclopédie, ou dictionnaire raisonné des sciences, des arts et des métiers; Briasson, 1755. [Google Scholar]
  6. Gao, Y.; Xiong, Y.; Gao, X.; Jia, K.; Pan, J.; Bi, Y.; Dai, Y.; Sun, J.; Wang, H. Retrieval-augmented generation for large language models: A survey. arXiv 2024, arXiv:2312.10997. [Google Scholar]
  7. Lewis, P.; Perez, E.; Piktus, A.; Petroni, F.; Karpukhin, V.; Goyal, N.; Küttler, H.; Lewis, M.; Yih, W.; Rocktäschel, T.; Riedel, S.; Kiela, D. Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems 2020, 33, 9459–9474. [Google Scholar]
  8. Liu, N. F.; Lin, K.; Hewitt, J.; Paranjape, A.; Bevilacqua, M.; Petroni, F.; Liang, P. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics 2024, 12, 157–173. [Google Scholar] [CrossRef]
  9. NLLB Team. No language left behind: Scaling human-centered machine translation; Meta AI, 2022. [Google Scholar]
  10. Ram, O.; Levine, Y.; Dalmedigos, I.; Muhlgay, D.; Shashua, A.; Leyton-Brown, K.; Shoham, Y. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics 2023, 11, 1316–1331. [Google Scholar]
  11. Santos, B. S. Epistemologies of the South: Justice against epistemicide; Paradigm Publishers, 2014. [Google Scholar]
  12. Sharma, C. Retrieval-augmented generation: A comprehensive survey of architectures, enhancements, and robustness frontiers. arXiv 2025, arXiv:2506.00054. [Google Scholar]
  13. Siriwardhana, S.; Weerasekera, R.; Wen, E.; Kaluarachchi, T.; Rana, R.; Nanayakkara, S. Improving the domain adaptation of retrieval-augmented generation (RAG) models for open-domain question answering. Transactions of the Association for Computational Linguistics 2023, 11, 1–17. [Google Scholar]
  14. Tao, W.; Xing, X.; Chen, Y.; Huang, L.; Xu, X. TreeRAG: Unleashing the power of hierarchical storage for enhanced knowledge retrieval in long documents. In Findings of the Association for Computational Linguistics: ACL 2025; Association for Computational Linguistics, 2025; pp. 356–371. [Google Scholar] [CrossRef]
  15. Wang, S.; Zhou, Y.; Fang, Y. BookRAG: A hierarchical structure-aware index-based approach for retrieval-augmented generation on complex documents. arXiv 2025, arXiv:2512.03413. [Google Scholar]
  16. Wang, Z.; Gao, C.; Xiao, C.; Huang, Y.; Si, S.; Luo, K.; Bai, Y.; Li, W.; Duan, T.; Lv, C.; Lu, G.; Chen, G.; Qi, F.; Sun, M. Document segmentation matters for retrieval-augmented generation. In Findings of the Association for Computational Linguistics: ACL 2025; Association for Computational Linguistics, 2025; pp. 8063–8075. [Google Scholar] [CrossRef]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.