Hyper-RAG: Combating LLM Hallucinations Using Hypergraph-Driven Retrieval-Augmented Generation

1. Results

To validate the effectiveness of the proposed Hyper-RAG method, we conduct experimental evaluations on nine corpus datasets across eight domains[19], with statistical details summarized in Table 1. While existing LLMs demonstrate strong performance on tasks with standardized answers, their performance on open-ended responses remains modest. Therefore, in this study, we employ domain-specific open-domain question-answering (QA) tasks to assess the Hyper-RAG strategy. We design two evaluation approaches for open-ended assessments: The first involve directly scoring each model’s output across five dimensions for comparative analysis, and the second entails conducting pairwise competitions where a large language model evaluates responses from two different models based on eight metrics and casts votes accordingly. Detailed procedures can be found in Section 4. We select prominent LLMs, including GPT-4o Mini, Qwen-Plus, LLaMa-3.3-70B, DeepSeek-V3, and Doubao-1.5-Pro, as baselines and applied various augmentation strategies, namely, RAG, GraphRAG, LightRAG, and our proposed Hyper-RAG—to evaluate their impact on model outputs. More information on those augmentation strategies is described in Section 3. Subsequently, we performed four sets of experiments to comprehensively assess our method.

1.1. Performance of Integrating with Diversity LLMs

We first conduct experiments to evaluate the performance of the proposed Hyper-RAG when collaborating with various LLMs in order to verify whether it can effectively enhance the accuracy of LLM responses while mitigating hallucinations. Given that medical data is replete with specialized knowledge—and that even the slightest deviation in terminology can precipitate severe consequences such as misdiagnosis—we performed a comprehensive comparative study using the NeurologyCorp dataset. This dataset comprises extensive records of neuroscience knowledge and clinical practices, making it an ideal benchmark for assessing precision in a high-stakes domain.

To set up the experiment, the corpus are segmented into 1,968,716 chunks, and distinct prior knowledge bases are constructed for each method. For the standard Retrieval-Augmented Generation (RAG) approach, embeddings are directly extracted from each chunk and stored in a vector database to facilitate knowledge retrieval. In contrast, both Graph RAG and Light RAG establish a graph-based knowledge base: from each chunk, entity information and paired correlations are extracted and stored in a graph database, with each entity and correlation accompanied by a brief textual description. Notably, when identical vertices or paired correlations emerge across multiple chunks, their descriptive texts are merged using a large model to ensure consistency. For Hyper-RAG, we built a hypergraph knowledge base from the original neuroscience corpus. From each chunk, we extracted not only entity information and paired correlations, but also higher-order correlations that transcend pairwise relationships, with every entity and correlation supplemented by a textual description. The primary distinction between Hyper-RAG and the conventional Graph RAG lies in its inclusion of non-paired, higher-order correlations, which results in a more comprehensive and structured representation of the source data. Detailed implementation specifics are provided in Section 3.

To facilitate a robust horizontal comparison across different methods and LLMs, we adopt a scoring-based assessment that quantifies response quality using five distinct metrics (see Section 4.1 for additional details). Prior to the experiments, 50 unique questions are randomly sampled from different chunks using the large models, ensuring that each LLM and augmentation strategy is evaluated on an identical set of queries. The experimental results are shown in Figure 3 and 4, where the baseline LLMs are presented alongside the performance improvements achieved via the different augmentation approaches.

From Figure 3 and 4, we have six key observations. First, compared to direct LLM responses, our proposed Hyper-RAG method yields an average improvement of 12.3%, as shown in Figure 3. Notably, Hyper-RAG enhances performance by 15.8% relative to Qwen-Plus and by 14.3% in comparison to GPT-4o mini, underscoring the value of constructing a hypergraph-based prior knowledge repository for elevating the quality of LLM outputs. Second, our findings corroborate that integrating a domain-specific prior knowledge base using the RAG strategy significantly boosts response quality. Specifically, a naive RAG approach improves baseline responses by 4.9% on average, while Graph RAG and Light RAG achieve enhancements of 6.3% and 6.0%, respectively. In contrast, Hyper-RAG delivers a 12.3% enhancement over the baseline, highlighting the critical role of domain knowledge in reinforcing LLM capabilities. Third, organizing the underlying corpus with structured associative frameworks markedly bolsters RAG performance. The introduction of relational structures yields a 7.0% improvement over unstructured methods. This increase likely stems from the fact that a well-structured representation assists in more efficient retrieval of pertinent information and promotes the diffusion of contextual cues along the relational network, thereby fostering a broader, more innovative spectrum of responses.

Fourth, both the baseline LLM and the RAG-augmented approaches demonstrate high scores in logical coherence and readability. This result reflects the extensive pre-training on large-scale corpora, which endows these models with inherent abilities to produce logically sound and accessible text regardless of the query context. Fifth, LLMs tend to score lower on metrics of comprehensiveness, diversity, and empowerment. These lower scores likely reflect intrinsic challenges in capturing nuanced domain-specific details and expressive capability. Encouragingly, the incorporation of prior corpus information via the RAG strategy results in an average improvement of 9.4% for these metrics, thereby partially offsetting these limitations. Sixth, baseline LLMs generally exhibit modest diversity scores—typically around 60, with model such as GLM, scoring as low as 53. In contrast, implementing Graph RAG elevates diversification by 19.3%, and our Hyper-RAG method further boosts the diversity score by 31.6%. This substantial gain can be attributed to the integration of additional correlation information, which more effectively steers responses towards greater divergence. Moreover, the comprehensive coverage of both lower-order and higher-order correlations cultivates a richer prior knowledge base, thereby driving significant improvements across all evaluation metrics.

1.2. Performance of Different Questioning Strategies

Given that our Hyper-RAG method provides a more comprehensive coverage of the knowledge embedded in the raw data, we further evaluated its capabilities by varying the difficulty of the questions. In our framework, questions are nested and asked progressively; the deeper the nesting, the greater the complexity of the task. This design is premised on the fact that a series of interdependent queries will magnify the impact of any inaccuracies in earlier responses, thereby serving as a stringent test of the LLM’s grasp of the domain knowledge. Table 2 illustrates examples of these progressive questions, where each subsequent inquiry is formulated based on the preceding one and connected by transitional terms (indicated in bold font). We categorized the questions into three tiers according to their escalation in difficulty: single-stage, two-stage, and three-stage questions. For this experiment, GPT-4o mini is employed as the baseline LLM, and we compare several enhancement strategies, including RAG, Graph RAG, Light RAG, and our Hyper-RAG. The experimental results are presented in Figure 5.

Figure 5 yields three key insights from our experiments. First, as evident from the first panel, Hyper-RAG consistently demonstrates stable performance improvement across various levels of question difficulty. This indicates that employing hypergraphs to represent the full spectrum of prior corpus knowledge effectively captures domain information and guides the LLM toward more accurate responses. Second, we observe that as the complexity of the questions increases, the performance of the baseline LLM gradually declines, from 75.2 to 73.7 and then to 72.8. A similar trend is observed in other RAG methods, such as Graph RAG, which reinforces the notion that more heavily nested queries place a higher demand on prior knowledge. Although all methods exhibit some degradation in performance with increasing difficulty, RAG-based approaches still manage to enhance performance relative to the original LLM. Notably, Light RAG, which omits clustering steps, loses a portion of this vital information, and its performance deteriorates more significantly as the questions become more complex.

Finally, our Hyper-RAG shows a more pronounced improvement as the difficulty increases. Specifically, relative to the baseline LLM, our method achieves incremental gains of 12.7%, 14.3%, and 15.0% as the question complexity escalates, while, when contrasted with Light RAG, the improvements are 8.7%, 9.7%, and 5.3%, respectively. These results substantiate the superiority of a hypergraph-based, comprehensive information representation. For straightforward queries, direct responses from an LLM or simple pairwise (i.e., low-order) correlations may suffice. However, as queries become more intricate, the availability of complex higher-order correlations becomes essential to constrain and enrich the model’s outputs. This experimental trend underscores the importance of developing hypergraph-based structural representations and retrieval methods to meet the challenges posed by increasingly complex questions.

1.3. Performance in Diversity Domains

To validate the adaptability of Hyper-RAG across various data domains, we further evaluate its effectiveness using nine corpora spanning eight different fields (for statistical details, see Table 1). In this experiment, we select GPT-4o mini as the baseline architecture and compared our method against Light RAG, the latest graph-based RAG approach. Notably, we adopt a Selection-Based Assessment to comprehensively compare the performance of the graph-based and hypergraph-based RAG methods across these diverse domains. This assessment involved voting across eight distinct evaluation metrics that collectively capture the strengths and weaknesses of both approaches. For further details on the evaluation criteria, please refer to Section 4.2. The experimental results are presented in Figure 6.

Based on experimental results shown in Figure 6, our Hyper-RAG method has demonstrated impressive improvements across nine datasets, with an average performance increase of 35.5%. Specifically, the method delivers a 55.3% improvement on LegalCrop, 41.3% on AgricCrop, and 37.5% on FinCrop, underscoring its effectiveness and adaptability across diverse domains. Although the enhancements in Accuracy and Relevance are relatively modest—averaging 29.8% and 32.0% respectively, indicating that existing low-order correlations provide a substantial baseline, Hyper-RAG exhibits notably stronger gains in Comprehensiveness and Coherence, with average improvements of 35.1% and 39.6%. These results can be attributed to Hyper-RAG’s unique capability to leverage both low- and high-order correlations, offering a more complete representation of the underlying data and enhancing response consistency through embedding-based retrieval from a vector database. Overall, while graph-based RAG methods may suffice for tasks primarily focused on accuracy and relevance, our Hyper-RAG method shows significant promise for more complex tasks that demand a broader and deeper domain understanding, paving the way for extensive future applications.

1.4. Experiments of Different Knowledge Representations

In this paper, we claim that using hypergraphs to structurally extract information from raw corpus data can more completely represent the inherent structure of the data. In this subsection, we perform an ablation study on our data organization methods based on three types of prior knowledge representations: D, $E_{\mathrm{low}}$, and $E_{\mathrm{high}}$. Here, D refers to directly splitting the raw corpus into chunks and using the embedding representation of each chunk for data retrieval; $E_{\mathrm{low}}$ captures pairwise knowledge correlations to construct a knowledge graph; and $E_{\mathrm{high}}$ extracts non-pairwise higher-order correlations to build a knowledge hypergraph. Based on these three fundamental representations, we can construct eight different types of knowledge representations, as shown in Table 3. Evidently, directly using the LLM involves no additional data organization; using only D corresponds to a simple RAG; combining D with $E_{\mathrm{low}}$ creates a variant similar to Graph RAG (with clustering removed from the original Graph RAG for a fair comparison); and employing all three yields our proposed Hyper-RAG. We employ GPT-4o mini as the foundational LLM and evaluated the various prior knowledge representation strategies using a scoring-based assessment (Section 4.2). The experimental results are presented in Table 3.

From the experimental results presented in Table 3, we draw the following four key observations. First, employing the comprehensive data representation strategy, referred to as Hyper-RAG, yields the highest performance with a score of 84.16. This superiority is attributed to the holistic organization of data, which effectively imparts prior knowledge to the large language model. Second, we observe that augmenting the model with knowledge representations significantly enhances performance compared to the baseline without such augmentation, as illustrated in the first row. Specifically, the use of any single knowledge representation method results in an improvement of at least 4.9%. This underscores the efficacy of knowledge augmentation strategies in enhancing the model’s ability to respond accurately within domain-specific contexts. Third, our findings indicate that utilizing the original corpus as supplementary information leads to better performance than relying solely on descriptions of entities and their correlations. The latter approach may introduce errors or hallucinations due to summarization by the large model, thereby negatively impacting the augmentation effectiveness. Lastly, when enhancing the model with a single type of correlation, high-order correlations outperform low-order ones. High-order correlations encompass more extensive information and cover a broader spectrum of knowledge within the correlation representation space, as depicted in Figure 1. In our current experiments, approximately 4,000 high-order and 13,000 low-order correlations were extracted from the prior corpus. Remarkably, the use of only high-order correlations resulted in superior performance, demonstrating a more effective enhancement of the large model. This indicates that a relatively smaller set of high-order correlations can encapsulate more substantial knowledge, thereby offering a promising new direction for the future development of RAG techniques.

1.5. Efficiency Analysis

We further conduct an efficiency analysis of the proposed method. Utilizing GPT-4o mini as the base model, we perform efficiency experiments on the NeurologyCrop dataset, comparing our Hyper-RAG approach with the fundamental RAG, Graph RAG, and Light RAG methods. To ensure a fair comparison unaffected by network latency, we exclusively evaluate the time required for local retrieval from the database to acquire relevant knowledge and the construction of the prior knowledge prompt. For the standard RAG, this primarily involves the direct retrieval time of chunk embeddings. In contrast, Graph RAG, Light RAG, and our Hyper-RAG method encompass both the retrieval time from node and correlation vector databases and the time required for a single layer of graph or hypergraph information diffusion. Since the retrieval time is influenced by both the specific questions and the methods employed, we calculate the average response time for each method by posing 50 questions from the dataset. Consistent with our selection metric outlined in Section 4, we employ a scoring-based assessment as the evaluation criterion. Additionally, to accommodate practical applications, we develop a lightweight variant of Hyper-RAG, termed Hyper-RAG-Lite, which preserves the essential enhancements for entity retrieval. The experimental results, presented in Figure 7, demonstrate the comparative efficiency of each method.

Section Figure 7a presents a comparative analysis of performance versus time, where points closer to the top-left corner indicate faster speeds and superior performance. From the figure, we derive the following four key observations: Firstly, we observe that both the proposed Hyper-RAG and Hyper-RAG-Lite are positioned near the top-left corner of the plot, indicating that these methods excel in both speed and performance. This demonstrates the efficacy of our approach in maintaining high efficiency without compromising answer quality. Secondly, we note that RAG is situated at the far left of the plot. This positioning is attributed to its sole reliance on document corpus retrieval without incorporating structural diffusion, which confers an efficiency advantage. However, this method’s performance significantly lags behind other structure-based enhanced methods, highlighting a trade-off between speed and accuracy. Thirdly, Graph RAG achieves a performance level that is second only to our Hyper-RAG method, yet it incurs considerable time delays. The primary reason for this sluggishness is the necessity of retrieving community information in addition to node information retrieval and diffusion. Community information is derived through hierarchical clustering of nodes and lacks indexing via vector databases, necessitating layer-by-layer matching and retrieval, thereby slowing down the process. Nevertheless, the inclusion of community information, which embodies high-order correlations, effectively complements pairwise graph correlations, thereby enhancing performance. Additionally, we observe that Light RAG omits the retrieval of community information, resulting in a reduction in performance. However, this omission leads to a substantial increase in processing speed, as the computational overhead associated with managing high-order correlations is eliminated. Lastly, our Hyper-RAG method exhibits performance comparable to Light RAG while maintaining superior speed. Both methods essentially rely on prompt-based correlation extraction and indexing through vector databases and graph/hypergraph databases. However, Hyper-RAG differentiates itself by extracting both low-order and high-order correlations via prompts and utilizing a hypergraph database for indexing, thereby achieving similar efficiency levels. Crucially, Hyper-RAG compensates for the information loss inherent in Light RAG by incorporating additional high-order correlations, resulting in enhanced performance. It is noteworthy that the Hyper-RAG-Lite variant, although retaining only entity information retrieval, still implements diffusion through high-order correlations. This ensures that Hyper-RAG-Lite introduces additional high-order information, thereby achieving performance improvements over both Light RAG and Graph RAG.

2. Discussion

In our study, we integrate Hyper-RAG with six widely used LLMs, demonstrating a significant enhancement in performance. On average, Hyper-RAG improves the models’ accuracy by 12.3% compared to their direct application without retrieval augmentation. When juxtaposed with the conventional Graph RAG approach, our method yielded an additional 5.3% improvement. This superior performance can be attributed to Hyper-RAG’s comprehensive coverage of domain-specific knowledge. By modeling both low-order (pairwise) and high-order (beyond-pairwise) correlations within the data, Hyper-RAG facilitates a more complete and structured representation of domain knowledge, thereby reducing information loss and enhancing the quality of the generated responses.

We also evaluate the robustness of Hyper-RAG by increasing the complexity of the questions through added nesting. The experimental results reveal that existing methods, including RAG and Graph-RAG, experienced a noticeable decline in performance under these more challenging conditions. In stark contrast, Hyper-RAG maintains its performance levels, underscoring the pivotal role of high-order correlations in enabling LLMs to handle complex queries effectively. This finding suggests that current LLMs possess untapped potential for improvement in complex question-answering scenarios and that the incorporation of high-order relational modeling can significantly bolster their ability to provide accurate and reliable responses.

Our investigations into knowledge representation highlight that the impact of prior knowledge on model performance varies across different scenarios. In simpler contexts, leveraging low-order correlations alongside the original prior corpus suffices to cover the necessary information. However, in more intricate scenarios, the inclusion of high-order correlations becomes imperative to enhance the accuracy of the model’s responses. This adaptability in knowledge representation allows for the selection of appropriate prior knowledge based on the complexity of the task at hand, thereby optimizing the model’s performance across diverse application domains.

Despite its advantages, Hyper-RAG presents certain limitations. The construction of the knowledge base necessitates the extraction of high-order correlations, which introduces additional steps into the knowledge base development process. Nonetheless, the number of high-order correlations is considerably smaller compared to low-order ones, mitigating the overall impact. Moreover, these extraction processes can be performed offline, thereby not impeding the real-time application of the models. In comparison, the classic Graph RAG approach relies on clustering to represent group correlations within the data, a process that is both time-consuming and resource-intensive. Light RAG, while alleviating the computational burden by omitting clustering, consequently loses high-order relational information, leading to diminished performance.

In the knowledge retrieval phase, Hyper-RAG offers distinct advantages over Graph-RAG by eliminating the need for redundant local and global retrieval processes. Instead, it allows for direct retrieval of nodes or relational structures. The retrieval of relational structures is optional; incorporating it can enhance performance but at the cost of additional computational resources. Alternatively, when only node information is retrieved, the system operates in a mode we designate as Hyper-RAG-Lite. In this mode, the integration of both low-order and high-order correlations enables the diffusion of information, thereby utilizing high-order knowledge embedded within the data. Consequently, Hyper-RAG-Lite not only accelerates the retrieval process but also improves the quality of responses generated by the LLMs, presenting a promising avenue for future research.

However, our current approach to knowledge construction faces challenges in directly extracting correlations across different data chunks, necessitating post-processing steps to merge these relations. A significant portion of the relevant information spans multiple chunks, making it inadequate to capture through a single chunk alone. Future research should focus on developing methods for the fusion and extraction of cross-chunk correlations. Additionally, modeling relationships between different documents could further enhance the dimensionality and scalability of the knowledge base. This study has demonstrated that improved representation and organization of domain knowledge can significantly enhance the capabilities of large language models. Future work may explore automating data organization and knowledge representation techniques, fostering a deeper integration with large language models to further mitigate issues such as hallucinations.

3. Methods

In this section, we provide a comprehensive overview of the proposed Hypergraph-Driven Retrieval-Augmented Generation (Hyper-RAG) framework, encompassing the processes of knowledge extraction, indexing, and retrieval. Subsequently, we present the architecture of the open-domain question answering tasks utilized to evaluate Hyper-RAG, alongside two distinct evaluation metrics that assess both the accuracy and comprehensiveness of the generated responses. These metrics are chosen to rigorously measure the reduction in hallucinations and the enhancement in answer reliability provided by our approach. Finally, we describe the data collection and construction procedures, detailing the sources and criteria for dataset assembly. Additionally, we illustrate the various prompt templates employed at different stages of the Hyper-RAG pipeline, demonstrating how they are tailored to optimize knowledge retrieval and generation processes.

Table 4 presents a mathematical comparison of various LLMs enhancement strategies. Here, $P_q$ denotes the function that transforms the input into a prompt, where q represents the input query and D is the prior corpus. The symbols V, $E\mathrm{low}$, and $E_{\mathrm{high}}$ correspond to the vertices, low-order correlation knowledge, and high-order correlation knowledge of the knowledge base constructed from the corpus, respectively. It is evident that the original LLM generates responses directly based on the input question q. In contrast, RAG retrieves relevant data from the prior corpus D to assist the LLMs in providing answers. Graph-RAG further extracts low-order structural information from the prior knowledge. Our proposed Hyper-RAG simultaneously constructs both low-order and high-order correlation information from the prior corpus, enabling a more comprehensive representation of knowledge. This enhanced knowledge representation effectively reduces information loss, thereby mitigating the occurrence of hallucinations in LLMs.

3.1. Framework Schema

Figure 8 illustrates the proposed Hyper-RAG framework, encompassing both the offline construction of the knowledge database and the online retrieval-augmented response generation processes. Initially, we collect domain-specific corpora, which may include manuals, books, reports, and other relevant documents. These raw corpora are then processed using LLMs to segment the text and extract entities and their relationships. In the Hyper-RAG framework, relationships are categorized into pairwise low-order correlations and beyond-pairwise high-order correlations that represent correlations among groups of entities. The extracted knowledge is subsequently stored in a database to facilitate rapid retrieval during the query phase.

During the question-answering process, consider an example where a user with neurological disorders poses a question to the LLMs, as shown in Figure 8. When using a naive LLM, the model responds directly to the patient’s query without additional context. In contrast, the Hyper-RAG strategy involves a two-step approach: first, we extract keywords from the user’s question; second, we retrieve knowledge related to these similar keywords from the knowledge database. The retrieved relevant knowledge is then provided as supporting information alongside the original question to the LLMs, resulting in more accurate and reliable responses. The subsequent sections will provide a detailed description of each component and process within the Hyper-RAG.

3.2. Knowledge Extraction

The objective of knowledge extraction is to systematically organize raw corpora into a structured format, thereby enabling more efficient retrieval of prior information. In our approach, the corpus data can comprise various types of documents, including books, manuals, reports, and other relevant texts. We begin by preprocessing the original documents and partitioning them into uniformly sized chunks, denoted as $D_i$, thereby forming the corpus collection:

$$D=\{D_1,D_2,\cdots ,D_N\}.$$

(1)

Subsequently, a document structuring function, $\varphi$, is employed to extract structural information from the corpus, resulting in a hypergraph $\mathcal{G}$:

$$G=\varphi \left(D\right)\mathrm{and}G=\{V,E_{\mathrm{low}},E_{\mathrm{high}}\},$$

(2)

where $\mathcal{G}$ represents the hypergraph structure extracted from the documents, comprising a set of vertices V, low-order correlations $E_{\mathrm{low}}$, and high-order correlations $E_{\mathrm{high}}$. The elements within the vertex set V can be of various types, such as names of entities, task titles, or skills. The relation sets $E_{\mathrm{low}}$ and $E_{\mathrm{high}}$ describe the connections between entities, where $E_{\mathrm{low}}$ captures pairwise relationships and $E_{\mathrm{high}}$ encapsulates correlations involving multiple entities. For each chunk $D_i\in D$, we extract entities and their descriptions using LLMs as follows:

$$K_v=\mathrm{LLM}\left(P_{\text{ext\_entity}}\left(D_i\right)\right)\mathrm{for}{D}_i\in D,$$

(3)

where $K_v=\{v_1,v_2,\cdots \}$ denotes the set of entities, each accompanied by a generated description, as illustrated in Figure 2. It is important to note that if multiple chunks contain the same entity, their descriptions are merged using the LLMs to ensure consistency and completeness. The function $P_{\text{ext\_entity}}$ serves as the prompt filler that converts the input into an appropriate prompt for entity extraction, which is detailed in Section 5.

Following entity extraction, we proceed to identify the corresponding low-order and high-order correlations within each chunk based on the extracted entities:

$$\left\{\begin{array}{cc}\hfill K_e^{\mathrm{low}}& =\mathrm{LLM}\left(P_{\text{ext\_low}}(D_i,K_v)\right)\hfill \\ \hfill K_e^{\mathrm{high}}& =\mathrm{LLM}\left(P_{\text{ext\_high}}(D_i,K_v)\right)\hfill \end{array}\right.\mathrm{for}{D}_i\in D,$$

(4)

where $K_e^{\mathrm{low}}=\{(u,v),\cdots \}$ represents the set of low-order correlations between pairs of entities, while $K_e^{\mathrm{high}}=\{(u,v,\cdots ),\cdots \}$ denotes the high-order correlations involving multiple entities. Each relation within the knowledge base is also accompanied by a descriptive narrative, as depicted in Figure 2. When identical correlations are extracted from different chunks, their descriptions are amalgamated to maintain a unified representation. This comprehensive extraction of both low-order and high-order relational information from the corpus ensures a robust and detailed knowledge base, which is critical for minimizing information loss and enhancing the retrieval process in the Hyper-RAG framework.

3.3. Knowledge Indexing

Hyper-RAG utilizes two distinct types of databases to effectively organize and manage the extracted knowledge: a vector database for storing the embedding representations of vertices and a hypergraph database for maintaining both high-order and low-order relational structures.

Vector Database 

The vector database stores fixed-dimensional vector representations derived from the descriptions of each entity. These embeddings are organized into a matrix M, where each row corresponds to the vector representation of an entity or a hyperedge. During retrieval, a query vector q is compared against the vectors in the database using distance metrics such as cosine similarity or Euclidean distance. The system then retrieves the top-k nearest vectors, which may correspond to relevant entities. This approach ensures that the most semantically similar entries are efficiently identified and utilized to enhance the generation process.

Hypergraph Database 

The hypergraph database stores the structural information extracted from the raw corpus, encompassing both low-order and high-order correlations. This database comprises two primary components: vertex adjacency lists and relation adjacency lists. A hypergraph extends a traditional graph by allowing hyperedges to connect more than two vertices, thereby uniformly storing both low-order (pairwise) and high-order (beyond-pairwise) correlations within this structure. Each hyperedge is represented as a tuple of vertex names, facilitating the representation of complex relationships among multiple entities. During retrieval, given a specific vertex name, the hypergraph database can swiftly access the vertex’s descriptive information as well as its connected relational structures. Additionally, the database supports relational queries where, upon inputting a particular relation structure, it returns the corresponding descriptive information and the neighboring vertices associated with that relation. This dual capability allows Hyper-RAG to efficiently navigate and utilize intricate relational data, thereby enhancing the accuracy and contextual relevance of responses generated by the LLMs.

Integration of Vector and Hypergraph Databases 

The integration of the vector database and the hypergraph database within the Hyper-RAG framework provides a comprehensive mechanism for knowledge storage and retrieval. While the vector database excels in capturing and retrieving semantically similar entities and correlations through embedding spaces, the hypergraph database ensures that the structural and relational integrity of the knowledge base is maintained and easily accessible. Together, these databases enable Hyper-RAG to leverage both the semantic richness and the structural complexity of the underlying knowledge, thereby effectively mitigating hallucinations in large language models by providing accurate and contextually relevant information.

3.4. Knowledge Retrieval and LLMs Augmentation

After constructing the hypergraph knowledge base offline, we detail the methodology for augmenting the LLMs’ response capabilities using knowledge databases. Given a user query q, we first extract two distinct sets of keywords: the entity keyword set $X_{ent}$ (fundamental components) and the correlation keyword set $X_{cor}$ (complex interdependencies), as follows:

$${\mathcal{X}}_{\text{ent}},{\mathcal{X}}_{\text{cor}}=\text{LLM}\left({\mathcal{P}}_{\text{ext\_key }}\left(q\right)\right)\text{ and }{\mathcal{X}}_{*}=\left\{x_1,x_2,\dots \right\},$$

(5)

where $P_{\text{ext\_key}}$ is the prompt used to extract keywords from the input question, as detailed in Section 5. The entity keyword set comprises specific detailed nouns, such as personal names, locations, and other discrete identifiers. In contrast, the correlation keyword set encompasses more sophisticated descriptions, typically involving interactions between two or more entities. These correlations often capture narratives, systems, responses, reactions, and various forms of interactions that emerge from the relationships among entities. By distinguishing between these two types of keyword sets, our approach effectively models both the fundamental components and the complex interdependencies within the data. This comprehensive representation enhances the retrieval-augmented generation process, enabling the LLMs to leverage a richer and more nuanced foundation of prior knowledge. Subsequently, based on these two categories of extracted keywords, we retrieve relevant information from the hypergraph database. It is important to note that entity keyword retrieval targets vertices, while correlation keyword retrieval targets hyperedges. This distinction arises because entity keywords predominantly describe individual entities, making vertices the appropriate retrieval objects. In contrast, correlation keywords describe abstract information that typically involves relationships among multiple entities, thereby necessitating hyperedges as retrieval targets. For entity information retrieval, we employ the following formulation:

$$\{\begin{array}{l}\begin{array}{cc}{\mathcal{V}}_{\text{rel}}& =\left\{{\psi }_{\text{ret}}\left(x_i,\mathcal{V}\right)|x_i\in {\mathcal{X}}_{\text{ent}}\right\}\end{array}\text{ //Entity information }\\ \\ \begin{array}{cc}{\epsilon }_{\text{more}}& =\left\{\mathcal{e}\right|\mathcal{v}\in \mathcal{e} \text{and} \mathcal{v}\in {\mathcal{V}}_{\text{rel}}\}\end{array}\text{ //Extended information via diffusion}\end{array},$$

(6)

where ${\psi }_{\mathrm{ret}}$ denotes the vector-based retrieval function, which retrieves vertices similar to the input $x_i$ from the vertex vector database. Subsequently, $E_{\mathrm{more}}$ is used to diffuse through the associated structural relationships, thereby obtaining hyperedges connected to these vertices as supplementary information.

Similarly, for correlation information retrieval, we use the following formulation:

$$\{\begin{array}{l}\begin{array}{cc}{\epsilon }_{\text{rel}}& =\left\{{\psi }_{\text{ret}}\left(x_i,\epsilon \right)|x_i\in {\mathcal{X}}_{\text{cor}}\right\}\end{array}\text{ //Correlation information}\\ \\ \begin{array}{cc}{\mathcal{V}}_{\text{more}}& =\left\{\mathcal{v}\right|\mathcal{v}\in \mathcal{e} \text{and} \mathcal{e}\in {\epsilon }_{\text{rel}}\}\end{array}\text{ //Extended information via diffusion}\end{array},$$

(7)

where ${\psi }_{\mathrm{ret}}(x_i,E)$ retrieves hyperedges related to the correlation keywords from the hyperedge vector database. Through one-step diffusion, the vertices associated with these hyperedges are acquired as supplementary information. Due to the constraints on the LLM’s input context length, we aggregate and rank the retrieved entity and correlation information, selecting the most relevant data based on the maximum permissible context length to serve as prior knowledge for augmenting the LLM. In practical applications, we incorporate the textual content from the original chunks associated with the relevant vertices and hyperedges as prior information. This approach is employed to mitigate the potential hallucinations that may arise from descriptions synthesized by the LLM, thereby ensuring the reliability of the supplementary knowledge.

3.5. Dataset

To assess the efficacy of the proposed Hyper-RAG framework, we curate an extensive collection of corpora encompassing both domain-specific and mixed-domain datasets. Recognizing that domain-specific data is more susceptible to hallucinations, owing to the heightened demands for lexical precision in specialized fields, we selected nine corpora across eight distinct domains: medicine, mathematics, agriculture, finance, physics, law, and art, the comprehensive statistics of which are detailed in Table 1. Additionally, to evaluate the model’s performance in managing general knowledge across diverse areas, we construct a mixed-domain dataset. These corpora, referenced in [19,43], are primarily extracted from books, reports, academic papers, narratives, and encyclopedias, with an average token count of $2,733,191$ per dataset. Each raw corpus underwent preprocessing to eliminate special symbols and non-textual elements, retaining solely the textual information. Subsequently, the sanitized corpora were partitioned into fixed-size chunks of 1200 tokens, with an overlapping segment of 100 tokens between consecutive chunks to ensure contextual coherence. For performance evaluation, a LLMs (GPT-4o mini) is employed to generate 50 questions per dataset. The question generation leverage the $P_{\text{ext\_q}}\left(q\right)$ prompt tailored to each chunk, facilitating automatic question formulation by the LLM. Furthermore, the origin chunk for each question are recorded to enable Scoring-Based Assessment, wherein the corresponding source chunk served as the reference answer for evaluating the responses.

4. Evaluation Criteria

The evaluation of LLMs has predominantly been conducted using benchmarks with predefined answers, such as SQuAD, GPT-3 Benchmarks, and others. These benchmarks typically involve generating concise responses that align with standard answers. However, in real-world applications, obtaining standard answers is often impractical, especially for open-ended questions where responses can vary widely. In such scenarios, the absence of supervisory information makes LLMs more prone to hallucinations, leading to the confusion of critical entities like names, dates, and locations. This issue is particularly detrimental in sensitive domains such as medicine, where inaccuracies can result in significant consequences. To effectively evaluate LLMs in open-ended contexts, we introduce two assessment strategies: Scoring-Based Assessment and Selection-Based Assessment.

4.1. Scoring-Based Assessment

Scoring-Based Assessment is designed to facilitate the comparative evaluation of multiple model outputs by quantifying their performance across various dimensions. This approach allows for a nuanced assessment of model capabilities by providing scores on several key metrics. However, a notable limitation is its reliance on reference answers. In our preprocessing steps, we leverage the source chunks from which each question is derived as reference answers. Using these references, we construct a scoring prompt, denoted as ${\mathcal{P}}_{\text{eval\_score}}$, which directs the LLM to evaluate open-ended responses based on five dimensions:

• Comprehensiveness (0-100): Assesses whether the response sufficiently addresses all relevant aspects of the question without omitting critical information.
• Diversity (0-100): Evaluates the richness of the content, including additional related knowledge beyond the direct answer.
• Empowerment (0-100): Measures the credibility of the response, ensuring it is free from hallucinations and instills confidence in the reader regarding its accuracy.
• Logical (0-100): Determines the coherence and clarity of the response, ensuring that the arguments are logically structured and well-articulated.
• Readability (0-100): Examines the organization and formatting of the response, ensuring it is easy to read and understand.

Each evaluation dimension is scored on a scale from 0 to 100, with higher scores indicating better performance. Recognizing the challenges associated with assigning broad numerical scores directly, we implemented a hierarchical scoring system by dividing each dimension into five distinct levels. Each level corresponds to specific criteria that provide clear and consistent guidelines for scoring. To illustrate, we present the classification for the Comprehensiveness dimension:

• Level 1 | 0-20: The answer is extremely one-sided, leaving out key parts or important aspects of the question.
• Level 2 | 20-40: The answer has some content but misses many important aspects and is not comprehensive enough.
• Level 3 | 40-60: The answer is more comprehensive, covering the main aspects of the question, but there are still some omissions.
• Level 4 | 60-80: The answer is comprehensive, covering most aspects of the question with few omissions.
• Level 5 | 80-100: The answer is extremely comprehensive, covering all aspects of the question with no omissions, enabling the reader to gain a complete understanding.

While the detailed five-level classification is exemplified for the Comprehensiveness dimension, similar hierarchical structures have been established for the other evaluation metrics (Diversity, Empowerment, Logical, and Readability) to ensure uniformity and precision in the scoring process. Finally, an overall performance score is calculated as the average of the individual dimension scores. A higher overall performance score indicates greater accuracy in expression and a lower probability of hallucinations.

4.2. Selection-Based Assessment

Selection-Based Assessment is tailored for scenarios where preliminary candidate models are available, enabling a comparative evaluation through a binary choice mechanism. This method does not require reference answers, making it suitable for diverse and open-ended questions. However, its limitation lies in its comparative nature, as it only allows for the evaluation of two models at a time.

In this strategy, the outputs from two methods, denoted as $A_{\mathrm{out}}$ and $B_{\mathrm{out}}$, are simultaneously presented to the LLM, denoted as ${\mathcal{P}}_{\text{eval\_select}}$. The model is then instructed to select the better response based on eight evaluation criteria:

• Comprehensiveness: How much detail does the answer provide to cover all aspects and details of the question?
• Empowerment: How well does the answer help the reader understand and make informed judgments about the topic?
• Accuracy: How well does the answer align with factual truth and avoid hallucination based on the retrieved context?
• Relevance: How precisely does the answer address the core aspects of the question without including unnecessary information?
• Coherence: How well does the system integrate and synthesize information from multiple sources into a logically flowing response?
• Clarity: How well does the system provide complete information while avoiding unnecessary verbosity and redundancy?
• Logical: How well does the system maintain consistent logical arguments without contradicting itself across the response?
• Flexibility: How well does the system handle various question formats, tones, and levels of complexity?

For each of these eight criteria, the LLM selects the superior response between $A_{\mathrm{out}}$ and $B_{\mathrm{out}}$. The cumulative votes across all criteria determine the overall score for each model. This voting mechanism ensures a balanced evaluation based on multiple facets of response quality, thereby providing a robust assessment of the models’ relative performance without the need for predefined reference answers.

5. Prompts

5.1. Extracting Entities, Correlations and Keywords

 

5.2. Evaluation