GNN-RAG: Enhancing the Reasoning Capabilities of LLMs using GNNs

Graph Neural Networks (GNNs) are a type of deep learning model specifically designed to work with graph data where nodes represent entities and edges represent the relationships between the entities. Question Answering over Knowledge Graphs relies on understanding the relationships between entities in a knowledge graph. GNNs excel at this because they are specifically designed to process graph data and extract meaning from the relationships between the corresponding nodes and edges. GNNs can also handle multi-hop reasoning as they can follow the relationship chains and reach the answer node. GNN-RAG is a recent development which combines the strengths of GNNs and LLMs. GNNs excel at reasoning over graph data and LLMs are known for their language understanding and generation capabilities. GNN-RAG utilises these benefits to achieve an effective KGQA (Question Answering over Knowledge Graphs). This article explores GNN-RAG and explains it. 

Table of Contents

1. Existing KGQA Methods
2. Understanding GNN-RAG
3. Benchmarks

Existing KGQA Methods

KGQA uses knowledge graphs to store information in a structured format with entities and relationships between them. For instance, if Ned Stark and Jon Snow are 2 entities then the relationship between them is Father-Son. This approach can access, process and generate factually grounded answers that are relevant to the context of the user’s query. 

Initially, A KGQA system tries to understand the intent behind the user query using NLP. It identifies the key nodes and relationships. Once this identification is complete, the system searches for relevant nodes and relationships within the knowledge graph using techniques such as NER. The system then navigates the knowledge graph and retrieves the answer for further translation it into a natural language response as the final answer to the user query. 

The Landscape of Existing KGQA Methods

The performance of retrieval augmented generation highly relies on the knowledge graph-based retrieval in these scenarios. The primary challenge is that the knowledge graphs store complex and heavy graph-based information which might be confusing for the LLM during KGQA reasoning. Also, the multi-hop reasoning might suffer if the knowledge graphs lack clear paths when understood by the LLM. These challenges can be avoided if there is a graph representation learner, which can handle complex and heavy graph-based information stored in a knowledge graph. GNN-RAG is one such powerful graph representation learner that can be used for retrieval purposes and the reasoning part can be taken care of by the LLM with RAG. 

GNN-RAG outperforms existing KG-RAG

Understanding GNN-RAG

GNN-RAG uses the concept of GNN on the knowledge graph, where it analyses the graph structure and the different relationships between the nodes to retrieve relevant answer candidates. Once, the GNN identifies the potential answer candidates, GNN-RAG extracts the shortest paths in the knowledge graph which connects nodes to the answer candidates. These paths represent the chain of relationships leading to the final answer (output). The system translates the identified shortest paths into human-readable sentences that explain the reasoning process behind the answer candidates. After this process, known as verbalising paths, GNN-RAG passes the question along with the verbalised shortest paths to LLM. The LLM is then able to analyse and select the most relevant and factually accurate answer.

GNN-RAG Framework

The retrieval of reasoning paths is implemented using SOTA GNNs for KGQA as GNNs can explore diverse reasoning paths that result in a high answer recall rate. After obtaining the reasoning paths, verbalisation (conversion to natural language) is accomplished and this is passed to an LLM and inference is generated through RAG.  

Benchmarks

GNN-RAG achieves SOTA performance in two widely used KGQA benchmarks namely, WebQSP and CWQ. WebSQP (WebQuestions Semantic Parses) is a small QA dataset with 4,737 questions which are 1-hop and 2-hop in nature whereas, CWQ (Complex Web Questions) is a dataset for answering complex questions that require reasoning over multi-web snippets. It contains 34,689 examples, each containing a complex question, answers, an average of 366.8 snippets per question and a SPARQL query. 

GNN-RAG outperforms GPT-4 performance with a 7B tuned LLM and improves performance on standard LLMs in KGQA performance with fewer LLM calls during RAG. It outperforms RoG (LLM-based retrieval) by 6.5-17.2% at F1 on WebQSP and by 8.5-9.5% points at F1 on CWQ. 

Final Words

Overall, GNN-RAG is a promising approach for efficient and increased performance in terms of question answering over knowledge graphs. The combination of GNN’s reasoning power and the expertise of LLMs leads to more accurate, explainable and powerful KGQA systems. 

References

1. GNN-RAG: Graph Neural Retrieval for Large Language Model Reasoning
2. A Survey on Knowledge Graphs: Representation, Acquisition and Applications
3. Improving Multi-hop KGQA using Knowledge Base Embeddings 
4. WebSQP Benchmark Dataset
5. CWQ Benchmark Dataset

Learn more through our hand-picked modules: