ADaSci Banner 2024

GNN-RAG: Enhancing the Reasoning Capabilities of LLMs using GNNs

GNN-RAG, a recent development, synergises GNNs and LLMs to excel in KGQA.
GNN-RAG: Enhancing the Reasoning Capabilities of LLMs using GNNs

Graph Neural Networks (GNNs) are a type of deep learning model specifically designed to work with graph data where nodes represent entities and edges represent the relationships between the entities. Question Answering over Knowledge Graphs relies on understanding the relationships between entities in a knowledge graph. GNNs excel at this because they are specifically designed to process graph data and extract meaning from the relationships between the corresponding nodes and edges. GNNs can also handle multi-hop reasoning as they can follow the relationship chains and reach the answer node. GNN-RAG is a recent development which combines the strengths of GNNs and LLMs. GNNs excel at reasoning over graph data and LLMs are known for their language understanding and generation capabilities. GNN-RAG utilises these benefits to achieve an effective KGQA (Question Answering over Knowledge Graphs). This article explores GNN-RAG and explains it. 

Table of Contents

  1. Existing KGQA Methods
  2. Understanding GNN-RAG
  3. Benchmarks

Existing KGQA Methods

KGQA uses knowledge graphs to store information in a structured format with entities and relationships between them. For instance, if Ned Stark and Jon Snow are 2 entities then the relationship between them is Father-Son. This approach can access, process and generate factually grounded answers that are relevant to the context of the user’s query. 

Initially, A KGQA system tries to understand the intent behind the user query using NLP. It identifies the key nodes and relationships. Once this identification is complete, the system searches for relevant nodes and relationships within the knowledge graph using techniques such as NER. The system then navigates the knowledge graph and retrieves the answer for further translation it into a natural language response as the final answer to the user query. 

The Landscape of Existing KGQA Methods

The performance of retrieval augmented generation highly relies on the knowledge graph-based retrieval in these scenarios. The primary challenge is that the knowledge graphs store complex and heavy graph-based information which might be confusing for the LLM during KGQA reasoning. Also, the multi-hop reasoning might suffer if the knowledge graphs lack clear paths when understood by the LLM. These challenges can be avoided if there is a graph representation learner, which can handle complex and heavy graph-based information stored in a knowledge graph. GNN-RAG is one such powerful graph representation learner that can be used for retrieval purposes and the reasoning part can be taken care of by the LLM with RAG

GNN-RAG outperforms existing KG-RAG

Understanding GNN-RAG

GNN-RAG uses the concept of GNN on the knowledge graph, where it analyses the graph structure and the different relationships between the nodes to retrieve relevant answer candidates. Once, the GNN identifies the potential answer candidates, GNN-RAG extracts the shortest paths in the knowledge graph which connects nodes to the answer candidates. These paths represent the chain of relationships leading to the final answer (output). The system translates the identified shortest paths into human-readable sentences that explain the reasoning process behind the answer candidates. After this process, known as verbalising paths, GNN-RAG passes the question along with the verbalised shortest paths to LLM. The LLM is then able to analyse and select the most relevant and factually accurate answer.

GNN-RAG Framework

The retrieval of reasoning paths is implemented using SOTA GNNs for KGQA as GNNs can explore diverse reasoning paths that result in a high answer recall rate. After obtaining the reasoning paths, verbalisation (conversion to natural language) is accomplished and this is passed to an LLM and inference is generated through RAG.  

Benchmarks

GNN-RAG achieves SOTA performance in two widely used KGQA benchmarks namely, WebQSP and CWQ. WebSQP (WebQuestions Semantic Parses) is a small QA dataset with 4,737 questions which are 1-hop and 2-hop in nature whereas, CWQ (Complex Web Questions) is a dataset for answering complex questions that require reasoning over multi-web snippets. It contains 34,689 examples, each containing a complex question, answers, an average of 366.8 snippets per question and a SPARQL query. 

GNN-RAG outperforms GPT-4 performance with a 7B tuned LLM and improves performance on standard LLMs in KGQA performance with fewer LLM calls during RAG. It outperforms RoG (LLM-based retrieval) by 6.5-17.2% at F1 on WebQSP and by 8.5-9.5% points at F1 on CWQ. 

Final Words

Overall, GNN-RAG is a promising approach for efficient and increased performance in terms of question answering over knowledge graphs. The combination of GNN’s reasoning power and the expertise of LLMs leads to more accurate, explainable and powerful KGQA systems. 

References

  1. GNN-RAG: Graph Neural Retrieval for Large Language Model Reasoning
  2. A Survey on Knowledge Graphs: Representation, Acquisition and Applications
  3. Improving Multi-hop KGQA using Knowledge Base Embeddings 
  4. WebSQP Benchmark Dataset
  5. CWQ Benchmark Dataset

Learn more through our hand-picked modules:

Picture of Sachin Tripathi

Sachin Tripathi

Sachin Tripathi is the Manager of AI Research at AIM, with over a decade of experience in AI and Machine Learning. An expert in generative AI and large language models (LLMs), Sachin excels in education, delivering effective training programs. His expertise also includes programming, big data analytics, and cybersecurity. Known for simplifying complex concepts, Sachin is a leading figure in AI education and professional development.

The Chartered Data Scientist Designation

Achieve the highest distinction in the data science profession.

Elevate Your Team's AI Skills with our Proven Training Programs

Strengthen Critical AI Skills with Trusted Generative AI Training by Association of Data Scientists.

Our Accreditations

Get global recognition for AI skills

Chartered Data Scientist (CDS™)

The highest distinction in the data science profession. Not just earn a charter, but use it as a designation.

Certified Data Scientist - Associate Level

Global recognition of data science skills at the beginner level.

Certified Generative AI Engineer

An upskilling-linked certification initiative designed to recognize talent in generative AI and large language models

Join thousands of members and receive all benefits.

Become Our Member

We offer both Individual & Institutional Membership.