Understanding FLAME, The Factuality Aware Alignment for LLMs

The ability of large language models (LLMs) to comprehend intricate instructions and produce thorough responses has completely changed artificial intelligence. Hallucinations, or the creation of misleading information, are still a serious problem, though. In order to overcome this difficulty, FLAME (Factuality-Aware Alignment for LLMs) introduces novel training procedures intended to increase factual correctness without sacrificing the ability to follow instructions. The architecture, technique, and effects of FLAME on lowering hallucinations in LLMs are examined in this article.

Table of Content

1. What is FLAME?
2. Key Innovations of FLAME
3. FLAME’s Methodology
4. Pilot Studies and Results

What is FLAME?

FLAME is a novel alignment technique designed to enhance the factuality of LLMs during the training process. It combines factuality-aware supervised fine-tuning (SFT) and reinforcement learning (RL) through Direct Preference Optimisation (DPO), in contrast to traditional approaches that emphasise instruction-following capabilities, frequently at the expense of factual correctness. It ensures that LLMs generate dependable results by focussing on particular sources of hallucinations while maintaining user engagement.

(a) Few-shot LLM response generation; (b) Factuality-aware alignment.

Key Innovations of FLAME

Factuality-Aware Supervised Fine-Tuning (SFT )

• Leveraging Internal Knowledge: FLAME seeks responses from the pre-trained LLM (PT) itself for fact-based instructions rather than fine-tuning directly on human-generated data. This reduces the amount of new information that is introduced that might not be supported by the model’s current body of knowledge.
• Selective Data Usage: To optimize data utilisation for various instruction types, it deliberately uses LLM-generated responses for fact-based instructions and human-generated responses for non-fact-based instructions.

Factuality-Aware Direct Preference Optimization (DPO )

Multi-Objective Training: FLAME incorporates two distinct reward models:

• RMIF (Instruction-Following Reward Model): Assesses how well the generated answer complies with the specified instruction.
• RMfact (Factuality Reward Model): It assesses the accuracy of factual claims within the response.

Addressing Single-Scalar Limitations: FLAME overcomes the drawbacks of single-scalar reward functions, which might not adequately account for both factual correctness and instruction adherence, by utilising distinct reward models.

FLAME’s Key Features

FLAME’s Methodology

Step 1: Identifying Fact-Based Instructions

FLAME categorises if a particular instruction calls for factual correctness. It adjusts alignment tactics for fact-based instructions so that factuality takes precedence over other answer attributes.

Step 2: Factuality-Aware Supervised Fine-Tuning

It employs the replies of the pre-trained LLM rather than human-curated training data, which might contain new information. By doing this, new information that can cause hallucinations is avoided.

Step 3: Factuality-Aware Reinforcement Learning

During the DPO stage, FLAME takes factuality-specific preference pairs into account. Retrieval-augmented claim verification is used to evaluate responses, guaranteeing that factually accurate answers are rewarded.

Pilot Studies and Results

Biography Generation Task

FLAME demonstrated a significant improvement in factual accuracy during a pilot study on biography generation. By avoiding reliance on external knowledge sources and focusing on the model’s own output, it reduced hallucination as measured by FACTSCORE.

Diverse Instruction Tasks

When tested on Alpaca Eval and other datasets, FLAME achieved:

• A 5.6-point increase in factual accuracy (FACTSCORE).
• A win rate of 51.2% in instruction-following tasks, comparable to standard alignment methods.

Instruction generation comparisons: rare vs. frequent entities.

Final Words

The alignment of LLMs to prioritise factual correctness while maintaining their capacity to produce interesting and creative responses is greatly improved by FLAME. It establishes a new benchmark for ethical AI development by using multi-objective optimization to address the underlying causes of hallucinations. In the quest for reliable and adaptable AI systems, this represents a significant advancement.

References

1. FLAME Research Paper