Debug a RAG trace: the assistant cited a superseded policy

    Medium20 minDebug the trace · Exact answer + fix reviewCFDE Pro
    • retrieval freshness
    • eval design

    The brief

    An HR policy assistant at a GCC in Bengaluru gave this answer, and the employee's manager raised it with HR. The current India leave policy (v4, effective 1 April 2026) gives 15 working days of paternity leave. Here is the trace for the request.

    [1] user        How many days of paternity leave do I get? I'm in Bengaluru, grade L4.
                    (user profile: country=IN, grade=L4)
    
    [2] rewrite     query = "paternity leave entitlement India grade L4"
    
    [3] retrieve    index=hr-policies, top_k=6
                    filters = {"country": "IN"}
        results
          #1 score 0.891  IN-Leave-Policy-v3.pdf §4.2  status=superseded
                          effective_from=2023-04-01  effective_to=2026-03-31
                          "Male employees are eligible for 5 working days of paternity leave..."
          #2 score 0.874  IN-Leave-Policy-v4.pdf §4.2  status=current
                          effective_from=2026-04-01  effective_to=null
                          "Employees are eligible for 15 working days of paternity leave,
                           to be availed within 6 months of the child's birth or adoption..."
          #3 score 0.802  IN-Leave-FAQ-2024.pdf  status=superseded ...
          #4 score 0.711  IN-Holiday-Calendar-2026.pdf  status=current ...
    
    [4] rerank      keep top 2 -> [#1, #2]
    
    [5] generate    model answer:
                    "You are eligible for 5 working days of paternity leave
                     (IN-Leave-Policy-v3, section 4.2)."

    The generation prompt says: "Answer the question using the context. Cite the source."

    Your answer

    Which step is the root cause of the wrong answer?

    The change you would make so this cannot happen again. Two or three concrete points.

    0/120 words

    Graded by AI against the ADaSci rubric. Your answers stay private; only your score shows on your profile.