I’m working on an AI application that uses a large language model for question answering over internal documentation. While retrieval augmentation has improved factual accuracy, the model still occasionally generates confident but incorrect responses when the retrieved context is incomplete or ambiguous.

A simplified version of the inference pipeline looks like this:

retrieved_docs = retriever.search(query, top_k=5)

prompt = f"""
Use ONLY the information below to answer the question.

Context:
{retrieved_docs}

Question:
{query}
"""

response = llm.generate(prompt)

I’ve experimented with increasing retrieval depth, adjusting chunk sizes, and rewriting prompts, but there’s always a trade-off between factual accuracy, latency, and response quality.

For those building production AI systems:

• How do you measure and mitigate hallucinations beyond prompt engineering?
• Have you found techniques like reranking, verification models, or multi-agent validation to be effective?
• What evaluation metrics do you rely on to determine whether changes actually improve factual reliability?

I’m particularly interested in approaches that have worked well in production rather than benchmark experiments.