Skip to main content

RAG Hallucination

Overview​

Retrieval Relevance represents the relevance of the documents retrieved from the vector database using the embedding model for each query. To measure the retrieval relevance, Dynamo AI generates a relevance label with an LLM. Studies have shown the effectiveness of LLMs as reference-free evaluators for tasks such as content relevance.

Response Faithfulness represents the faithfulness of model generated responses to the retrieved documents. Dynamo AI uses a proprietary LLM to label the (context, response) pair as either faithful or not faithful. Our definition of faithfulness requires that every piece of information in the answer must be supported by information present in the context."

Response Relevance represents the relevance of model-generated responses to the query. To measure the response relevance, Dynamo AI generates a Response Relevance Score with an LLM.

RAG Eval Components

Metrics​

Retrieval Relevance This metric is computed by prompting an LLM to evaluate the sufficiency of the content in the retrieved documents to answer a given user query. For each query, DyanmoFL computes the relevance score against individual context retrieved from the vector database. If all of the retrieved contexts are flagged as not relevant, the retrieval will be flagged as not relevant.

Response Faithfulness To measure the response faithfulness, Dynamo has both custom fine-tuned models as well as prompted LLMs to evaluate whether the generated response contains well-supported facts from the retrieved context. It will evaluate the individual retrieved contexts with the response. If the response is not supported by all of the retrieved contexts, the response will be flagged as not faithful.

Response Relevance This metric is computed by prompting an LLM to generate a score between 0 and 1 that indicates how relevant the generated response is to the question. Based on this score, each (query, response) will be classified as either positive (relevant) or negative (not relevant). A negative classification indicates that the response does not contain information that addresses the user's input query.