How does RAG improve LLM-based applications?

Here is an example. You built a copilot application on top of OpenAI API. The first realization will be that the latest update was on Jan 2022. As of today, the delay is almost two years. Then, for some questions, you will receive wrong answers. You can add context to the question to fix hallucinations, but a context window has a token limit. The last problem is that ChatGPT was trained on publicly available data and knows nothing about your private data.

RAG or Retrieval-Augmented Generation adds the following capabilities to your app:

• Knowledge of up-to-date information

• Better quality by reduced hallucinations

• Access to private data

• Solves limitations of the context window

In the most basic case, RAG implementation requires:

1. Split your documents into chunks

2. Generate embeddings for every chunk

3. Save embeddings in the vector store

4. During user request, find the most relevant content in the vector store - retrieval phase.

5. Add relevant content to a prompt when you request a LLM - synthesis phase

Once RAG is in place, we can test it. There are two types of evaluation - we can evaluate RAG end-to-end or in isolation. Evaluation in isolation means to test independently retrieval and synthesis parts. For both these types of evaluation, generated datasets work pretty well.

It might look like RAG is only for text documents, but RAG is good for querying API, too. Browser capabilities in ChatGPT are one of many forms of RAG.

In addition, here are a few examples of RAG implementations:

• https://python.langchain.com/docs/expression_language/cookbook/retrieval - Python LangChain

• https://gpt-index.readthedocs.io/en/latest/end_to_end_tutorials/low_level/root.html - Python LlamaIndex

• https://github.com/Azure-Samples/semantic-kernel-rag-chat - C#, SemanticKernel

Refer to these articles for further information on the topic:

1. https://explodinggradients.com/evaluating-rag-pipelines-with-ragas-langsmith -Evaluating RAG pipelines with Ragas + Langsmith

2. https://medium.com/@kelvin.lu.au/disadvantages-of-rag-5024692f2c53 - Disadvantages of RAG

3. https://twitter.com/cwolferesearch/status/1695160469588177354?s=52&t=S2R0f61sTIKsg6l5CK1F8Q

4. https://medium.com/@crskilpatrick807/diving-deep-with-rag-when-ai-becomes-the-ultimate-search-assistant-e55b06dadea3 - Diving Deep with RAG: When AI Becomes the Ultimate Search Assistant

5. https://docs.google.com/presentation/d/1v7T6ejrSo87ndGeGC7tt6zeq-cftu03WWw7WL8Jskug/edit#slide=id.p - Evaluating and Optimizing your RAG App