4 min read ai

RAG vs fine-tuning

Why now?

2023 is definitely a year of LLMs - GPT-4 and later on GPT-4-Turbo, LLaMA and LLaMA 2, Falcon, Mistral, Claude - a lot of different models from different parties.
While GPT-4 is still a state-of-the-art model, there are other models that may suit you well, that you can run on your own hardware (even on the smartphone with some tweaks!)

Understanding Retrieval Augmented Generation (RAG)

Retrieval-augmented generation, or RAG combines the strengths of both retrieval-based and generative approaches, creating a unified framework that incorporates a retriever and a generator. This unique combination allows RAG to retrieve information from a vast set of documents and subsequently generate responses based on the retrieved data. To get started with RAG you will usually incorporate a few mechanisms like an embeddings model (to create vector representation of your data), a vector database to store and search through your embeddings, and, in many cases - reranker, to further improve accuracy of the data you'd like to fetch from your knowledge base for LLM.

Applications of RAG

RAG has found its applications in various domains, enhancing AI capabilities in different contexts:

  • Chatbots and AI Assistants: RAG-powered systems shine in question-answering scenarios, providing context-aware and detailed answers from extensive knowledge bases. These systems enable more informative and engaging interactions with users.
  • Education Tools: RAG can significantly improve educational tools by offering students access to answers, explanations, and additional context based on textbooks and reference materials. This facilitates more effective learning and comprehension.
  • Legal Research and Document Review: Legal professionals can leverage RAG models to streamline document review processes and conduct efficient legal research. RAG assists in summarizing statutes, case law, and other legal documents, saving time and improving accuracy.
  • Medical Diagnosis and Healthcare: In the healthcare domain, RAG models serve as valuable tools for doctors and medical professionals. They provide access to the latest medical literature and clinical guidelines, aiding in accurate diagnosis and treatment recommendations.
  • Language Translation with Context: RAG enhances language translation tasks by considering the context in knowledge bases. This approach results in more accurate translations, accounting for specific terminology and domain knowledge, particularly valuable in technical or specialized fields.

Exploring Fine Tuning

Fine-tuning refers to making small adjustments or modifications to a system or model that has already been trained on a larger dataset. Fine-tuning typically involves taking a pre-trained model and adjusting its parameters or architecture slightly to adapt it to a specific task or dataset.

How Does Fine Tuning Work?

Fine-tuning involves several steps:

  • Pre-trained Models: Start with a pre-trained model. These models are neural networks trained on large datasets, usually for tasks like image recognition (e.g., ImageNet) or natural language understanding (e.g., GPT-3.5, GPT-4, LLaMA, Mistral).
  • Task-Specific Data: Gather a smaller dataset specific to your task. While the pre-trained model has general knowledge, it needs to learn the nuances of your problem. This dataset should be related to the task you want the model to perform.
  • Adjusting Layers: Modify the top layers of the pre-trained model. Freeze the early layers (which capture general features) and modify the later layers to suit your task. For example, in a neural network for image recognition, you might remove the last few layers and add new layers tailored to your specific classes.
  • Training: Train the modified model on your task-specific dataset. Since you're starting with a model that has learned many features from the original dataset, you often need fewer epochs (training iterations) than training a model from scratch.
  • Fine-tuning Parameters: Experiment with hyperparameters during training. This could include learning rates, batch sizes, and regularization techniques. Fine-tuning these parameters is crucial for achieving your task's best performance.

Comparing RAG and Fine Tuning

  • Basic Ideas: RAG combines traditional text generation with a retrieval mechanism. It means the model generates text, but it retrieves relevant information from a set of documents or passages before generating each token. On the other hand, fine-tuning is a training technique where a pre-trained model (like GPT) is further trained on a specific dataset related to a particular task. The model learns task-specific patterns and information from the provided dataset during fine-tuning.
  • Use Cases: RAG is handy for tasks that require the model to incorporate specific, up-to-date, or domain-specific knowledge from large datasets, like recent news articles or medical research papers. Fine-tuning is commonly used when applying a pre-trained model to a specific task or domain is needed. It's efficient because the model doesn't start learning from scratch but refines its existing knowledge for the given job.
  • Benefits: By incorporating information from external sources, RAG can generate more contextually relevant and accurate responses. It's mighty when the knowledge required for generating text is only partially present in the model's pretraining data. Fine-tuning allows the model to be customized for specific applications without training a massive model from the ground up. It leverages the general language understanding capabilities of the pre-trained model while tailoring it to perform well on a specialized task.
  • External Knowledge: RAG is designed to augment LLM capabilities by retrieving relevant information from knowledge sources before generating a response. It's ideal for applications that query databases, documents, or other structured/unstructured data repositories. RAG excels at leveraging external sources to enhance responses. While it's possible to fine-tune an LLM to learn external knowledge, it may not be more practical for frequently changing data sources. Usually, training and evaluating models can be difficult and time-consuming.
  • Model Customization: RAG primarily focuses on information retrieval and may not inherently adapt its linguistic style or domain-specificity based on the retrieved information. It excels at incorporating external knowledge but may not fully customize the model's behavior or writing style. Fine-tuning allows you to adapt an LLM's behavior, writing style, or domain-specific knowledge to specific nuances, tones, or terminologies. It offers deep alignment with styles or expertise areas.
  • Costs: fine-tuning is expensive compared to RAG - you have to prepare the dataset in the proper format and clear it from unnecessary noise, ideally having thousands of examples to provide. RAG on the other hand, to work really well, requires only embeddings creation and in most cases - reranker on the data gathered from the vector database. This can be a fully automated task that keeps your knowledge database up-to-date.

I hope this sheds some light on when to use RAG and when to fine-tune - many people think one is superior compared to the other, but the reality is that every use case is different. The same goes for model usage - you don't necessarily need to use GPT-4 for every single use case. But that's the story for another post!

Mateusz Kozak

Mateusz Kozak

Warsaw, Poland