RAG Chunk Calculator

Retrieval-Augmented Generation (RAG) is a technique that enhances large language models by retrieving relevant information from external knowledge sources before generating responses. Instead of relying solely on the model's training data, RAG systems first query a vector database or knowledge base to find the most relevant documents or passages, then provide these as context to the language model. The key components of a RAG system include: a document store or vector database, an embedding model to convert text into vectors, a retrieval mechanism (usually approximate nearest neighbor search), and a language model that uses the retrieved context to generate accurate responses. Chunk size is one of the most important hyperparameters in RAG. Smaller chunks (128-256 tokens) produce more precise retrieval but may miss important context. Larger chunks (512-1024 tokens) provide more context per retrieved passage but may include irrelevant information and cost more to embed. Overlap between chunks helps prevent important information from being split across chunk boundaries. A good rule of thumb is to use 10-20% overlap relative to the chunk size. For example, with a 512-token chunk size, use 50-100 tokens of overlap.