Modern RAG systems
Chunking strategies, embedding models, vector databases & reranking — best practices
Full report · English translation
Between 2024 and 2025, Retrieval-Augmented Generation (RAG) underwent a fundamental transformation: simple semantic search mechanisms evolved into complex, multi-stage systems incorporating hybrid search, intelligent reranking, and agentic approaches. A modern RAG stack is no longer just a choice of vector database and embedding model. It requires careful optimization of data preprocessing (advanced chunking), late interaction and reranking mechanisms, and knowledge graphs (GraphRAG) for complex reasoning.
The key trends shaping the state of the art in RAG are:
- Contextualizing chunks (Contextual Retrieval): Moving from straightforward text splitting to methods that preserve global context within individual chunks, such as Anthropic's Contextual Retrieval and Jina AI's Late Chunking.
- Hybrid search as the standard: Combining dense vector retrieval with lexical search (sparse retrieval/BM25), using rank-fusion algorithms such as Reciprocal Rank Fusion, has become a standard requirement for production systems.
- Specialized embedding models: Models supporting Matryoshka Representation Learning (MRL) optimize storage, while long-context models, including those from Voyage AI and Cohere, support contexts of up to 32k–128k tokens.
- Agentic RAG and GraphRAG: Knowledge graphs support tasks requiring multi-hop reasoning, where standard vector search is ineffective.
This report provides a comprehensive technical analysis of each component of a modern RAG pipeline, focusing on best practices supported by MTEB and ruMTEB benchmarks.
1. Chunking strategies: from mechanical to semantic
Splitting text into chunks is the first critical stage of RAG. A poorly chosen chunking strategy can lose context, sever semantic relationships, and ultimately cause LLM hallucinations. In 2025, the focus shifted from fixed splitting to more intelligent methods.
1.1. Traditional methods and their limitations
Basic approaches such as fixed-size chunking (splitting by a fixed number of tokens) and recursive character splitting (recursively splitting on delimiters) remain popular because they are simple and computationally inexpensive.
- Recommended settings: For basic systems, a chunk size of 512 tokens with a 50–100-token overlap is considered a “gold standard” [1]. This balances context preservation and retrieval precision.
- Limitations: Fixed boundaries often break up logical units, separating a question from its answer or splitting an entity's description, which reduces retrieval recall [2].
1.2. Semantic chunking
Semantic chunking uses embeddings to determine chunk boundaries. The process involves:
- Splitting the document into sentences.
- Generating embeddings for each sentence.
- Analyzing cosine similarity between adjacent sentences.
- Starting a new chunk where similarity drops sharply, indicating a change of topic [2, 3].
This method ensures that each chunk contains a semantically coherent idea. However, it is significantly more computationally expensive because it requires multiple embedding-model calls.
1.3. Anthropic's Contextual Retrieval
In September 2024, Anthropic introduced Contextual Retrieval, addressing the loss of global context when documents are split into chunks.
The problem: A chunk saying “Revenue increased by 3% compared with the previous quarter” is useless without knowing which company and reporting period it refers to.
The solution: An LLM generates a short contextual description of the entire document or section. This description is prepended to every chunk before indexing.
Results: Combined with reranking, this method reduces retrieval failures by 67% [4, 5, 6].
Implementation:
# Conceptual example context = llm.generate_summary(full_document) chunk_with_context = f"{context}\n\n{original_chunk}" embedding = embed_model.encode(chunk_with_context)
1.4. Jina AI's Late Chunking
Jina AI proposed an alternative called Late Chunking, which uses long-context models, for example with an 8,192-token context window.
- Mechanism: Instead of splitting text before feeding it into the model, the entire document, or a large portion of it, is passed through the transformer. The model generates vector representations for every token while accounting for their global relationships through attention. Only then is the token sequence split into chunks and pooled [7, 8, 9].
- Advantage: Each chunk “knows” the context of the whole document because attention has processed the full text. This resolves broken anaphoric references, such as “he” referring to a name in the previous chunk.
- Effectiveness: Tests show that Late Chunking outperforms traditional methods, achieving higher cosine similarity for relevant passages. However, processing long contexts requires more memory [9, 10].
1.5. Agentic chunking
The most advanced and expensive approach uses an LLM to analyze a document and determine its logical boundaries, headings, and hierarchy independently. It is justified for complex, unstructured documents where high retrieval accuracy is important [11].
2. Embedding models: market leaders and benchmarks
The embedding model sets the quality ceiling for the entire RAG system. In 2024–2025, both proprietary APIs and open-weight models competed for the lead.
2.1. Global leaders: the MTEB leaderboard
MTEB (Massive Text Embedding Benchmark) remains the main comparison tool. As of late 2024 and early 2025, the following models stand out:
OpenAI text-embedding-3-large:
- Specifications: 3,072 dimensions, with dimensionality reduction available; an 8,191-token context window.
- Advantages: High performance and native Matryoshka Representation Learning (MRL), allowing vectors to be shortened without a significant loss of quality [12, 13].
- MTEB score: Approximately 64.6, although some newer models perform better [14].
Voyage AI (voyage-3.5 / voyage-3-large):
Google Gemini Embedding-001/004:
NV-Embed-v2 (NVIDIA):
Alibaba Qwen3-Embedding:
2.2. Models for Russian: ruMTEB and Russian RAG
For Russian-language data, English-centric models, even multilingual ones, are not always optimal. In 2024–2025, the specialized ruMTEB benchmark and new models emerged.
ru-en-RoSBERTa:
- Description: A specialized model introduced in August 2024. Based on ruRoBERTa and further trained on approximately four million pairs of Russian and English texts.
- Performance: Matches or exceeds state-of-the-art multilingual models such as mE5 on ruMTEB tasks, including semantic textual similarity, classification, and retrieval [17, 18, 19, 20, 21].
- Usage: Prefixes such as
search_query:are recommended [20].
DeepPavlov RuBERT:
Multilingual heavyweights:
- BGE-M3 (BAAI): Combines multi-vector (ColBERT-like) and sparse retrieval in a single model. It performs very well on Russian thanks to its extensive multilingual training corpus [2, 12].
- E5-mistral-7b-instruct: A powerful LLM-based model that delivers high quality but requires substantial GPU resources [24, 25].
2.3. Best practices for choosing a model
- For Russian: If resources permit, Qwen3-Embedding-8B or BGE-M3 are the best general-purpose choices. For lighter systems, consider ru-en-RoSBERTa.
- For code: Voyage-code-2 or Qwen3-Embedding, a leader in code benchmarks [12].
- For cost savings: Matryoshka models, including OpenAI, Voyage, and ModernBERT, can reduce vector dimensions, for example from 1,536 to 256, with minimal quality loss (less than 2%). This significantly reduces memory requirements in the vector database [1, 12].
3. Vector databases: the memory infrastructure
In 2025, the vector database market is consolidating around solutions that offer hybrid search, metadata management, and serverless architectures alongside vector storage.
3.1. Comparing the market leaders
| Database | Type | License / model | Key advantages | Best fit | Sources |
|---|---|---|---|---|---|
| Pinecone | Managed | SaaS / Serverless | Simplicity, pay-per-use serverless architecture, and high reliability. | Enterprise RAG and large projects without dedicated DevOps resources. | [26, 27, 28] |
| Milvus / Zilliz | Open source | Apache 2.0 / SaaS | Scales to billions of vectors; high GPU performance. | High-load systems working with very large datasets. | [26, 27] |
| Weaviate | Open source | BSD-3 / SaaS | Modularity, built-in hybrid search, and a GraphQL API. | Complex data schemas and hybrid search out of the box. | [26, 27] |
| Qdrant | Open source | Apache 2.0 / SaaS | Written in Rust for speed and efficiency; powerful metadata filtering. | Systems requiring complex filtering and on-premises deployment. | [26, 27, 28] |
| Chroma | Open source | Apache 2.0 | Simplicity, a focus on developer experience, and local operation. | Prototyping, local agents, and smaller projects. | [27, 28] |
| pgvector | Extension | Open source | PostgreSQL integration: one database for data and vectors. | Projects already using Postgres that do not need billions of vectors. | [26, 29] |
3.2. Key trends in vector databases
- Hybrid search: Most databases now natively support both dense vectors and sparse/BM25 representations, enabling simultaneous keyword and semantic search [29, 30].
- Metadata filtering: Efficient pre-filtering, including filtered HNSW, has become standard. Qdrant stands out for its efficient filtering engine [27].
- Quantization: Binary and scalar quantization reduce memory consumption by up to 30–50 times while preserving accuracy [12].
4. Retrieval and reranking: maximizing accuracy
Simple dense vector retrieval often suffers from semantic drift and struggles with exact matches such as product codes, names, and specialized terms. Modern RAG uses a multi-stage retrieval pipeline.
4.1. Hybrid search
Hybrid search combines results from vector retrieval and keyword search (BM25/SPLADE).
- BM25 (Best Match 25): A term-frequency-based algorithm related to TF-IDF. It is ideal for exact matches, rare words, and abbreviations that embedding models may blur [30, 31, 32].
- Reciprocal Rank Fusion (RRF): The standard algorithm for combining results from different retrieval sources, such as vector and lexical search.
- Use case: Hybrid search is essential when user queries contain specific identifiers, product names, or technical terms [29, 30].
4.2. Reranking
Reranking reorders the top-K documents returned by initial retrieval using a more accurate but slower model.
- Bi-encoders vs. cross-encoders:
- Bi-encoders (retrieval): Quickly compute cosine similarity between two vectors. They process the query and document independently and are less accurate at capturing subtle relationships.
- Cross-encoders (reranking): Take a query–document pair and process it jointly through a neural network with full attention. This allows the model to “see” how every word in the query interacts with every word in the document.
- Performance: Cross-encoders are much more accurate but slower. They are therefore applied only to a small candidate set, such as the top 50 results [34, 35, 36, 37].
- Popular reranking models:
4.3. Query transformation
- HyDE (Hypothetical Document Embeddings): An LLM generates a hypothetical answer to the user's question, and its vector is used for retrieval. This bridges the semantic gap between a short question and a long document [38].
- Query decomposition: Splits a complex question into subquestions for independent retrieval [39].
- Self-querying: Extracts metadata from the query, such as “reports from 2023,” to use as database filters rather than relying exclusively on semantic search [40, 41, 42].
5. Advanced architectures: GraphRAG and Agentic RAG
When standard RAG is insufficient for complex analytical tasks, more advanced architectures come into play.
5.1. GraphRAG: graph-based RAG
GraphRAG uses knowledge graphs instead of, or alongside, vector indexes.
- Principle: An LLM processes documents to extract entities and their relationships, creating a graph that can be traversed.
- Advantage: It addresses multi-hop reasoning. For example, given “How are Company A's counterparties connected to the 2020 scandal?”, vector search may retrieve documents about Company A and the scandal without finding an explicit connection. GraphRAG traverses graph edges to uncover non-obvious paths [43, 44, 45].
- Drawbacks: High graph-construction costs and latency [45, 46].
- Microsoft GraphRAG: Microsoft popularized this approach by showing that it produces more comprehensive answers to global questions about a corpus, such as “What are the main themes in these documents?” [46].
5.2. Agentic RAG
Agentic RAG turns passive retrieval into an active problem-solving process.
- Workflow: An agent has access to tools such as search, a calculator, and a vector database. After receiving a query, it can:
- Formulate a search query.
- Evaluate the results.
- Reformulate the query or search another source if the information is insufficient.
- Synthesize an answer.
- Applications: Ideal for complex research tasks requiring iterative search [47, 48, 49].
6. RAG evaluation
RAG cannot be optimized without metrics. In 2025, frameworks using LLM-as-a-Judge are popular.
6.1. Frameworks
- RAGAS (RAG Assessment): The most popular open framework. Its metrics include:
- TruLens: Introduces the “RAG Triad” of groundedness, context relevance, and answer relevance, plus tools for tracking experiments [52, 54].
- DeepEval: Focuses on unit testing RAG pipelines and CI/CD integration [50, 52].
7. Best practices summary
The following reference architecture is recommended for a production RAG system in 2025:
- Ingestion:
- Use Contextual Retrieval, adding context to chunks, or Late Chunking to preserve meaning.
- Chunk size: approximately 512 tokens with a 10% overlap.
- Embedding:
- Use state-of-the-art models: OpenAI text-embedding-3-large, Voyage-3.5, or Qwen3-8B.
- For Russian, consider ru-en-RoSBERTa or BGE-M3.
- Enable MRL dimensionality reduction to optimize costs.
- Database:
- Choose Pinecone for a managed solution or Qdrant/Milvus for self-hosting.
- Always use metadata filtering.
- Retrieval:
- Implement hybrid search (vector + BM25), merging results with RRF.
- Use query expansion (HyDE) for complex questions.
- Post-processing:
- Add a reranking stage with a cross-encoder for the top 50 candidates before passing them to the LLM. This delivers the largest quality improvement, up to 15–30% [38].
- Evaluation:
- Integrate RAGAS/TruLens into the development cycle. Do not rely on subjective assessment.
This approach balances accuracy, cost, and performance, making it possible to build systems that handle complex queries and specialized data robustly.
Sources:
- ↩weaviate.io
- ↩firecrawl.dev
- ↩towardsdatascience.com
- ↩anthropic.com
- ↩datacamp.com
- ↩medium.com
- ↩jina.ai
- ↩reddit.com
- ↩milvus.io
- ↩elastic.co
- ↩medium.com
- ↩zenml.io
- ↩agentset.ai
- ↩ailog.fr
- ↩modal.com
- ↩baseten.co
- ↩arxiv.org
- ↩huggingface.co
- ↩aclanthology.org
- ↩huggingface.co
- ↩arxiv.org
- ↩deeppavlov.ai
- ↩arxiv.org
- ↩modal.com
- ↩plainenglish.io
- ↩firecrawl.dev
- ↩latenode.com
- ↩database.tools
- ↩dbi-services.com
- ↩aimind.so
- ↩medium.com
- ↩superlinked.com
- ↩microsoft.com
- ↩medium.com
- ↩eyka.com
- ↩apxml.com
- ↩gopenai.com
- ↩medium.com
- ↩deepset.ai
- ↩dev.to
- ↩medium.com
- ↩gigaspaces.com
- ↩memgraph.com
- ↩datasciencedojo.com
- ↩cognilium.ai
- ↩thenewstack.io
- ↩dev.to
- ↩medium.com
- ↩neo4j.com
- ↩deepeval.com
- ↩deepchecks.com
- ↩anyscale.com
- ↩qdrant.tech
- ↩medium.com





