Your AI app might be silently burning money - and you won't notice until it's too late
Think of your LLM as a sharp analyst who does excellent work with focused material. The naive approach buries that analyst under an entire filing cabinet before every meeting - even when the client only asked about one page. Every single time.
That's what happens when teams first integrate a large language model into a document-heavy product. It's called the naive approach. It works in demos, but breaks in production.
.svg?table=block&id=332a1c4a-c8fb-806b-a5ba-f85d18921c41&cache=v2)
Three ways the naive approach will hurt your product
1. Token costs scale with document size, not with the value delivered
LLM providers charge per token - for every token in the input, not just the relevant ones. If a user uploads a 400-page manual and asks about the warranty period, you're paying to process all 400 pages to retrieve one sentence.
Multiply that across thousands of daily requests, and your cost-per-active-user climbs fast - not because you added features, but simply because users uploaded larger files.
The danger isn't the invoice you see today - it's the unit economics trajectory. A product that's profitable at 500 users can become loss-making at 5,000 if cost-per-request isn't architecture-level controlled.
2. Context window limits cause real production crashes
Every LLM has a hard cap on how many tokens it can process in one call. A 200K-token window sounds large until your user uploads an actual book, a multi-year audit trail, or a full product documentation set.
When the input exceeds the limit, the app doesn't degrade gracefully. It throws an error your users see directly. Critically, these crashes are nearly impossible to reproduce in development, because developers test with short documents while real users have real-world content.
3. "Lost in the Middle" - LLMs forget what's in the center of long prompts
This is the most underestimated problem on the list. Research from Stanford and UC Berkeley has consistently shown that transformer models pay far more attention to content at the beginning and end of a prompt, systematically ignoring information in the middle.
Feed a model a 200-page contract and ask about a clause on page 94. The model may generate a confident, fluent - and completely wrong - answer.
"Lost in the Middle" is not a bug to be patched in the next model version. It's a fundamental property of how transformers work. In practice, the most reliable fix is architectural: reducing the amount of irrelevant context the model receives.
What is RAG?
Retrieval-Augmented Generation (RAG) is an architectural pattern that connects a retrieval system - a vector database, a hybrid search engine, or both - to a generative language model. The model no longer depends exclusively on knowledge compressed into its training weights. Instead, at inference time, relevant documents are fetched from an external knowledge base and injected into the context window, giving the model factually grounded material to reason from rather than statistical patterns to hallucinate from.
This separation is what makes RAG architecturally significant. The knowledge base and the model evolve independently - you can update, expand, or replace either one without touching the other.
Production RAG systems rarely stop at basic retrieval. Mature pipelines incorporate multi-stage retrievers, re-rankers, query reformulation, and structured evaluation frameworks. Some extend further into knowledge graphs and agentic retrieval loops. The common thread across all of them: make the model's output grounded, auditable, and safe enough to deploy against real business data - where a confident wrong answer isn't just annoying, it carries risk. Widely regarded as an industry standard, LangChain's RAG conceptual guide remains one of the most comprehensive public resources
Here's the analogy that makes it click:
You have a library with 10,000 books. Two approaches to answering a client's question:
- Naive: Hand the expert all 10,000 books. Expensive, slow, and the expert loses focus halfway through.
- RAG: A librarian scans the index, pulls the 4 most relevant pages, and hands only those to the expert. Sharp answer, fraction of the cost.
RAG doesn't make the model smarter. It makes the model's job efficient and accurate by ensuring it only processes what's actually relevant.
What RAG solves
Pain Point | How RAG Addresses It |
High token costs per request | Only 3-10 relevant chunks reach the LLM, not the full document |
Context window overflow crashes | Document size becomes irrelevant to the query pipeline |
"Lost in the Middle" degradation | The model works with a tight, high-signal context |
Hallucinations on domain content | Answers are grounded in retrieved real content, not model memory |
Adding knowledge without retraining | Index new documents without touching the model |
Real-world proof: up to 30x token reduction in production
In one of the AI projects we built, the original architecture passed the full document content into every LLM call. Simple to implement - and it worked until documents got large enough to matter. Once we switched to RAG, the system stopped sending entire documents and started retrieving only the chunks relevant to each query. In the worst case, where a document previously filled the entire context window, that shift reduced token consumption by up to 30x. The exact number varies with document size and query pattern, but the directional improvement was consistent.
The table below reflects that comparison:
Metric | Before | After |
Token consumption per query | Baseline (up to 100%) | ~3.3% of baseline - 30x reduction |
Context window overflow errors | Recurring in production | Eliminated completely |
Maximum supported document size | Capped by model context limit | Effectively unlimited |
Answer quality on large documents | Degraded on long files | Consistent, regardless of size |
Cost per active user | Unsustainable at growth targets | Predictable and margin-positive |
The business takeaway: A 30x reduction in token consumption changes the unit economics of the product - not just the infrastructure bill. Per-user costs become predictable, gross margins stay healthy at scale, and document sizes that were previously impossible to support become a feature.
Where RAG drives the most ROI: 3 industry use cases
1. LegalTech and Compliance
Legal teams work across hundreds of contracts, regulatory filings, and internal policies. Finding a specific clause or cross-referencing obligations across documents takes hours - and mistakes carry material risk.
With RAG: An attorney asks "Does our MSA with Vendor X include an automatic renewal clause?" The system retrieves the relevant contract sections and generates a structured summary with source references. Due diligence that took a day takes minutes.
ROI: Faster contract review, reduced outside counsel dependency, lower risk of missing critical obligations.
2. Enterprise Customer Support
Product documentation, release notes, and support runbooks grow into thousands of articles over time. Agents either spend too long searching or rely on memory - both lead to inconsistent, sometimes wrong answers.
With RAG: When a support ticket arrives, the agent's AI assistant surfaces the top relevant documentation sections in real time, with direct links to source pages. Agents verify before they respond - instead of guessing.
ROI: Lower Average Handle Time (AHT), higher First Contact Resolution (FCR), faster onboarding for new support hires.
3. HR and Internal Knowledge Management
Company policies, benefits documentation, and onboarding materials are scattered across Confluence, SharePoint, Notion, and email threads. Employees ask HR the same questions repeatedly because finding the right document takes longer than just asking someone.
With RAG: An employee asks "How many days of parental leave am I entitled to?" and gets an accurate, sourced answer in seconds. HR handles fewer repetitive queries. New hires self-serve from day one.
ROI: HR team focused on higher-value work, faster onboarding, reduced risk of policy misinterpretation.
How to integrate RAG into your service
Here's how the full pipeline looks in practice - and where the decisions that actually matter live
1. Text extraction
Documents are converted to raw text - digital PDFs with parsing libraries like
pdfplumber or PyMuPDF, scanned documents via OCR tools like Tesseract. OCR quality matters more than most teams expect: poor extraction produces poor embeddings, which produces poor retrieval results downstream.2. Chunking with overlap
The text is split into smaller, overlapping segments called chunks. The overlap is important - without it, a key sentence that falls at a boundary gets separated from its context. A typical starting point is 512 tokens per chunk with ~64 tokens of overlap, tuned from there based on document type.
3. Embeddings
Each chunk is converted into a vector - a numerical representation of its meaning - by an embedding model. Semantically similar chunks produce similar vectors. One thing worth being careful about: always use the same embedding model for indexing and querying. Mixing models will silently degrade retrieval quality.
4. Vector storage
Vectors and their source text are stored in a vector database. A practical and often overlooked production choice: PostgreSQL with the pgvector extension, which handles vector search inside your existing database without adding new infrastructure.
5. Retrieval and generation
When a user asks a question, the question is also embedded and compared against all stored chunk vectors using cosine similarity. The top matching chunks are retrieved and injected into the LLM prompt as context. The model answers based only on that context.
Why PostgreSQL + pgvector over a dedicated vector database? For most B2B applications, Postgres is already in the stack. pgvector adds vector search to your existing database - no new infrastructure, no new operational overhead, and familiar tooling for your entire engineering team. Dedicated vector databases (Pinecone, Weaviate) make sense at very large scale, but are often overkill for products under a few million chunks.
A minimal retrieval query in Django:
How to build RAG the right way - techniques most teams skip
A basic RAG pipeline is straightforward to ship. A reliable one is harder - because the failure modes are subtle.
HyDE - let the LLM write the answer before searching for it
The core problem with embedding a user's question and searching for similar chunks is that questions and answers are semantically dissimilar by nature. A user asks "What is the notice period for contract termination?" - but your document contains "Either party may terminate this agreement with 30 days written notice." Those two sentences don't embed close together, even though one directly answers the other.
HyDE (Hypothetical Document Embeddings) flips the approach. Before doing any vector search, you ask the LLM to generate a hypothetical answer to the question - as if it already knew the content. Then you embed that hypothesis and use it as the search query.
The hypothesis reads like the document does - same vocabulary, same sentence structure, same domain register. The tradeoff: one extra LLM call per query, which adds latency and a small cost. For precision-sensitive applications - legal, compliance, medical - the accuracy improvement is well worth it.
Contextual Retrieval - teach your chunks where they come from
Standard chunking strips context. Anthropic published research on Contextual Retrieval showing that prepending a short document-aware summary to each chunk before embedding reduces retrieval failures by around 49% - and up to 67% when combined with re-ranking.
This runs at indexing time, not query time - so there's no latency impact on the user. The cost is one cheap LLM call per chunk, paid once when the document is ingested.
Reciprocal Rank Fusion - run two searches, combine the results
Vector similarity search is great at finding semantically related content. Keyword search (BM25, a standard term-frequency ranking algorithm) is great at finding exact matches - product names, error codes, legal terms, proper nouns. In most production scenarios, combining both tends to outperform either approach on its own.
Reciprocal Rank Fusion (RRF) is a simple, parameter-free formula for combining ranked lists from multiple retrieval systems:
Where
k = 60 (smoothing constant), r(d) is the rank of chunk d in ranker r.The elegance of RRF is that it requires no tuning - the formula is stable across domains and dataset sizes.
If you're already on a cloud provider, managed alternatives like Azure AI Search, AWS Bedrock Knowledge Bases, or Google Vertex AI Search offer RAG-as-a-service with less setup overhead - at the cost of flexibility and vendor lock-in.
RAG vs. Long-Context Models - and Fine-tuning
GPT-4 supports 128K tokens. Gemini 1.5 Pro supports 1 million. So why not just send everything?
- Cost. Long-context calls are priced accordingly. Sending 500K tokens per query is technically possible - the unit economics make it unsustainable at any meaningful query volume.
- "Lost in the Middle" doesn't go away with more context. Longer windows don't fix attention degradation in the middle - they amplify it.
- Latency. Larger contexts mean slower inference. For conversational interfaces, time-to-first-token directly affects perceived product quality.
- Knowledge freshness. Long-context models still have training cutoffs. Documents your users upload today aren't in the model's weights.
The practical rule of thumb: If your entire knowledge base fits comfortably in a context window and query volume is low, full-context prompting can work. Once you're dealing with libraries of documents, growing knowledge bases, or meaningful query volume - RAG is usually the better-suited architecture.
RAG vs. fine-tuning
RAG | Fine-tuning | |
Best for | Dynamic, growing knowledge bases | Consistent tone, format, or domain behavior |
Adding new knowledge | Index a document in minutes | Requires a new training run |
Source traceability | Built-in - answers cite sources | The model "just knows" - no citation |
Hallucination risk | Low - grounded in retrieved content | Moderate - baked-in knowledge can drift |
Typical use case | Contracts, product docs, HR policies | Output formatting, brand voice, specialized vocabulary |
In practice, the strongest production systems use both: RAG for factual grounding, fine-tuning for consistent output behavior. But if you're choosing one to start with document-heavy use cases - RAG tends to be the faster path to a working system for document-heavy use cases.
Conclusion: RAG is an architectural decision, not a feature
The naive approach - sending full documents as context - is a local optimum that fails at scale. It's expensive, brittle under real-world document sizes, and produces lower-quality outputs than a well-built retrieval pipeline.
RAG tends to be a strong architectural fit for LLM applications where:
- The knowledge base is larger than fits in a single context window
- The knowledge base changes over time (without retraining the model)
- Per-request cost is a meaningful factor in your unit economics
- Answer accuracy and source traceability matter to your users
30× is a starting point. Tuning chunking strategies, embedding models, and retrieval thresholds to your domain pushes it further.
You can start without adding any new infrastructure -
PostgreSQL and pgvector slot into your existing stack.
