Fordel StudiosLegal practice runs on precedent, and precedent runs on documents. Contracts, case law, regulations, filings, memos — a mid-size law firm maintains millions of pages of documents that represent the collective knowledge of the practice. RAG makes this knowledge accessible through natural language, but legal documents have structural complexity that generic RAG implementations handle poorly.
Why Generic RAG Fails for Legal
Standard RAG implementations chunk documents into fixed-size text blocks, embed them, and retrieve the most semantically similar chunks to a query. This works for articles and general documents. It fails for legal documents because legal meaning is structural, not just textual.
A contract clause has meaning because of its position in the document hierarchy (which section, which subsection, which defined terms govern it), its relationship to other clauses (indemnification clauses modify liability clauses), and its jurisdictional context (the same language means different things under New York law versus California law). Naive chunking destroys this structural context.
The Legal RAG Stack
Building Legal-Grade RAG
Parse documents preserving hierarchy: sections, subsections, clauses, defined terms, cross-references. OCR for scanned documents with layout analysis to distinguish body text, footnotes, exhibits, and signature blocks.
Chunk by legal unit (clause, section) rather than token count. Include parent section context and relevant defined terms in each chunk. A clause about "Indemnification" should carry the definitions of key terms referenced within it.
Combine vector similarity search with keyword matching. Legal queries often include specific statutory citations, case names, or defined terms that benefit from exact match retrieval alongside semantic search.
Every RAG response must cite the specific document, section, and page number that supports each claim. In legal practice, an uncited assertion is worthless. Build citation extraction into your response generation pipeline.
Legal answers are jurisdiction-dependent. The retrieval layer must filter or prioritize documents from the relevant jurisdiction. A question about employment law in Texas should not return California precedent without flagging the jurisdictional difference.
| Capability | Generic RAG | Legal-Grade RAG |
|---|---|---|
| Document chunking | Fixed token windows | Section/clause-aware hierarchical |
| Retrieval | Vector similarity only | Hybrid: vector + keyword + citation |
| Context window | Flat text chunks | Chunks with parent context + defined terms |
| Output format | Narrative response | Response with inline citations |
| Quality assurance | General coherence | Citation verification + jurisdictional accuracy |
Production Deployments
Harvey AI, backed by significant venture funding and a partnership with Allen & Overy, has become the most prominent legal RAG platform. CoCounsel (now part of Thomson Reuters via the Casetext acquisition) provides AI-assisted research within the Westlaw ecosystem. Spellbook focuses on contract review and drafting. Each has made different architectural choices about how deeply to integrate with existing legal workflows.
The firms seeing the best results are not using these tools to replace lawyers. They are using them to accelerate the research phase — turning a 4-hour research task into a 1-hour review task. The lawyer still evaluates the AI output, verifies the citations, and applies professional judgment. The AI handles the retrieval and synthesis that previously consumed the bulk of research time.
- Hallucinated citations — LLMs can generate plausible but fictional case names and statutory references
- Confidentiality — client documents in a shared RAG corpus require access control at the document and matter level
- Stale retrieval — legal databases update constantly; the RAG corpus must reflect current law
- Over-reliance — junior associates may accept AI research without adequate verification
- Unauthorized practice — AI-generated legal analysis raises UPL concerns in client-facing contexts
“RAG does not make AI a lawyer. It makes AI a very fast, very thorough research assistant that still needs a lawyer to evaluate its work and apply professional judgment.”