CaseCloud: multi-tenant legal SaaS with RAG over 80M+ judgments

Challenge

Law firms run on deadlines, documents and trust. CaseCloud needed three things at once:

1. Hard isolation between firms. A missed WHERE tenant_id = ? is a career-ending bug in legal software.
2. Search that lawyers believe. Semantic search alone returns plausible-sounding paragraphs; lawyers need the actual judgment, paragraph number and court.
3. Drafting that saves hours, not drafting that invents citations.

The corpus is large (80M+ judgments across courts), the templates are messy (6K+ Word files with inconsistent placeholders), and the product had to ship to paying firms while the pipeline matured.

Approach

I split the problem into two systems that share a database but nothing else: the case platform (CRUD, billing, reminders, cause lists) and the research & drafting engine (ingestion, retrieval, generation). The platform is boring on purpose; the engine is where the risk lives, so it got the observability budget.

Multi-tenancy

Schema-per-tenant on PostgreSQL with Row-Level Security as a second fence. Every request resolves the tenant from the session, sets search_path, and runs under a role whose RLS policies make cross-tenant reads impossible even if application code forgets a filter. I wrote the trade-offs up in Designing Multi-Tenant Database Architectures; CaseCloud is the "schema per tenant + RLS belt-and-braces" column of that comparison.

Unified RBAC sits above: firm admin, partner, associate, clerk. Permissions are data, not code paths, so a new firm can be onboarded without a deploy.

Retrieval

Hybrid retrieval, because neither side alone was good enough:

• Dense embeddings in Milvus for meaning ("limitation period for appeal in consumer disputes").
• Sparse (BM25-style) for the exact tokens lawyers care about: section numbers, case names, Latin terms.
• Reciprocal Rank Fusion merges the two ranked lists; a light reranker reorders the top 20.
• Every chunk carries court, year, citation and paragraph metadata, so answers link to a specific paragraph of a specific judgment.

Ingestion is a Playwright pipeline that crawls court portals, parses PDFs, normalises citations and writes chunks with stable ids so re-crawls don't duplicate. The design notes are in Advanced RAG Architecture.

Drafting

An agentic drafting flow grounds Gemini on the retrieved passages plus the firm's own templates. The hard part was placeholder canonicalisation: 6K templates used [CLIENT NAME], <<Client>>, _____ and worse. A one-off pass normalised them into a typed placeholder vocabulary, so the model fills slots instead of free-writing. Jurisdiction (court, state) is a first-class parameter, not a prompt hint.

Architecture

Everything emits OpenTelemetry spans with tenant_id and office_id in baggage, which the observability sidecar turns into structured log metadata rather than high-cardinality labels. When a firm says "search is slow today", I can filter traces to that firm in seconds.

What shipped

• Case, client, proceeding, payment, cause-list and reminder modules under one RBAC model.
• Court-portal ingestion for the first set of courts, with idempotent re-crawls.
• Hybrid search with citations in the UI (click the citation, land on the paragraph).
• Jurisdiction-aware drafting from natural-language instructions.
• Per-tenant usage metering for LLM tokens, so pricing follows cost.

Results

• 80M+ judgments searchable with citations; lawyers stopped asking "where did this come from?"
• 6K+ templates usable by the drafting agent without manual cleanup per draft.
• p95 under 2s for a grounded answer including retrieval, rerank and generation.
• Zero cross-tenant data incidents, which is the only acceptable number.

What I'd do differently

• Start with the evaluation set. I built golden question sets per court after the first firm complained about recall; it should have been week one.
• Put token budgets per tenant in from day one. Retrofitting metering into an already-live pipeline was fiddly.
• Milvus is excellent, but for the long tail of small firms pgvector inside the tenant schema would have simplified operations. I'd prototype that first next time.