A knowledge base can contain the right answer and still produce the wrong advice. The current policy may be buried beneath an older version. A search result may contain a rule but omit its exception. A familiar precedent may belong to another client. The model can then turn a retrieval mistake into a convincing recommendation.

This is the challenge that makes knowledge architecture central to useful AI. At TalkCounsel, our work on AI-enabled legal services puts that challenge in practical terms: how should a system find the information a question actually needs, recognize authoritative, up-to-date material, and respond quickly without importing irrelevant knowledge?

The following blueprint sets out the standard I believe a serious knowledge base should meet. It combines that operating perspective with published research and explicit engineering recommendations. The aim is to help founders and technical teams build systems whose answers can be inspected, tested, and improved.

A credible system should explain why a particular source was eligible, why it was selected, which version informed the answer, and what remains unknown. That is a more useful definition of quality than the number of documents uploaded.

Start with the decisions the system must support

The first design question is what someone needs to accomplish. “Search our documents” is too broad to provide a meaningful test of success. “Identify the controlling termination clause and any amendment affecting it” gives the system a defined job.

For a legal services business, tasks might include locating an approved negotiation position, checking which documents govern a matter, identifying missing facts before drafting, or explaining a process using current guidance. Each task requires a different mix of evidence. A drafting request needs approved language and instructions about when it applies. A question about a signed agreement needs the executed document and relevant amendments. A question about current law may require fresh primary-source research.

Write an answer specification for each recurring task. Define the required sources, facts the user must supply, acceptable output, and circumstances that require clarification or professional review. Decide who can approve the result. This specification becomes the basis for both retrieval and evaluation.

Retrieval-augmented generation, or RAG, combines external information retrieval with text generation. The foundational work by Lewis and colleagues demonstrated this approach on knowledge-intensive language tasks. In a business system, the surrounding design must also decide which information is eligible, which evidence is sufficient, and what the assistant is authorized to do.

Give every source a role and an owner

A knowledge base should preserve the distinction between source material and organizational judgment. An executed agreement records a transaction. An internal playbook expresses a preferred approach. A statute is a source of law. A previous negotiation illustrates what happened in one matter. These materials cannot safely be treated as interchangeable answers to the same question.

I would organize the system around source roles, then apply the appropriate hierarchy for the task. To explain a client’s contractual obligations, the executed agreement and valid amendments matter differently from a template. To explain applicable law, an internal note must not silently substitute for the controlling authority. Conflicts require contextual analysis; “newest document wins” is not a sound universal rule.

The minimum useful record extends beyond a title and a text field:

These fields serve different purposes. A verification date records a check; it does not necessarily mark when a rule became effective. A recent upload may contain old information. Unknown metadata should remain unknown and trigger review where material, rather than being confidently inferred.

Approval also needs a real owner. Domain experts should approve substance, engineering should maintain reliable ingestion and retrieval, and an identified product owner should accept the system’s performance for a defined use case. Without ownership, a knowledge base can quietly become an archive with no one responsible.

Preserve meaning when preparing documents for search

Before a model can retrieve a document, the system usually needs to extract and organize its contents. This stage deserves attention because corrupted input can look perfectly searchable. A scanned page may lose a negative word. A table may separate a fee from its condition. An amendment may be indexed without its relationship to the agreement it changes.

Keep the original file and test extraction on representative documents. Preserve headings, numbering, tables and page anchors. Flag uncertain optical character recognition for review where the wording is consequential. Remove repeated navigation and page furniture carefully, while retaining meaningful headers and definitions.

Then choose retrieval units that preserve the structure of the subject. A chunk is a portion of a document prepared for search. Its boundaries should respect the material: a clause, a policy section, a complete table, or a decision record. The goal is to make each unit findable without stripping away the information needed to interpret it.

Figure 1. An illustrative contract example shows why an isolated clause may be insufficient. The exact provision remains linked to definitions, exceptions, amendments, and its source record.

A useful pattern is to retrieve a precise passage and then expand to its necessary parent section or linked definitions. Retain the distinction between original text and any machine-generated summary. A summary can help discovery, but consequential claims should remain traceable to the underlying source.

There is no universal best chunk size. Test several approaches against your actual questions. A configuration that works for support articles may be poorly suited to transaction documents with cross-references. Treat chunk size and overlap as measured design choices, and change them only when evaluation shows a benefit.

Build retrieval around intent and eligibility

A strong retrieval process starts by understanding the task and establishing access. A request to shorten a paragraph may need no knowledge-base search. A question about a specific policy needs targeted retrieval. A current-law question may need an approved external research route. A comparison across documents may require several searches and an explicit synthesis step.

That routing should be tested. An incorrect decision to skip retrieval can be as harmful as an unnecessary search. If a missing jurisdiction or matter identity would materially change the answer, clarification belongs before substantive analysis.

Within the authorized collection, hybrid search is a useful baseline to evaluate. It combines keyword search with vector search, which finds conceptual similarity. Microsoft documents this combination and the use of reciprocal rank fusion to merge ranked results. Exact identifiers matter in professional work: a clause number, case citation, or product code may be more discriminating than general semantic similarity.

Figure 2. Recommended retrieval architecture. Access and source eligibility constrain candidate selection. A bounded second search handles missing evidence; unresolved gaps lead to clarification or escalation.

After candidate retrieval, a reranker can assess relevance more closely. Deduplication should prevent several near-identical passages from occupying the evidence budget. Context assembly should retain necessary exceptions and conflicting sources, rather than selecting only passages that support the most convenient answer.

I would optimize for the smallest sufficient evidence set: enough to answer the material parts of the question, with room for qualifications. This is different from simply minimizing tokens. If a conclusion depends on three linked provisions, retrieving only one is false economy.

Long context windows do not remove the need to test this choice. Liu and colleagues found that the models they studied often used information less effectively when it appeared in the middle of long inputs. That historical finding is a reason to test evidence placement and distractors with your own model, not a claim that every current model has the same limitation.

What published retrieval data actually tells us

Anthropic’s September 2024 experiments reported the following average top-20 retrieval failure rates for the selected embedding configuration across its tested domains. The metric was 1 minus recall@20.

Figure 3. External research results, not TalkCounsel measurements. Lower is better. Source: Anthropic, Introducing Contextual Retrieval, 19 September 2024.

The reported rates were 5.7% for embeddings alone, 3.7% for contextual embeddings, 2.9% for contextual embeddings plus contextual BM25, and 1.9% when reranking was added. Contextual retrieval supplies a passage with explanatory context before indexing. The change from 5.7% to 1.9% represents 3.8 percentage points, or approximately a 67% relative reduction in retrieval failure.

These are retrieval results from a particular experiment. They do not establish a 67% improvement in answer accuracy, nor predict performance on a legal corpus. Their practical value is to identify configurations worth testing locally.

Build a controlled comparison using the same questions and corpus snapshot. Compare keyword search, vector search, hybrid search, and selected enhancements. Measure final answers as well as retrieved passages. A more elaborate pipeline earns its place only if it improves the outcomes you care about at an acceptable cost and response time.

Follow one question through the system

Consider an illustrative request: “Can we accept the customer’s proposed liability cap?” This is a design example, not a description of a particular TalkCounsel client matter.

The system first establishes the matter, the relevant draft, and the user’s authority to access it. It retrieves the proposed clause, related exceptions, defined terms, and the applicable approved playbook. It checks whether the playbook addresses this transaction type. A precedent from another client is not eligible merely because its wording is similar.

The useful output separates what the draft says from the preferred position and the unresolved facts. It might identify an exception that changes the cap’s practical effect, explain how that differs from the approved position, and ask for missing information needed to assess the exposure. It should point to the precise provisions supporting those observations.

A polished answer with a link at the bottom is insufficient. The cited passage must support the claim attached to it, and the analysis must account for material qualifications elsewhere. A system can cite a real document while still misrepresenting it.

This matters across chat and task execution. A conversation may establish the user’s intention, while a later drafting task acts on it. Both should use a shared retrieval service and consistent source policies. Otherwise, the chat can rely on one version while the task retrieves another.

Carry a compact evidence record between the two: source identifiers, versions, anchors, and relevant assumptions. Recheck authorization and freshness before consequential execution. If the governing material has changed, surface the change and revise the task’s basis. A prior chat response should never become authority merely because it is convenient to reuse.

Make updates and access controls part of correctness

A trustworthy knowledge base needs a publication lifecycle. New material should move through extraction, validation, domain approval, and indexing before becoming eligible for ordinary answers. An update should replace or supersede the right version, rebuild affected retrieval units, and invalidate dependent caches. Publishing should be coordinated so a query does not combine incompatible versions during the transition.

Historical material still has value. A question about what applied six months ago requires an explicit historical route. A current question should not accidentally receive a retired rule. Effective dates, recorded dates, and supersession relationships allow the system to distinguish these tasks.

Speed requires similar discipline. Measure where time is spent: routing, search, reranking, generation, and validation. Cache stable work where appropriate, but include permission scope and knowledge version in cache eligibility. Permission revocation must take effect promptly even when a cached response or passage exists. A cache that serves yesterday’s access rights is a correctness failure.

Access control belongs in the application and retrieval layer. Microsoft describes a document-filtering pattern based on user or group identifiers, while noting that the filter itself is not authentication. The service must authenticate callers and supply trusted authorization constraints. Do not let the model invent the client’s identity or decide which confidential records it may inspect.

Apply the same principle to citations, exports, logs, and derived summaries. Restrict or redact operational traces appropriately. A passage should not become accessible through a debug screen after being correctly excluded from the answer.

Retrieved material should remain evidence rather than a source of executable instructions. OWASP identifies indirect prompt injection through external files and websites and explains that RAG does not eliminate this risk. Separate retrieved content from trusted instructions, constrain tool permissions, and test attempts to redirect the system through a document. Layered controls reduce exposure; a sentence telling the model to ignore malicious content is not a complete defense.

Measure the system at the level where it can fail

“The answer looks good” is not an evaluation method. A useful scorecard distinguishes finding evidence, using it correctly, maintaining boundaries, and delivering a usable result. Ragas research makes a related distinction between retrieval quality and the quality and faithfulness of generated answers.

The following is a recommended operating scorecard, not a report of TalkCounsel results:

For an initial test set, I would start with 200 expert-reviewed questions: 70 routine tasks, 35 jurisdiction or scope tests, 30 version conflicts, 25 missing-evidence cases, 20 permission tests, and 20 adversarial-document cases. These are proposed test allocations, not an industry benchmark or a statistically sufficient certification of safety. Expand them as the product’s scope and failure history grow.

Record the expected evidence and acceptable reasoning, not just a model-written reference answer. Separate development questions from a held-out set. Keep paraphrases of the same case from leaking across those sets. Run comparisons on a fixed corpus snapshot and inspect results by task category, because an aggregate score can conceal a serious weakness.

Model-assisted scoring can help triage large volumes, but professional review should calibrate it and resolve consequential disagreements. Report both refusal behavior and correctness so the system cannot appear safer simply by declining everything. Zero observed access violations is a necessary release expectation, but a finite test suite cannot prove that violations are impossible.

Build a feedback loop that improves knowledge

Launch with a bounded use case and a known source collection. Establish a simple baseline before adding query rewriting, reranking, or complex agent workflows. Introduce each enhancement against the same evaluation set, and preserve the ability to roll back a model, index, or prompt change.

Every reviewed failure should enter a diagnostic category. Was the source missing? Was it extracted incorrectly? Was the relevant passage not retrieved? Did a stale version outrank the current one? Did the model ignore an exception? The remedy depends on the failure. Rewriting the prompt will not repair a missing amendment.

Corrections should pass through review before becoming shared knowledge. One user’s preference is not automatically a company policy, and one accepted negotiation exception is not automatically a general rule. Where a correction is approved, update the source, repair its retrieval representation, and add a regression case that checks that the failure does not recur.

Track the economics at the level of completed work. Include model use, retrieval infrastructure, content maintenance, human review, and rework when calculating cost per accepted task. A faster first draft that creates more review work may be a poor trade. A slightly slower answer that reliably surfaces the controlling evidence may be more valuable.

The standard I want TalkCounsel’s knowledge systems to meet is straightforward: relevant evidence, visible reasoning about its application, current sources where currency matters, and a clear path to correction. That standard can guide any organization building AI around its expertise.

Authority is earned when the system performs under difficult questions: an ambiguous request, an outdated policy, an inaccessible document, or an exception hidden several pages away. The real achievement is making the right knowledge available under those conditions, then demonstrating that it was used responsibly.