The right question needs to find the right source

Procedures in one folder, guidance in messages and important answers that live in a few people’s heads: this scenario creates an opportunity to improve access to knowledge. A good starting point is to pick one team’s frequently asked questions and identify where the approved answers are.

RAG, short for retrieval-augmented generation, combines information retrieval with a language model. In the flow Microsoft describes, the application retrieves relevant passages and sends them to the model as context for composing the answer. This makes it possible to work with content specific to the organization.

Organize a small, reliable collection

In a hypothetical internal support scenario, start with the procedures for system access. For each document, record the owner, the review date and the authorized audience. Archive superseded versions so they don’t compete with current guidance. Ask the team to log the questions the documents don’t answer yet.

Technical preparation matters too. Long documents can be split into chunks, tagged with metadata and added to a search index. Microsoft’s documentation treats chunking, enrichment, retrieval and evaluation as connected decisions. The ideal chunk size should be evaluated against the project’s own content and questions.

Permissions are part of the answer

Search has to respect who can access each piece of information. Microsoft’s guidance is to restrict retrieval to content the user or agent is authorized to see. An assistant should not widen access simply because it brings together documents from several departments.

In the pilot, propose a clear usage rule: the answer must point to the material it consulted and say when the knowledge base does not support a conclusion. The team needs to be able to open the reference, see whether it is still current and route the question to its owner. A useful citation should help verify the answer.

Test search and answers separately

When a result is wrong, first check whether the passage it needed showed up in the search results. Then check how that passage was interpreted. This separation helps distinguish content, retrieval and generation failures. Stage-by-stage evaluation is part of the development approach recommended in the Azure Architecture Center.

  • A question clearly answered by a current procedure.
  • A question worded differently from the document.
  • A question the knowledge base cannot answer yet.
  • A query from someone without access to the required document.
  • An updated procedure: the answer should reflect the new guidance.

Turn open questions into curation work

Our operational suggestion is to schedule a regular review of unanswered questions and of references flagged as unhelpful. Each case should have a next step: fixing a document, adjusting the search, improving the assistant’s instructions or handing the question to a person.

Choose metrics close to day-to-day work: answers accepted after review, questions escalated and time to find valid guidance. Look at these results alongside the effort of maintaining the knowledge base. A well-defined pilot shows both where the assistant helps and which knowledge gaps the company needs to close.

Sources and references

The proposed applications and hypothetical examples are Rizumu’s editorial interpretations.

From this read to your business.

Let’s find where this topic meets a challenge in your company.

Talk to Rizumu