RAG on an existing .NET estate: three decisions that change everything

Most RAG tutorials start from an empty notebook and a fresh database: a brand new Postgres instance, clean documents already chunked, no access constraints to respect. A client running a Microsoft IT estate starts from somewhere else entirely: a fifteen year old SQL Server, an application in production, permissions already in place, and a security audit that does not forgive shortcuts. In that context, the real question is not which large language model to pick. It is where and how to wire retrieval in without breaking what already holds up.

Three decisions change everything when integrating a RAG pipeline into a .NET / SQL Server context.

Vector storage does not need to live outside the existing estate

The most common reflex is to provision a dedicated vector database, usually outside the existing application boundary. That choice duplicates a problem already solved elsewhere in the estate: who is allowed to see which document. A vector index directly inside SQL Server, or a service like Azure AI Search sitting next to the existing database, lets you reuse the access rules already in place instead of recreating a diverging copy of them. The first question is not the performance of the similarity engine, it is where the source of truth for permissions already lives.

Chunking follows business structure, not a token window

Most RAG guides split documents into fixed size chunks, sized to whatever token window the model accepts. That generic chunking ignores the structure of the document. A contract clause cut in half, a report section truncated mid table: the model receives an incomplete fragment and answers off target, even with the best LLM on the market. Chunking should follow the business structure that is already known: the clauses of a contract, the sections of a report, the paragraphs of a procedure. This is often the least visible part of the work, and the one that most determines answer quality.

Orchestration stays in the .NET service layer

The third reflex to avoid: pulling the retrieval logic into a separate Python microservice, bolted on next to the existing application. That architecture looks technically neutral, but it quietly recreates a second authorization system, with its own rules, usually less audited than the ones already governing the main application. With Semantic Kernel, orchestration stays in the .NET service layer that is already in place: the same access checks, the same logging, the same deployment cycle as the rest of the application. RAG becomes one more capability inside a system that is already governed, not a parallel project with its own rules.

What I would not do again

Treat RAG as a project on its own: its own database, its own access rules, decoupled from the rest of the IT estate. That setup works well in a demo. It becomes a security and maintenance problem the moment it touches production, right when nobody remembers exactly who has access to what anymore.

RAG does not require rebuilding the existing estate. It requires respecting it: reuse the permissions already in place, respect the business structure of the documents, keep orchestration in the layer that is already audited. It is an architecture question before it is a model question.