Attach RAG to your Agent

RAG (Retrieval-Augmented Generation) helps your AI agent find relevant information from documents to give better answers. It searches a knowledge base and uses that context to respond more accurately.

Architecture

The RAG pipeline flow:

Voice Input

STT converts speech to text.

Retrieval

The knowledge base fetches relevant documents based on the transcript.

Augmentation

Retrieved context is injected into the LLM prompt.

Generation

The LLM generates a grounded response using the context.

Voice Output

TTS converts the response to speech.

Managed RAG

Managed RAG lets you upload knowledge bases from the Zero runtime dashboard and connect them to your agent. During a conversation, the agent retrieves relevant information from these knowledge bases to provide more accurate, context-aware responses.

Managed RAG is supported in Python SDK 0.1.5+ and JavaScript SDK 0.1.1+.

Step 1: Upload a Knowledge Base

Upload your documents and create a knowledge base from the Zero runtime dashboard.

After the knowledge base is created, the dashboard provides a unique Knowledge ID.

Step 2: Configure the Knowledge Base

Use the Knowledge ID to attach one or more knowledge bases to your agent’s pipeline.

from zeroruntime import Agent, KnowledgeBase, Pipeline, Room
from zeroruntime.inference import TurnDetector
from zeroruntime.plugins import CartesiaTTS, DeepgramSTT, GoogleLLM, SileroVAD


pipeline = Pipeline(
    stt=DeepgramSTT(model="nova-2"),
    llm=GoogleLLM(model="gemini-2.5-flash"),
    tts=CartesiaTTS(),
    vad=SileroVAD(),
    turn_detector=TurnDetector(),
    knowledge_base=KnowledgeBase(
        knowledge_ids=["kb_id_1", "kb_id_2"],
        top_k=5,
        report_unanswered=True,
    ),
)

import {
  Agent,
  KnowledgeBase,
  Pipeline,
  TurnDetector,
  CartesiaTTS,
  DeepgramSTT,
  GoogleLLM,
  SileroVAD,
} from "zeroruntime";

const pipeline = new Pipeline({
  stt: new DeepgramSTT({ model: "nova-2" }),
  llm: new GoogleLLM({ model: "gemini-2.5-flash" }),
  tts: new CartesiaTTS(),
  vad: new SileroVAD(),
  turnDetector: new TurnDetector(),
  knowledgeBase: new KnowledgeBase({
    knowledgeIds: ["kb_id_1", "kb_id_2"],
    topK: 5,
    reportUnanswered: true,
  }),
});

You can attach multiple knowledge bases to the same agent by providing multiple Knowledge Base IDs.

Configuration

Option	Description
`knowledge_ids` / `knowledgeIds`	The IDs of the knowledge bases you want the agent to search.
`top_k` / `topK`	The maximum number of relevant results retrieved for each query.
`report_unanswered` / `reportUnanswered`	Reports questions that cannot be answered from the knowledge base to the dashboard. You can answer these questions there, and the updated knowledge is picked up during the next re-indexing.

Custom RAG

For full control, Build your own RAG pipeline using any vector database (ChromaDB, Pinecone, etc.) with the user_turn_start hook.This hook fires when the user’s transcript is ready, before the LLM is called, giving you the perfect place to retrieve documents and inject context. 1. Set up the vector store. Create a collection to hold your document embeddings. ChromaDB ships a Python client. The snippet below keeps embeddings in a minimal in-memory index.

self.chroma_client = chromadb.Client()
self.collection = self.chroma_client.create_collection(name="rag_docs")

// A vector store of your choosing; the SDK does not ship one.
this.collection = await vector_store.create_collection('rag_docs');

2. Implement retrieve(). Generate a query embedding and search the vector store for the top matching documents.

async def retrieve(self, query: str, k: int = 2) -> list[str]:
    response = await self.openai_client.embeddings.create(
        input=query, model="text-embedding-ada-002"
    )
    query_embedding = response.data[0].embedding
    results = self.collection.query(
        query_embeddings=[query_embedding], n_results=k
    )
    return results["documents"][0] if results["documents"] else []

async retrieve(query: string, k = 2): Promise<string[]> {
  const response = await this.openai_client.embeddings.create({
    input: query,
    model: 'text-embedding-ada-002',
  });
  const query_embedding = response.data[0].embedding;
  const results = await this.collection.query({
    query_embeddings: [query_embedding],
    n_results: k,
  });
  return results.documents?.[0] ?? [];
}

3. Inject context with the user_turn_start hook. Retrieve documents from the transcript and fold them into the system prompt with session.change_component(instructions=...) before the LLM is invoked. Rebuild from a base string each turn so the retrieved context doesn’t accumulate.

BASE_INSTRUCTIONS = "You are a helpful assistant. Ground your answers in the retrieved context."

@pipeline.on("user_turn_start")
async def on_user_turn_start(transcript: str):
    context_docs = await agent.retrieve(transcript)
    if context_docs:
        context_str = "\n\n".join(
            f"Document {i+1}: {doc}" for i, doc in enumerate(context_docs)
        )
        await agent.session.change_component(
            instructions=(
                f"{BASE_INSTRUCTIONS}\n\nRetrieved Context:\n{context_str}\n\n"
                "Use this context to answer the user's question."
            )
        )

const BASE_INSTRUCTIONS =
  'You are a helpful assistant. Ground your answers in the retrieved context.';

pipeline.on('user_turn_start', async (transcript: string) => {
  const context_docs = await agent.retrieve(transcript);
  if (context_docs.length) {
    const context_str = context_docs
      .map((doc, i) => `Document ${i + 1}: ${doc}`)
      .join('\n\n');
    await agent.session.change_component({
      instructions:
        `${BASE_INSTRUCTIONS}\n\nRetrieved Context:\n${context_str}\n\n` +
        "Use this context to answer the user's question.",
    });
  }
});

The LLM then sees the injected context and uses it to generate a grounded answer.

Best Practices

Start by retrieving k=2-3 documents and adjust based on performance.
Keep document chunk sizes between 300-800 words.
Use persistent storage for your vector database in production.
Cache embeddings for frequently asked questions to reduce latency.
Handle retrieval failures gracefully so the agent can still respond.

Overview

Build

Plugins

Deploy

Attach RAG to your Agent

Architecture

Managed RAG

Step 1: Upload a Knowledge Base

Step 2: Configure the Knowledge Base

Configuration

Custom RAG

Best Practices

What’s Next

Function Tools

MCP

​Architecture

​Managed RAG

​Step 1: Upload a Knowledge Base

​Step 2: Configure the Knowledge Base

​Configuration

​Custom RAG

​Best Practices

​What’s Next

Function Tools

MCP

Architecture

Managed RAG

Step 1: Upload a Knowledge Base

Step 2: Configure the Knowledge Base

Configuration

Custom RAG

Best Practices

What’s Next