Skip to content
Easy AI How-To

AdvancedGrokBuilding

Build a simple RAG pipeline with the xAI Grok API

RAG is how AI apps answer questions from private documents. This guide builds a minimal, working version in Python: chunking, retrieval, and a cited answer from Grok via the xAI API. It avoids extra databases so you can see every moving part.

Time
About 40 minutes
Checked
Tools
xAI API docs, xAI Console

If a button name on the screen does not match this guide, follow the action. These products rename buttons often. This is an advanced guide: it uses code, and each technical term is explained the first time it appears. Model names and prices change, so check the linked docs before you ship.

Steps

  1. Understand RAG in one paragraph

    RAG (retrieval-augmented generation) means: before asking the model, you retrieve the few passages from your documents that best match the question, and put them in the prompt. The model then answers from those passages instead of from memory, which cuts made-up answers and lets you cite sources. This guide uses simple keyword-based retrieval (TF-IDF) so it runs with no extra services.

  2. Install and set your key

    xAI's API is compatible with the OpenAI SDK, so you use that library with xAI's address.

    python3 -m venv .venv && source .venv/bin/activate
pip install openai scikit-learn
export XAI_API_KEY="your-key-here"

  3. Split documents into chunks

    Models work best with short, focused passages. A chunk is a slice of a document, here about 150 words, overlapping a little so sentences aren't cut off from context.

    from pathlib import Path

def load_chunks(folder="docs", size=150, overlap=30):
    chunks = []
    for path in Path(folder).glob("*.md"):
        words = path.read_text().split()
        for start in range(0, len(words), size - overlap):
            text = " ".join(words[start:start + size])
            chunks.append({"source": f"{path.name}#{start}", "text": text})
    return chunks

  4. Retrieve the best chunks

    TF-IDF scores words that are frequent in a chunk but rare overall, and cosine similarity measures how closely the question and each chunk match.

    from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

chunks = load_chunks()
vectorizer = TfidfVectorizer(stop_words="english")
matrix = vectorizer.fit_transform([c["text"] for c in chunks])

def retrieve(question, k=4):
    scores = cosine_similarity(vectorizer.transform([question]), matrix)[0]
    best = scores.argsort()[::-1][:k]
    return [chunks[i] for i in best if scores[i] > 0]

  5. Ask Grok with the retrieved context

    Send the passages and the question to the xAI Responses API. The prompt tells Grok to answer only from the context and cite sources.

    import os
from openai import OpenAI

client = OpenAI(api_key=os.environ["XAI_API_KEY"], base_url="https://api.x.ai/v1")

def answer(question):
    hits = retrieve(question)
    if not hits:
        return "I couldn't find that in the documents."
    context = "\n\n".join(f"[{h['source']}]\n{h['text']}" for h in hits)
    prompt = (
        "Answer the question using ONLY the context below. "
        "Cite sources in square brackets. If the answer isn't there, say so.\n\n"
        f"Context:\n{context}\n\nQuestion: {question}"
    )
    response = client.responses.create(model="grok-4.7", input=prompt)
    return response.output_text

print(answer("What is our refund window?"))

  6. Test with questions you know the answers to

    Write 10 questions with known answers, including 2 that the documents don't cover. Check that answers are right, cite the right file, and say "not in the documents" when they should. If retrieval misses, try smaller chunks or a larger k.

Tips

  • When keyword search isn't enough (synonyms, paraphrases), swap TF-IDF for embeddings, which match by meaning. The rest of the pipeline stays the same.

  • Keep the model name in one constant. xAI's models page lists current names; grok-4.7 was the recommended model when this was checked.

  • Log the retrieved sources for every answer. Most RAG bugs are retrieval bugs.

Common questions

Why not paste all my documents into the prompt?

For a few short files, you can. As documents grow, retrieval keeps prompts smaller, cheaper, and more focused.

Can I use Claude or ChatGPT instead of Grok?

Yes. Only the final call changes. The chunking and retrieval code works with any model.

Does Grok search X or the web here?

Not unless you enable xAI's search tools. In this pipeline Grok sees only the context you send.

More guides

Occasional notes

Hear when a new guide is up

A short note when a guide is added. No account, and no emails until this form is connected to a real list.

Placeholder. This form is not connected to an email service. Nothing you type is saved or sent.