Adding RAG: Chat with Documents Locally
Tech GuruThe short answer: Retrieval-Augmented Generation (RAG) lets a local chat model answer from selected documents without retraining the model. It can improve traceability and freshness, but only when extraction, retrieval, permissions, and source checking work as a complete pipeline.
This guide is for a beginner or homelab operator who already has Ollama and Open WebUI running and wants a cautious first document-chat workflow. You should be able to create a knowledge base, verify citations, detect failed retrieval, and return to a known-good index after a change.
Instead of permanently teaching the model your PDFs, manuals, notes, policies, or runbooks, the application extracts and indexes them. At question time it retrieves likely-relevant passages and places those passages in the model's context.
Beginner translation: the system searches first, then asks the language model to answer from what the search found.
Key Takeaways
Interactive RAG Flow: How Document Chat Works
Click each step to see what it means in a beginner-friendly local AI setup.
Choose a box above to view details.
- RAG does not train the model on your files.
- RAG finds relevant chunks of your documents and adds them to the prompt.
- Embeddings turn text into lists of numbers that can be searched by meaning.
- Vector databases store those embeddings and help find similar text.
- Open WebUI knowledge bases are the beginner-friendly way to add document chat.
- Local RAG is private only if the model, embeddings, vector database, and extraction tools are all local.
- Most RAG failures are caused by bad document text, bad chunking, stale indexes, or missing context.
Before You Start: Assumptions and Safe Defaults
- Use a small, non-sensitive document set whose correct answers you already know.
- Confirm the chat model, embedding model, extraction engine, and vector store you intend to use. A local chat model does not make a cloud embedding or OCR service local.
- Export the knowledge base when your Open WebUI release supports it, or preserve the original files and record the current embedding, chunking, and retrieval settings before re-indexing.
- Keep documents with different access rules in separate collections. A convenient shared index is not a substitute for authorization.
- Do not attach tools that can change files, send messages, or call privileged systems while testing untrusted documents.
The Open WebUI interface and defaults change between releases. The menu paths and behavior in this article were checked against the documentation available on July 15, 2026; reopen the Knowledge and RAG pages before publication or a major upgrade.
What Document Chat Really Means
When people say "chat with your documents," it sounds like the model reads every file and remembers it forever.
That is not usually what happens.
In a RAG setup, your system builds a searchable index of the documents. Later, when you ask a question, the system searches that index and gives the model only the most relevant chunks.
The model answers using:
- Your question.
- The conversation history.
- The retrieved document chunks.
- Its normal language ability.
This is why RAG is useful. It lets a smaller local model answer questions about information it was never trained on.
The RAG Pipeline
Here is the basic flow.
| Step | What Happens | Beginner Translation |
|---|---|---|
| 1. Upload | You add PDFs, text files, Markdown, docs, or other files. | "Here are my documents." |
| 2. Extract | The system pulls readable text out of the files. | "Turn the file into text the computer can process." |
| 3. Chunk | The text is split into smaller sections. | "Break a big manual into searchable pieces." |
| 4. Embed | An embedding model converts each chunk into a vector. | "Turn each piece into a meaning fingerprint." |
| 5. Store | The chunks, metadata, and vectors go into a vector database. | "Save the searchable index." |
| 6. Ask | You ask a question. | "What does the manual say about VLAN setup?" |
| 7. Retrieve | The system embeds your question and searches for similar chunks. | "Find the most relevant pieces." |
| 8. Generate | The LLM gets those chunks and writes an answer. | "Answer using the retrieved evidence." |
RAG is not one thing. It is a pipeline. If one step is weak, the answer gets weaker.
Embeddings in Plain English
An embedding is a list of numbers that represents meaning.
For example, these two sentences use different words but mean similar things:
- "How do I reset the router password?"
- "Where are the instructions to change my Wi-Fi admin login?"
A keyword search might struggle if the exact words do not match. An embedding search can do better because it compares meaning, not just exact words.
The embedding model turns each sentence, paragraph, or document chunk into a vector. Similar meanings land near each other in vector space.
You do not normally read these numbers. The computer uses them to search.
Important rule:
> If you change embedding models, re-index your documents.
Embeddings from different models live in different vector spaces. Mixing them is like putting two different map systems in the same folder and expecting the coordinates to line up.
What Is a Vector Database?
A vector database stores and searches embeddings.
It usually stores:
- The vector for each chunk.
- The original text chunk.
- Metadata such as file name, page, folder, date, or permissions.
When you ask a question, the system embeds your question and asks the vector database:
> Which stored chunks are closest in meaning to this question?
Common vector database options include Chroma, PGVector, Qdrant, Milvus, OpenSearch, Elasticsearch, and others. Open WebUI supports several options, with ChromaDB and PGVector commonly treated as maintained choices in its documentation.
| Vector Store | Good Beginner Use |
|---|---|
| Built-in / default Open WebUI setup | Easiest way to start. |
| Chroma | Simple local AI retrieval projects. |
| PGVector | Good if you already like PostgreSQL. |
| Qdrant | Good for more dedicated vector search setups. |
| OpenSearch / Elasticsearch | Useful when you already run search infrastructure. |
For a first homelab RAG setup, do not overthink the vector database. Start with what Open WebUI gives you, learn the workflow, then move to a dedicated database only when you have a reason.
Open WebUI Knowledge Bases
Open WebUI has a feature called Knowledge. This is where you can create collections of documents that the AI can search.
Typical beginner workflow:
- Open Open WebUI.
- Go to Workspace.
- Open Knowledge.
- Create a new knowledge base.
- Upload a small set of files.
- Attach that knowledge base to a chat or model.
- Ask questions and check the citations.
Open WebUI can use retrieval mode for large collections, where it searches for the most relevant chunks. It can also use full context mode for smaller files, where it injects the whole document into the prompt.
| Mode | What It Does | Best For |
|---|---|---|
| Focused Retrieval / RAG | Searches for relevant chunks and sends only those chunks to the model. | Large document sets, manuals, notes, policies. |
| Full Context | Sends the whole document content to the model. | Short documents where every line matters. |
Focused retrieval is the normal RAG path. Full context can be better for small documents, but it can quickly hit the model's context limit.
Local Embedding Models
For private local RAG, your embedding model should also be local.
As of the July 15, 2026 documentation check, Ollama's embeddings guide lists these examples:
embeddinggemmaqwen3-embeddingall-minilm
The chat model and embedding model are different jobs. Your chat model writes answers. Your embedding model turns text into searchable vectors.
| Model Type | Job | Example |
|---|---|---|
| Chat LLM | Reads the retrieved chunks and writes the answer. | A general or task-specific local chat model that fits your hardware. |
| Embedding model | Converts text into vectors for search. | embeddinggemma, qwen3-embedding, or all-minilm. |
| Reranker | Re-sorts search results after initial retrieval. | Optional; adopt only after a repeatable baseline shows a benefit. |
Do not use a giant chat model as your embedding model unless the tool specifically supports that. Embedding models are built for retrieval.
Choose Settings From Failure Signals
There is no universal chunk size, overlap, or Top K. Document structure, tokenizer, embedding model, question type, reranking, and the chat model's usable context all matter. Record the defaults in your installed release, then change one control at a time against a fixed question set.
| Observed failure | Controlled experiment | Guardrail |
|---|---|---|
| A section is repeatedly split away from its heading. | Increase overlap or use a structure-aware splitter, then re-index and rerun the same questions. | Check that extra overlap does not create duplicate, contradictory context. |
| Retrieved passages contain several unrelated topics. | Try smaller chunks or cleaner source documents. | Do not assume smaller is always better; fragments can lose necessary context. |
| The right passage ranks just below the cutoff. | Raise Top K by a small step or test a reranker. | Verify prompt size and answer quality; more passages can add noise. |
| The prompt is crowded or exceeds the model's context. | Lower Top K, reduce chunk size, or narrow the collection. | Do not remove the passages required to support the answer. |
Chunk size controls how big each document piece is. Overlap repeats a little text between chunks so sentences and sections do not get cut too harshly. Top K controls how many chunks come back from the search.
More is not always better. Too many chunks can bury the answer in noise or fill the context window.
Privacy: What "Local" Really Requires
Local RAG is private only if every step stays local.
| Component | Local Choice | Privacy Risk |
|---|---|---|
| Chat model | Run through Ollama, llama.cpp, vLLM, or similar. | Hosted chat models receive your prompt and retrieved chunks. |
| Embedding model | Run an Ollama or local sentence-transformer embedding model. | API embedding services receive document chunks. |
| Vector database | Store locally with Open WebUI, Chroma, PGVector, Qdrant, or similar. | Cloud vector databases store your document index. |
| Text extraction / OCR | Run local extraction tools. | Cloud OCR receives document contents. |
| Web search tools | Disable unless needed. | Retrieved private context could be combined with external requests. |
| Backups and logs | Encrypt or restrict access. | Logs may contain prompts, chunks, file names, and answers. |
The most common privacy mistake is running a local chat UI while using a cloud embedding API. In that setup, your documents may still leave your machine during indexing.
Ask this checklist before calling a setup private:
- Is the chat model local?
- Is the embedding model local?
- Is the vector database local?
- Is OCR or document parsing local?
- Are logs stored safely?
- Are cloud tools disabled unless I intentionally use them?
Common RAG Pitfalls
Most bad RAG answers are not because the model is "dumb." They usually come from retrieval problems.
| Symptom | Likely Cause | Fix |
|---|---|---|
| The model guesses instead of citing documents. | Knowledge base is not attached, citations are disabled, or retrieval found nothing. | Attach the right knowledge base and ask for answers based only on sources. |
| The answer uses old information. | Stale documents or stale index. | Update files and re-index. |
| Retrieval got worse after changing embedding models. | Old embeddings are incompatible with the new model. | Re-index the knowledge base. Re-upload standalone chat files. |
| Scanned PDFs return poor answers. | The PDF has images, not real text. | Run OCR or use better source files. |
| The answer misses a detail in a huge manual. | Chunks are too large, too small, or not enough chunks are retrieved. | Tune chunk size, overlap, and Top K. |
| The model mixes unrelated documents. | Knowledge base is too broad. | Split documents into focused knowledge bases. |
| Upload or indexing is slow. | Embeddings are running on CPU or the document set is large. | Use a smaller embedding model, batch gradually, or move embeddings to faster hardware. |
| The model cites the wrong page. | Metadata or extraction is messy. | Check extracted text and file parsing quality. |
| Private data appears in prompts or logs. | RAG context is being logged. | Restrict logs, scrub sensitive data, and limit admin access. |
How to Test Your RAG Setup
Do not test RAG with random questions. Build a tiny test set.
Use five types of questions:
| Test Type | Example | What a passing check supports |
|---|---|---|
| Known answer | "What port does the manual say the admin UI uses?" | The current pipeline can retrieve at least this expected fact. |
| Reworded answer | "How do I reach the management page?" | The current embedding and retrieval settings handle this paraphrase. |
| Negative answer | "Does this document mention SAML login?" | The tested model can abstain on this missing fact. |
| Source check | "Name the file and section used." | Source metadata survives this ingestion and answer path. |
| Conflict check | "Two docs disagree. Which one is newer?" | The system exposes enough date/version metadata for this case. |
Keep these tests in a note. When you change models, chunk size, embedding models, or vector databases, run the same tests again.
Evidence and Testing Method
This article is documentation-backed, not a TechGeeks lab benchmark. The workflow and re-index behavior were checked against current Open WebUI and Ollama documentation; the test matrix above is a method for readers to run on their own files. No retrieval-accuracy percentage, latency result, or universal setting is claimed here.
For a defensible local test, save the exact Open WebUI and Ollama versions, embedding model and digest, splitter, chunk size, overlap, Top K, reranker, chat model, and document hashes. Score retrieval separately from answer quality: first inspect whether the expected passage appeared, then inspect whether the answer used it correctly.
RAG vs Fine-Tuning
Beginners often ask whether they should fine-tune a model on their documents. Usually, the answer is no.
| Need | Better Tool |
|---|---|
| Ask questions about changing documents. | RAG |
| Cite current manuals or policies. | RAG |
| Add private knowledge without retraining. | RAG |
| Teach a model a style or repeated behavior. | Fine-tuning |
| Improve a narrow task format. | Fine-tuning |
RAG is usually the right first choice because you can add, remove, and update documents without retraining the model.
Pilot RAG with One Stable Manual
Try this with a small, low-risk document set:
- Create a knowledge base called
homelab-network-notes. - Upload your router manual, switch notes, and IP address plan.
- Ask: "Based only on the knowledge base, what device handles DHCP?"
- Ask: "Which file supports that answer?"
- Ask a question that should not be answered, such as: "What is my ISP account password?"
- Confirm the model says it cannot find that information.
That last test matters. A good RAG setup should know when the answer is not in the documents.
Keep Document Sets Clean
Good RAG starts with good documents.
Practical habits:
- Use clear file names.
- Remove outdated versions.
- Split unrelated topics into separate knowledge bases.
- Prefer text-based PDFs over scanned PDFs.
- Add headings to Markdown and text notes.
- Keep secret files out unless you truly need them.
- Re-index after changing chunking or embedding settings.
Garbage documents create garbage retrieval. Clean folders beat fancy settings.
A Conservative Starting Configuration
Start with Open WebUI Knowledge and a small local embedding model. Use one folder of documents that you understand well. Ask questions where you already know the answer. Check whether the model cites the right file.
Only after that should you tune chunk sizes, switch vector databases, add rerankers, or build agentic document tools.
The first acceptance target is intentionally narrow:
> Can my local AI answer from my documents without making things up?
Once that works, you can scale up.
Risk, Recovery, and Data Boundaries
Recovery: re-indexing deletes and rebuilds a knowledge base's vector collection in current Open WebUI behavior. Before changing the embedding model or splitter, preserve the source files, export the knowledge base where supported, and record the old settings. If quality drops, restore the prior settings and rebuild from the unchanged source set. Standalone files uploaded directly to chats are not covered by the knowledge-base re-index operation; re-upload them after an embedding-model change.
Security: treat retrieved text as untrusted data. A document can contain visible or hidden instructions intended to redirect the model. RAG does not fix prompt injection. Keep tool permissions least-privileged, require human approval for consequential actions, and never let a citation or model answer authorize a command by itself.
Privacy and legal scope: index only material you are authorized to process. Check employer, client, copyright, retention, and data-protection obligations before loading contracts, medical records, credentials, or personal data. Deleting a source file is not proof that every index, cache, chat, log, and backup copy has been deleted.
What This Evidence Does Not Prove
- A correct answer on five questions does not establish retrieval quality for every page, language, table, image, or scanned PDF in the collection.
- A citation proves that the application attached a source reference; it does not prove that the source is current, authoritative, or interpreted correctly.
- Running Ollama locally does not prove document privacy if extraction, embeddings, web search, telemetry, logs, or backups use external services.
- The original RAG paper and vendor documentation explain architecture and product behavior; they do not benchmark this reader's hardware, files, or selected models.
- Passing retrieval tests does not make an agent safe to execute commands from retrieved content.
Related TechGeeks Reading
- Implementing AgenticOps Safely: Human Approval, Audit Trails, and Rollback
- Linux and Homelab Notes: Start Here
References
- Open WebUI: Knowledge
- Open WebUI: Retrieval-Augmented Generation
- Open WebUI: RAG Troubleshooting
- Ollama: Embeddings
- Lewis et al.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Liu et al.: Lost in the Middle
- OWASP: RAG Security Cheat Sheet
Need help applying this?
Bring TechGeeks into the real environment.
If you are working through this on a live network, WordPress site, Linux server, AI workflow, or PisoWiFi deployment, send the context and we can help turn it into a practical plan.
