Build a Private RAG Box for Network Docs
Tech GuruQuick Answer
Build a narrow, authorized document search before indexing a whole network share. Preserve authoritative sources, review parsed tables and diagrams, attach version and access metadata, and test retrieval separately from generated answers. Show supporting citations and refuse unsupported questions, but enforce permissions outside the model. Keep network tools read-only: even a well-cited answer does not authorize a production change.
RAG rule: Keep authoritative documents outside the index, attach version and access metadata to every chunk, and require the answer to expose its sources. Retrieval can narrow evidence; it cannot authorize a network change.
The Reader Question
Can I chat with configs, diagrams, notes, and runbooks locally?
This guide is for a homelab operator or small network team that already has files worth searching and can run containers, protect storage, and restore a database. It assumes source documents have owners and access rules. It does not assume an existing vector database, nor does it recommend indexing a shared drive before the document lifecycle is understood.
Current Tooling Context
Treat parsing, embeddings, retrieval schema, and the source snapshot as versioned parts of the system. A parser upgrade can change reading order or table structure, while semantically similar text from the wrong firmware branch can answer the wrong question. Pin the components together and inspect the new output before replacing a working index.
Before You Start: Safe Defaults
- Redact secrets before indexing configs and tickets.
- Keep source documents authoritative and versioned.
- Use access controls if docs contain private network details.
- Create 30-50 real questions before trusting the system.
Reference Model
Collect an authorized source set, inspect the parsed chunks, retrieve the correct version, and evaluate whether the answer is supported by its citations.
Decision Matrix
| Choice | Best Fit | Watch Point |
|---|---|---|
| Qdrant | Dedicated vector/hybrid search service | Another service to back up and monitor. |
| pgvector | Postgres shops and simpler stack | Operational database now carries retrieval workload. |
| Open WebUI RAG | Fast starter UI | Still needs source hygiene and evals. |
| Cloud RAG | Hosted retrieval or integrations that meet a tested requirement | Data exposure and cost tradeoffs. |
What to Ingest First
Start with documents that already answer real questions: network diagrams, IP plans, device configs, change records, outage notes, standard operating procedures, and source-of-truth exports. Do not start with every PDF ever downloaded.
Chunking and Metadata
A network config chunk should know its device, site, vendor, OS, firmware branch, timestamp, source path, document version, and stanza when possible. A runbook chunk should know service, owner, approved status, last review date, and recovery priority. Preserve page numbers, headings, table titles, figure captions, and diagram descriptions so citations return readers to the evidence. Do not mix current and superseded firmware procedures in one unqualified collection.
Difficult Documents Need a Parse Gate
Scanned PDFs need OCR; tables need row and header structure; multi-column manuals need correct reading order; charts and diagrams need captions or a deliberate multimodal path. Review a sample of parser output before embedding. Quarantine empty pages, scrambled columns, missing command blocks, low-confidence OCR, and password-protected files. A parser that finishes without an error has not proved that the rendered document survived.
Trust Boundary
The RAG box may draft explanations, compare docs, summarize configs, and propose change plans. It should not auto-apply changes. Network changes still need source-of-truth validation, review, and rollback evidence.
A Practical Pilot Scenario
Start with one site, one current runbook set, a sanitized configuration snapshot, and two deliberately stale or wrong-version documents. Write questions whose answers are known, questions that require a table or scanned page, and questions the corpus cannot answer. Build a separate collection for the pilot rather than pointing ingestion at production shares.
The pilot passes when current questions retrieve the correct version and page, stale branches are filtered, unsupported questions refuse, and the answer cites text that actually supports the claim. Score retrieval separately from answer generation: a fluent response cannot repair missing evidence, and correct retrieval does not guarantee faithful synthesis.
Implementation Details
Assign immutable document IDs and content hashes, then version parser output, chunks, embeddings, and schema. For Qdrant, evaluate a new collection before switching its alias; for pgvector, design and test the PostgreSQL table/index migration and application switchover separately. Apply source deletion and access revocation to every serving version, cache, and rollback path. Retained backups need restricted access and an explicit retention/restore policy so rollback cannot silently republish deleted or newly unauthorized material.
- Choose a narrow, authorized source set and define owners, versions, retention, and exclusions.
- Remove secrets, credentials, customer data, and private keys before the parser sees the files.
- Parse to inspectable text or JSON; review tables, scans, diagrams, page order, and command blocks.
- Quarantine parse failures and attach source, page, site, device, firmware branch, owner, and approval metadata.
- Create embeddings in a versioned Qdrant collection or PostgreSQL tables with pgvector. Enforce and test caller-specific access before returning source content; metadata labels alone do not enforce permission.
- Configure citations and refusal behavior without giving the model write access to network systems.
- Run retrieval, answer, stale-version, access-control, malicious-document, deletion, and restore tests.
- Switch to the new collection only after acceptance, then schedule monitored reindexing and backups.
Evidence To Collect
- Source manifest with document ID, hash, owner, approval state, version, access class, and retention rule.
- Parser samples for prose, tables, scans, diagrams, and configuration blocks, including quarantined failures.
- Per-question retrieved chunks, ranks, metadata filters, expected source/page, answer, citations, and refusal result.
- Negative tests for stale firmware branches, unauthorized documents, embedded prompt injection, deleted sources, and an empty corpus.
- Reindex logs, collection or table version, backup artifact, isolated restore result, and backend-specific switchover timing, including deletion and access-revocation checks after rollback.
Validation Checklist
- Answers cite the source documents used.
- The system refuses questions outside indexed evidence.
- Secret material is not present in the vector store.
- A changed source document is reflected after reindex.
- Eval questions catch at least one bad answer before production use.
Maintenance Cadence
- After each source change: verify ingestion status, hash, version metadata, deletion handling, and one known question.
- Monthly: inspect failed parses, stale approvals, access-filter denials, citation misses, storage growth, and backup completion.
- Before parser or embedding upgrades: rebuild into a new collection and rerun the frozen eval set.
- Quarterly: restore the vector store and configuration in isolation, then compare counts, filters, citations, and refusal behavior.
Troubleshooting
| Symptom | Likely Cause | First Check |
|---|---|---|
| Answer cites wrong site | Weak metadata or retrieval | Add site/device/source metadata and improve chunking. |
| Answer makes up command | LLM unsupported by retrieved docs | Require citations and refusal behavior. |
| Fresh doc ignored | Reindex job failed or stale vector store | Check ingestion logs and source timestamps. |
Common Mistakes
- Indexing stale configs and calling the answer current.
- Skipping redaction because the box is 'local.'
- Using embeddings without citations.
- Trusting RAG before testing with real network questions.
- Mixing personal docs, vendor PDFs, and production configs without access controls.
Useful Gear And Buyer Notes
Buy storage or compute only after the pilot records corpus size, parse time, index size, query latency, backup window, and model memory. Verify RAM expansion, SSD endurance, encryption support, idle power, replacement parts, and return terms; a larger box does not correct stale sources, broken parsing, or missing access control.
Affiliate disclosure: As an Amazon Associate, TechGeeks may earn from qualifying purchases. The product links below are buying references, not a requirement to buy a specific brand or seller. Verify compatibility, seller quality, warranty, and current specs before ordering.
- Amazon search: Intel N100 mini PC 32GB
- Amazon search: NVMe SSD 4TB
- Amazon search: NAS backup drive
- Amazon search: 64GB RAM mini PC
- Amazon search: UPS USB
Related TechGeeks Reading
- AI Workflow Notes: Start Here
- Building a Network Digital Twin Workflow
- Implementing AgenticOps Safely: Human Approval, Audit Trails, and Rollback
What This Does Not Protect or Validate
A citation proves only that a passage was presented with the answer. It does not prove that retrieval was complete, the document was current or authorized, the passage supports the entire claim, or the model interpreted it correctly. Parser confidence and a successful index count do not prove that tables, diagrams, OCR, or reading order survived.
No original RAG ingestion or retrieval lab was performed for this article. Documentation-backed design cannot establish accuracy, performance, privacy, or restore time for your corpus. Retrieved documents are untrusted input and can contain indirect prompt injection; enforce access in code, keep tool credentials out of the model, require human approval for changes, and retain the authoritative source and non-AI recovery path.
Practical FAQ
Can this replace NetBox?
No. RAG helps ask questions over docs; NetBox/Nautobot remain better sources of structured intent.
Should I use Qdrant or pgvector?
Use Qdrant for a dedicated retrieval service and hybrid search. Use pgvector if Postgres is already your operational comfort zone.
Can it read diagrams?
Only if the diagrams are exported or described in text well enough for retrieval, or you add a multimodal pipeline deliberately.
References
- https://github.com/docling-project/docling/releases
- https://www.ibm.com/new/announcements/docling-for-ibm-watsonx-turn-complex-documents-into-ai-ready-data
- https://qdrant.tech/documentation/search/hybrid-queries/
- https://qdrant.tech/blog/branch-aware-code-search/
- https://github.com/pgvector/pgvector
- https://genai.owasp.org/llmrisk/llm01-prompt-injection/
