RAG Pipeline

At a glance

Who it serves

AI-product builders needing attributable context from separate source collections.

What I owned

Standalone ingestion, chunking, embedding, retrieval and context-packing package.

Key decision

Scope retrieval by namespace and carry source tags into the packed context.

Current state

Public package with an offline demo and provider/store adapters. Reranking remains planned.

Evidence and gap

Repository and deterministic tests. Semantic retrieval quality and private-app deployment are not established here.

Problem

Every AI product eventually needs product-specific memory. Dreamsea needs dream interpretation material. nsso needs profile and identity context. Qadam needs financial research notes. 24Seven needs travel and catalogue knowledge. Rebuilding chunk, embed, store, retrieve, and prompt-inject logic for every product creates duplicated plumbing and inconsistent retrieval quality.

The deeper product problem is authority. AI systems are only useful when they know which sources they are allowed to trust, which product namespace they are answering inside, and which retrieved chunks shaped the answer. One-off prompts and copy-pasted context cannot provide that discipline.

The opportunity was a shared ingestion-to-retrieval layer: add a source once, tag it with a namespace, and let multiple products query the right context without polluting each other. A shared namespace can carry reusable knowledge, while product namespaces preserve isolation.

The public repo is deliberately standalone. It is not pretending to be wired into private products; it is the reusable context layer extracted from the repeated need to build RAG infrastructure across several AI products.

Architecture

Model: Unified RAG Pipeline is a Python 3.9+ package with a zero-dependency core, an offline default, and a production path for Gemini embeddings plus Supabase/pgvector. It exposes a Pipeline object that ties chunking, embedding, vector storage, retrieval, and context packing together. Each chunk is embedded through a pluggable embedder. HashingEmbedder is the offline default: deterministic feature hashing with L2 normalisation, no API key, no network, and reproducible tests. GeminiEmbedder uses text-embedding-004 for production and can report embedding token usage to a sink such as AI Costs Dashboard.

Context: ingestion starts with Pipeline.ingest(text, namespace, source_id). Text is split by a heading-aware chunker that keeps Markdown headings with their bodies, size-bounds each section, adds overlap, and tries to break on sentence or whitespace boundaries instead of cutting mid-word. The store interface has two implementations. SQLiteVectorStore stores JSON embeddings locally and performs brute-force cosine search in pure Python, which is enough for local development and modest corpora. SupabaseStore targets pgvector through a match_documents RPC, ivfflat cosine index, and the schema in supabase/schema.sql. Retrieval is namespace-scoped: Pipeline.query embeds the query, searches the selected namespaces, applies optional metadata filters, ranks by cosine score, and returns Hit objects with namespace, source_id, content, score, and metadata.

Orchestration: every chunk gets a content_hash, and the store enforces uniqueness on namespace plus content_hash. Re-ingesting identical chunks becomes a no-op; edited or near-duplicate text still needs separate handling. context_pack retrieves hits, greedily packs them under an approximate token budget, and prefixes each chunk with a source tag such as [dreamsea:jung-water]. The CLI exposes urag ingest, query, namespaces, and stats.

Governance: namespace-scoped queries keep selected corpora separate, and content hashes prevent exact chunk duplicates within a namespace. Packed chunks carry namespace and source_id tags so a reader can trace the supplied context. These are structural controls: they do not verify source truth, enforce tenant authorization, or prove that a generated answer follows its citations. The offline demo exercises namespace selection, exact-duplicate ingestion, and cited context packing; retrieval-quality evaluation remains planned.

Human: a curator decides what belongs in each namespace and which sources are credible enough to supply context. The pipeline preserves source labels and scopes queries; it cannot decide whether the source is truthful, current, or appropriate for a particular answer. Those judgments need curation policies and retrieval evaluation.

Why this approach

The most important product choice is namespaces. A shared RAG layer is only useful if different products can reuse infrastructure without sharing every source by default. Namespace scoping gives each product its own corpus while still allowing a shared knowledge layer.

The offline HashingEmbedder is not pretending to be production semantic retrieval. It exists so the package can be installed, tested, and demonstrated without keys, network access, or provider spend. That lowers adoption friction and makes the architecture inspectable.

The production path is intentionally pluggable rather than hard-coded. Gemini embeddings and Supabase/pgvector are supported, but the Pipeline interface can accept another embedder or store if a product later needs OpenAI embeddings, a different vector database, or a hosted retrieval service.

Idempotent ingestion is treated as a first-class product feature because retrieval systems degrade quietly when duplicate chunks accumulate. The content-hash key makes repeated ingestion operationally safe and reduces the maintenance burden on teams that sync sources on a schedule.

Citations are included in the context pack because a RAG pipeline should not just retrieve; it should preserve source accountability. The model prompt can then carry attributable context instead of anonymous pasted text.

Tradeoffs

The first tradeoff is the zero-dependency SQLite path's brute-force cosine search. It is clear and reliable for demos or modest corpora, but it is not the right retrieval engine for high-volume production search.

The second tradeoff is the HashingEmbedder. It makes tests deterministic and offline, but it cannot match the semantic quality of a production embedding model - it is a development default, not a claim of state-of-the-art retrieval.

Third, the current retrieval layer is vector-only. The roadmap correctly calls for reranking, MMR diversity, hybrid keyword/vector search, and incremental re-embedding when embedding models change.

Fourth, the context pack uses a simple greedy token budget approximation. That is enough to create prompt-ready context, but a mature product would need more precise token accounting, stronger source diversity, and task-specific packing rules.

Finally, namespaces prevent obvious corpus pollution, but they do not solve source authority by themselves. A real product still needs curation rules, freshness policies, deletion workflows, and evaluations for retrieval quality.

Demo / proof

The public GitHub repository includes README, package source, CLI, examples, Supabase schema, architecture diagram, tests, pyproject metadata, and MIT license.

The local test suite was run against the cloned repository and passed 14 tests covering chunking, deterministic embeddings, L2 normalisation, relevance ranking, idempotent ingestion, namespace isolation, context-pack budgets, SQLite store idempotency, and cosine ranking.

The offline demo shows Dreamsea, shared, and 24Seven namespaces; re-ingesting the same Dreamsea source is skipped; querying dreamsea plus shared excludes the 24Seven travel source; context_pack returns cited chunks.

The production path is documented through GeminiEmbedder, SupabaseStore, and supabase/schema.sql, which defines the pgvector table, cosine index, and match_documents RPC expected by the store.

GeminiEmbedder exposes a usage-reporting hook that can send embedding spend to a sink such as AI Costs Dashboard. That is an integration option between two standalone tools, not evidence that both are deployed across the private apps.

What I would improve

The first improvement is reranking and MMR diversity so retrieved context is not only similar but also varied and useful under a limited prompt budget.

The second improvement is hybrid search. Keyword and vector retrieval together would improve exact-name, identifier, and quote-heavy queries where embeddings alone can miss important terms.

The pipeline needs a retrieval evaluation harness: labelled queries, expected source chunks, precision/recall, citation coverage, hallucination-resistance tests, and regression runs when chunking or embedding settings change.

Incremental re-embedding should become explicit product behavior. When the embedding model changes, the system needs to know which chunks are stale, which namespaces are affected, and what cost the re-embedding job will create.

The context trace should become visual in the portfolio: query, namespaces searched, hits returned, chunks packed, citations injected, and answer generated. That would make the invisible context layer legible to non-engineering visitors.