Owning the Retrieval Pipeline - Ingestion, Embeddings, Configuration, and Security

A self-managed RAG system begins with an ingestion pipeline that the application controls from upload to vector storage. This replaces the managed file-search black box with explicit stages that can be inspected, configured, tested, and improved.

The visible entry point is a document interface. A user drags a file into the browser. The interface creates a document item and shows a status such as pending, processing, completed, skipped, or failed. The status should update in real time as the back end moves through the pipeline.

The back-end flow is:

1. Authenticate the user.
2. Validate the file type and size.
3. Write the original file to object storage.
4. Create a document record containing the owner, file name, storage path, and status.
5. Extract text.
6. Divide the text into chunks.
7. Generate embeddings for the chunks.
8. Store chunk content, vectors, indexes, ownership, and metadata.
9. Mark the document as completed.
10. Return errors in a form the interface can display.

The storage path should include the authenticated user's identifier. This creates a clear relationship between the account and the object. A database record should also include the user identifier so row-level security can apply. The document table and storage bucket must agree about ownership.

The document table represents the file as a managed object. Useful fields include:

• Document identifier.
• User identifier.
• Original file name.
• Storage path.
• File size.
• Processing status.
• Content hash.
• Extracted metadata.
• Created and updated timestamps.
• Failure details when processing does not complete.

The chunk table represents retrieval units. Each row should contain:

• Chunk identifier.
• Parent document identifier.
• User identifier.
• Chunk index or sequence.
• Text content.
• Embedding vector.
• Metadata copied or derived from the parent document.

The parent relationship is important for deletion and whole-document analysis. When a document is removed, the original file, document row, and all child chunks should be removed. The interface should react immediately. A bug in the early build deleted the back-end data correctly but left the document visible until refresh. Enabling the appropriate real-time update or updating local state fixed the user-facing inconsistency.

The pipeline should initially support simple formats such as text and Markdown. This keeps early failures easy to diagnose. Once storage, chunking, embeddings, deletion, and user isolation work, the parser layer can add PDFs and office documents.

Embedding configuration must be separate from chat-model configuration. The generation model answers questions. The embedding model defines how chunks and queries are represented in vector space. These are different jobs and may use different providers.

A global settings interface can expose:

• Chat provider.
• Chat model name.
• Chat base URL.
• Chat API credential.
• Embedding provider.
• Embedding model name.
• Embedding base URL.
• Embedding API credential.
• Embedding dimension.
• Reranker configuration.
• Web-search configuration.

The settings should be global when the underlying vector store is shared. Allowing every user to choose a different embedding model would create conflicting embedding configurations and dimension requirements in the same database.

The system should prevent the embedding configuration from changing while chunks exist. The administrator must delete and re-embed the knowledge base before switching models. The interface can disable the fields and explain why. This makes an architectural constraint visible instead of allowing a change that silently corrupts retrieval.

A concrete failure illustrates the dimension problem. The database vector column was created for 1,536 dimensions. The administrator then selected an embedding model configured to return 4,096 dimensions. Parsing succeeded and the embedding API returned a vector, but PostgreSQL rejected the write because the vector did not match the schema.

The immediate fix removed the hardcoded dimension from the column so the upload could succeed. That solved the insertion problem but raised a second issue: vector indexes and search behavior need a deliberate dimension strategy. The lesson is that a front-end dimension field is not enough. Embedding dimensions affect schema design, indexing, migration, search performance, and the cost of changing models.

Provider abstraction should be tested rather than assumed. The system can use a cloud-routing service for chat, a different cloud or local endpoint for embeddings, and LM Studio for local models. An OpenAI-compatible API shape makes the integrations similar, but the application still has to handle base URLs, model names, credentials, stream formats, errors, and context limits.

One test configuration used a cloud-routed GLM model for generation and a Qwen embedding model with 4,096 dimensions. Another configuration used LM Studio on a local network with a Qwen vision-language mixture-of-experts model for chat and a small Qwen embedding model. A text version of a Samsung electric-dryer manual was uploaded, embedded locally, and then queried through the chat interface. The result showed that the same application could work without sending the knowledge base to a cloud model provider.

The local test also demonstrated that model configuration should be visible and explicit. A chat initially worked even though the settings interface showed no provider values. The back end was silently falling back to environment variables. That behavior made the interface misleading. A correct system either shows the active fallback or fails clearly until the administrator saves a valid configuration.

Secret handling requires a separate design decision. Storing API keys as plain text in a general settings table is not acceptable. Global provider settings should be available only to administrators. A user-profile table can record whether an account is an administrator. New users receive a profile, and row-level policies restrict settings reads and writes.

Some secrets may be better kept in server environment variables. A server-wide service key that ordinary users never change may not need a database-backed interface. Other values may need controlled administrative editing. The critical requirement is that the browser and ordinary users cannot retrieve privileged credentials.

Supabase service-role credentials, database passwords, model API keys, and web-search keys must not be placed in version control. Front-end code must never receive a service-role secret. A public browser client can use the anonymous or publishable key with row-level security. Privileged operations remain on the back end.

Multi-user isolation must be tested with at least two accounts. A practical test is:

1. Sign in as the second user.
2. Upload a document.
3. Confirm the document, chunks, and storage object exist under that user's identifier.
4. Sign out.
5. Sign in as the first user.
6. Confirm the second user's document is invisible.

The database may contain both users' records. The policy must ensure that each authenticated session sees only its permitted subset.

Shared tables require explicit policy as well. A table intended to be readable by all signed-in users should still have row-level security enabled with a policy for authenticated read access. Leaving the table unrestricted can expose it more broadly than intended.

The ingestion interface should also make failures understandable. If embedding fails because the dimension is wrong, the user should not see a truncated database error. The document should move to failed state, and the stored failure information should help an administrator identify the stage. Processing should be restartable after the configuration is corrected.

Practical ingestion and configuration rules:

• Keep original files, document records, and chunks linked by stable identifiers.
• Store ownership on every user-controlled record.
• Use row-level security and user-specific storage paths.
• Separate chat, embedding, reranking, and parsing configurations.
• Treat embedding changes as a knowledge-base migration.
• Lock embedding settings while vectors exist.
• Do not hide active fallbacks from the administrator.
• Restrict global settings to administrators.
• Keep privileged secrets out of the browser and repository.
• Verify deletion, real-time updates, and multi-user isolation directly.