From Application Shell to Managed RAG - Fast Start and Black-Box Limits

The application shell should be built before the complete custom retrieval pipeline. This creates a working environment in which authentication, chat state, streaming, persistence, and observability can be tested independently of document parsing and vector indexing.

The first responsibility is authentication. Supabase authentication provides user accounts and sign-in. Authentication must be connected to row-level security so each user sees only their own records. A successful login page is not enough. The authenticated user identifier must be carried into thread, message, document, chunk, and storage operations.

The chat shell needs at least four persistent entities or behaviors:

1. A conversation thread.
2. User and assistant messages belonging to that thread.
3. Streamed assistant output.
4. A way to reload the same conversation later.

A thread list appears in the left navigation. The active thread loads in the main panel. When the user sends the first message, the application creates a thread and stores the message. The back end calls the model and streams the response. The final assistant message is saved. On refresh, the thread and message history are loaded from the database.

Several details that appear minor are important for usability and correctness. A new thread should not remain permanently titled "New chat." The title can be generated from the first user message. The interface needs a visible loading state while the back end is working. A stop control should cancel a long response. After one streamed answer finishes, the input must become usable for the next turn. The user menu should remain accessible even when the thread list becomes long.

A managed retrieval service is a useful first retrieval layer because it allows the chat experience to be tested before the custom ingestion system exists. Files are uploaded to a hosted vector store. The application sends the vector-store identifier to a responses-style model API. The provider manages parsing, chunking, embeddings, file search, and retrieval.

This creates a complete early path:

1. Create a hosted vector store.
2. Upload a document.
3. Attach the document to the vector store.
4. Store the vector-store identifier as a server setting.
5. Send a user question through the chat interface.
6. Allow the managed service to search the file and generate a grounded answer.

An appliance-manual example demonstrates the value. A refrigerator manual is uploaded to the hosted store. The user asks for information about the refrigerator or asks how to install a filter cartridge for a particular model. The service can return model-specific steps, including cautions such as not overtightening a component. The answer can be useful even though the application has not implemented its own parser, chunker, embedding pipeline, or vector query.

Managed retrieval is therefore a practical way to prove the shell. It can also be a valid final architecture for a small wrapper application when the team accepts the provider's storage, models, pricing, and retrieval behavior.

The limitation is that the retrieval process is hidden. The application cannot inspect or control:

• How the document was chunked.
• Which embedding model was used.
• Whether headings or sections were preserved.
• Which retrieval thresholds were applied.
• How similar files were distinguished.
• Whether keyword and semantic search were combined.
• Whether metadata was extracted or used.
• Why a particular passage was selected.
• How ranking changes as the knowledge base grows.

This matters when many similar documents exist. A few appliance manuals may work well. A collection containing hundreds or thousands of refrigerator manuals creates a harder problem. A user may include a model number, but the system still has to select the correct file and avoid mixing instructions from another product. When the service is a black box, the operator can test the answer but cannot inspect the retrieval design that produced it.

Managed retrieval also creates provider lock-in. The application is tied to the provider's storage system, file-search implementation, available models, response latency, data-handling rules, and pricing. The example pricing used during the build was 10 cents per gigabyte of file storage and $2.50 per 1,000 file-search tool calls. Those figures illustrate the cost structure: storage and retrieval calls both accumulate. A second managed file-search provider was cheaper in the example, but it carried the same uncertainty about future pricing and the same loss of retrieval control.

The model can also become a lock-in point. A slow stream from the managed responses API cannot simply be replaced with a local model if the file-search feature is tightly coupled to that provider. By contrast, a self-managed vector store can retrieve evidence and then send it to any compatible generation model.

Observability is mandatory even in the application-shell phase. LangSmith-style tracing should record every model call. A trace should show the system prompt, conversation history, user message, tool definitions, tool calls, tool arguments, retrieved results, and generated response.

A tracing failure is itself a product failure. When traces were missing, the application still produced answers, but the operator could not verify what was happening. Investigation revealed that an older Assistants-style API had been implemented even though the intended design had moved to a Responses-style API. The requirement itself contained the older choice, which is a reminder that specifications must be reviewed rather than treated as infallible.

Getting tracing to work may require checking several layers:

• The API key and project name are loaded from the correct environment.
• The model client version supports the tracing integration.
• The application wrapper does not bypass instrumentation.
• The stream is recorded as a complete model call rather than only partial chunks.
• The active server process is the one that received the new configuration.

The last point caused repeated confusion. Multiple Python and front-end services were running in the background. Restarting one process did not stop the others, so the browser sometimes connected to an older server. A restart script must terminate all matching processes before launching a fresh pair of services. Without that discipline, a correct code change can appear ineffective.

Browser automation can validate the shell. A test account signs in, creates a thread, sends a message, and confirms that the reply appears. The database is inspected to confirm that the thread and messages were stored. The observability platform is checked to confirm that the model call was traced. A refresh proves that history reloads. A second message proves that the input state was released after the first stream.

The shell should also support front-end corrections before deeper RAG work begins. Useful improvements include dynamic titles, a stop button, a loading indicator, stable navigation, and markdown rendering. These are not separate from RAG quality. A user cannot understand or trust an agentic system if tool activity and response state are invisible or erratic.

Managed retrieval should be treated as a controlled starting point. It proves authentication, messaging, streaming, persistence, and model connectivity. It also provides a baseline answer against which the later self-managed pipeline can be compared. The transition to custom retrieval is justified when the application needs control over chunking, embeddings, metadata, hybrid search, reranking, local models, security, cost, or provider choice.

Practical decision rule:

Use managed retrieval when speed of implementation and operational simplicity matter more than internal control. Use self-managed retrieval when the system must explain and tune how evidence is selected, support local or multiple model providers, enforce specialized security, or operate at a scale where black-box behavior and per-call cost become unacceptable.