Testing, Debugging, Local Operation, and Production Readiness

A working demonstration is not the same as a production-ready RAG system. A full path may succeed from upload to answer while important failures remain hidden in permissions, migrations, concurrency, retrieval quality, event persistence, or model behavior. The correct final stage is not deployment by default. It is systematic validation and an honest assessment of maturity.

Testing should begin early. Waiting until all modules are complete creates a large and expensive regression problem. Each new feature can break an earlier feature, especially when AI coding agents make broad changes across the front end, back end, database, and configuration.

A regression suite should be derived from the requirements and saved implementation plans. Each feature needs explicit acceptance criteria. The suite can cover:

Authentication and identity:

• The login page loads.
• Valid users can sign in and sign out.
• Two users see isolated data.
• Administrator-only settings are unavailable to ordinary users.

Chat and persistence:

• A thread is created.
• A response streams.
• The stream can be cancelled.
• A second message can be sent.
• Messages persist after refresh.
• Dynamic titles are created.
• Older threads reload correctly.
• Tool and subagent events remain visible after reload.

• Supported files upload.
• Unsupported types fail clearly.
• Status moves through pending, processing, completed, skipped, or failed.
• Original files reach the correct user-specific storage path.
• Document and chunk rows are created.
• Embeddings have the expected dimensions.
• Deletion removes storage, document, and chunk data.
• The interface updates without refresh.
• Duplicate content is skipped.
• Changed content replaces old chunks.

• Semantic search returns relevant paraphrases.
• Keyword search finds exact identifiers.
• Hybrid search returns a useful fused candidate set.
• Metadata filters work when correct.
• A broader retry occurs when filters are too restrictive.
• Reranking changes or confirms candidate order.
• Traces contain the expected scores and tool arguments.

• SQL aggregates return the correct value.
• SQL follow-ups preserve the intended constraint.
• The SQL role cannot modify data.
• The SQL role cannot read unrelated tables.
• Web search is called only when required.
• Subagents receive the correct document identifier.
• The main agent produces a final answer after tool use.
• Tool-round limits stop uncontrolled loops.

• Tool calls render in chronological order.
• Completed steps remain visible.
• Reasoning panels work for models that emit tags.
• Models without tags still render normally.
• Subagent output is not duplicated.
• Long threads do not hide the user menu.
• Light and dark modes render correctly.

Browser automation such as Playwright can exercise the full interface. API tests can validate back-end endpoints. Database checks can verify rows, vectors, policies, and cascades. A small fixture document provides repeatable input. Larger realistic manuals are needed for performance and retrieval evaluation.

Not every test needs a language model. Deterministic behavior should be tested with ordinary scripts where possible. Authentication redirects, database constraints, file deletion, schema shape, and permission denials do not require an agent to inspect them. Codifying these tests reduces model usage and makes repeated validation faster.

Agent-driven testing remains useful for exploratory behavior. An agent can navigate the interface, inspect traces, compare the implementation with a plan, and identify edge cases. The two approaches are complementary: deterministic tests protect stable contracts, while exploratory tests investigate complex interactions.

Smoke tests prove only that the main path works. The build used smoke tests after each module: upload a file, ask a question, inspect the trace, and verify a few records. This is useful for rapid progress, but it does not establish reliability under load, across many users, or across difficult retrieval cases.

Debugging should start by locating the failed layer. A practical classification is:

1. Browser rendering or state.
2. Streaming protocol.
3. Back-end application logic.
4. Model-provider call.
5. Document parser.
6. Database query or migration.
7. Authentication or permission policy.
8. Retrieval and ranking.
9. Prompt or tool-selection behavior.
10. Local hardware or service configuration.

Different tools reveal different layers.

Model traces show prompts, tools, arguments, results, and generation. They are the best place to diagnose a wrong metadata filter, a missing tool call, an incorrect SQL query, or a model that stopped after retrieval.

Server logs show application exceptions, database-client behavior, subagent coordination failures, and event-stream errors. They are needed when the trace shows that a tool was called but the application failed before returning the result.

Direct database queries verify whether a stored procedure, restricted user, aggregate, migration, or row-level policy works. The Metro Office SQL bug was resolved by comparing the generated SUM query with the result of running it directly.

Browser automation and screenshots reveal layout, scrolling, persistence, and chronological-rendering defects.

Resource monitoring shows whether parsing or local models are exhausting the host.

A dedicated debug mode can create hypotheses, insert temporary logging, reproduce the issue, narrow the cause, and remove the instrumentation afterward. When one coding agent repeatedly fails on the same issue, switching to another agent or debugging mode can be more effective than repeating the same instruction.

Fresh context also improves debugging. An agent that has consumed most of its context may continue an earlier wrong assumption. Before clearing the session, it should update the progress file with the exact error, reproduction steps, attempted fixes, relevant trace identifiers, and next hypothesis. A new agent can then investigate from a cleaner state.

Repeated problems should create durable improvements. If agents repeatedly start the wrong services, improve the restart script. If migrations are repeatedly misunderstood, add the exact remote-migration instructions to the project rules. If a filter failure recurs, add a regression test and prompt rule. If a rendering event disappears, update the event schema and persistence test.

Local operation introduces additional variables. LM Studio can serve OpenAI-compatible local endpoints for chat and embeddings. The build used Qwen-family models, including a vision-language mixture-of-experts model, a 32-billion model with reasoning tags, and local embedding models.

A system with a high-end GPU containing 32 gigabytes of VRAM could run a model with a context window configured around 70,000 tokens, although this approached the hardware limit. The exact practical limit depends on the model and the rest of the workload.

Server restarts can reset model settings. A model that had been loaded with a 70,000-token context returned to a 4,000-token default after restart. The resulting failures looked like application or agent problems until the context configuration was checked. Operational procedures should therefore verify which models are loaded, what context limit is active, and how much memory remains.

Model behavior is not uniform. One local model emitted think tags. Another did not. A smaller model could enter repeated tool searches or fail to synthesize after retrieval. The application must support provider differences and use traces to distinguish model limitations from code defects.

Local document processing competes for the same hardware. Docling, embedding models, rerankers, and generation models can use CPU, RAM, and GPU simultaneously. A batch ingestion job may be stable on a dedicated server but disruptive on a laptop running other memory-heavy applications. Concurrency limits and separate services are therefore operational controls, not only performance settings.

Cloud models remain useful when local hardware is unavailable or when a larger model is needed for difficult orchestration. A hybrid architecture can keep documents and embeddings local while calling a cloud model with retrieved evidence, or it can operate entirely locally for an air-gapped deployment. The security and privacy decision determines which path is acceptable.

Security review must occur before deployment. The system should be treated as unsafe until each boundary is verified.

Required checks include:

• Row-level security is enabled on every user-controlled table.
• Storage policies isolate user folders.
• Shared tables have explicit authenticated-access policies.
• Administrative settings are restricted to administrators.
• API keys are not readable by ordinary users.
• Service-role credentials are never sent to the browser.
• Environment files are excluded from version control.
• The SQL agent uses a restricted database identity.
• Destructive SQL and unrelated-table access are rejected.
• Subagents can access only permitted documents.

One final inspection found that the sales-data table was unrestricted. The intention was that all signed-in users could read it, but unrestricted access could make it available more broadly. The correct design is to enable row-level security and create an explicit policy for authenticated read access. This is a typical alpha-stage defect: the feature works, but the security policy is not yet production-grade.

Deployment requires a formal release process. Pushing directly to the main branch and immediately updating production is not sufficient. A mature flow has development, staging, and production environments.

A controlled release should move changes through development, staging, and production in an orderly sequence. Database migrations must be applied deliberately at each stage, and the application must be validated before promotion. If a release fails, the team needs a way to roll back the application and address the related database changes rather than continuing to push directly to the main environment.

Operational limits should be documented:

• Maximum file size.
• Supported formats.
• Maximum ingestion concurrency.
• Model context limits.
• GPU and memory requirements.
• Tool-round limits.

Retrieval quality also needs broader evaluation than the initial smoke tests. Testing should cover exact identifiers, similar documents, metadata failures, SQL calculations, follow-up questions, and whole-document tasks so that changes to the retrieval pipeline do not silently break earlier behavior.

The appropriate maturity label for the completed build is alpha. It contains an end-to-end multi-user RAG application with authentication, chat, ingestion, multi-format parsing, embeddings, metadata, deduplication, hybrid search, reranking, SQL, web search, subagents, local-model support, and observability. It has also passed smoke tests. It has not completed the full security, scale, regression, release, and retrieval-evaluation work required for production.

Calling it alpha is an analytical judgment, not a criticism of the feature set. The application proves the architecture and major flows. Production readiness requires a different standard: repeatable tests, secure defaults, controlled deployment, documented limits, and evidence that the system remains correct under realistic use.

The integrated operating model is now clear.

For ingestion, the authenticated user uploads a document. The system stores it in a user-specific path, hashes it, parses it, extracts metadata, chunks it, embeds it, and writes it to pgvector. Duplicate content is skipped. Changed content replaces old chunks. Status and errors are visible.

For retrieval, the main agent interprets the question and selects a strategy. It can run hybrid document search, apply justified metadata, rerank candidates, query structured data through a read-only role, search externally, or delegate a complete document to a subagent. Tool rounds are limited. A final synthesis is guaranteed.

For interaction, the browser streams text, tool calls, reasoning panels, subagent activity, and errors in chronological order. The event history persists. The user can stop the process and return to old threads.

For control, the database enforces user isolation and tool permissions. Administrators manage global models and providers. Embedding changes are blocked while vectors exist. Secrets remain outside ordinary user access.

For operations, traces expose model and tool behavior, logs expose internal failures, tests protect stable contracts, context is managed deliberately, plans and progress preserve project memory, and version control provides recovery points.

Mastering RAG means mastering this complete evidence pathway. The final answer is only the visible surface. Reliability comes from the hidden system that decides what can be accessed, how information is represented, which retrieval method is used, what evidence reaches the model, how the interaction is observed, and whether the entire process has been validated.