Maintaining Knowledge Quality - Deduplication, Metadata, Parsing, and Performance

A retrieval system becomes noisy and expensive when the knowledge base is not actively managed. Uploading files is not enough. The system must recognize duplicates, handle changed documents, extract useful metadata, support multiple formats, and control the resource cost of processing.

The record manager solves the duplicate-ingestion problem. Without it, uploading the same file twice creates two document records and two sets of nearly identical chunks. Retrieval can then return repeated passages, crowding out other evidence and making the ranking scores look stronger than they should.

A content hash provides a stable comparison. When a file arrives, the system calculates a hash from its content and compares it with the stored value for the corresponding record.

The decision logic is:

• If no matching record exists, ingest the file as new.
• If the content hash is identical, skip processing and mark the file unchanged.
• If a logical record exists but the content hash changed, replace the old processed content.

There are two possible strategies for changed files. One strategy compares old and new chunks and re-embeds only the changed portions. This can reduce model calls, but it introduces difficult edge cases. A small edit may shift chunk boundaries. Removed text can leave orphan chunks. The record manager must determine which old chunks no longer correspond to the new version.

The chosen strategy was simpler and safer: delete all old chunks for the document and process the new content from the beginning. This uses more embedding work, but it avoids stale or orphaned evidence. For many business document collections, correctness and operational clarity are more important than saving a small number of embedding calls.

The hash should be stored on the document record. The interface can report "document unchanged, skipped processing" rather than creating a second record. A first upload may need to backfill the hash if the record-manager feature was added after documents already existed.

Metadata adds a second layer of knowledge quality. Vector similarity retrieves passages with related meaning, but users often need to narrow the search by file type, topic, language, product, date, or another structured attribute.

Metadata extraction can run during ingestion. A language model receives a bounded portion of the document, such as the first 8,000 characters, and produces a structured object. Limiting the input controls latency and model cost for very large files. The extracted object can include:

• A generated title.
• A concise summary.
• Document type.
• Topics.
• Language.
• Additional administrator-defined fields.

The metadata should be validated with a schema before it is stored. A Pydantic-style schema can define required fields, optional fields, arrays, text values, numbers, booleans, and other supported types. The metadata is stored on the document and propagated to its chunks so retrieval can filter without joining every result back to the parent record.

Hardcoding one metadata schema limits the application. A more flexible design stores the schema as JSON in global settings and provides an administrative interface for changing the field definitions. This allows one deployment to extract product categories and another to extract departments, legal jurisdictions, or publication types without changing the ingestion code.

Dynamic metadata creates trade-offs. A wider schema increases extraction cost and creates more opportunities for inconsistent values. An optional field may be absent. A model may invent a category that was not intended. The retrieval prompt must know which fields exist and how they should be used. Configuration therefore needs validation and clear field descriptions.

Metadata filtering is powerful but dangerous. A filter is a hard exclusion, not a soft preference. If the agent selects the wrong value, every correct chunk can be removed before vector similarity or reranking has a chance to help.

A specific failure shows the risk. The user asked for material related to Samsung refrigerators and the topic of smart-home integration. The agent correctly selected the topic value, but it also chose the document type "article." The actual document was labeled "reference." The combined filter returned zero results even though the knowledge base contained the right material.

The lesson is to be conservative. The system prompt should explain when each metadata field is reliable. Filters should be applied only when the user clearly supplied the constraint or when an earlier retrieval step established it. A broad search should run without filters when the agent is uncertain. If a filtered search returns nothing, the agent can retry with fewer constraints.

The filter behavior should be visible in traces. An operator should be able to see the query, selected metadata fields, returned count, and scores. Without that view, a zero-result answer may look like a missing document rather than an overrestrictive tool call.

Multi-format support belongs in a dedicated parsing layer. Docling can process PDFs and office formats rather than relying on plain text and Markdown. Its standard pipeline converts supported files into structured text. A vision-language pipeline can assist with visually complex content when suitable hardware or a cloud model is available.

This introduces substantial dependencies. The parser may install PyTorch and download machine-learning models on first use. The dependency size was estimated at roughly two gigabytes in the implementation plan. Local parsing can use CPU, RAM, and GPU resources even before embeddings and metadata extraction begin.

A practical parser configuration includes:

• A supported-format list in the front end and back end.
• A maximum file size, such as 50 megabytes.
• A standard parsing path.
• An optional vision-language path.
• Clear failure states when a file cannot be parsed.

PDF manuals provide a realistic test because they contain multiple pages, headings, model identifiers, installation instructions, warnings, and tables. A parsed manual can produce several chunks, receive generated metadata, and become searchable by product number and topic. The purpose of the test is not only to get text out of the PDF. It is to verify that the text remains useful for retrieval.

Batch ingestion exposes performance problems that single-file testing hides. Uploading a group of appliance manuals initially processed files one at a time and made the system feel slow. An early background-task implementation had no concurrency limit, which created the opposite risk: the server could start too many parsing jobs and exhaust the machine.

The improved design uses a bounded queue. Files enter a pending state. A fixed number move into processing. Completed jobs release capacity for the next files. An initial maximum of three concurrent ingestions made the behavior stable and visible. Increasing the number to ten improved throughput somewhat, but it did not solve the underlying bottleneck.

Resource observation provided more information. During one batch, system memory reached about 92 percent, and Python processes used roughly eight gigabytes. Closing an unrelated memory-heavy application reduced pressure. GPU activity remained low, suggesting that the active bottleneck was not the language model. Possible bottlenecks included Docling parsing, metadata calls, embedding calls, database writes, repeated file loading, or serialized code.

Code inspection found that files were being loaded into memory more than once and that background tasks could be created without a safe bound. Removing duplicate loads and adding concurrency control made the pipeline leaner.

Increasing concurrency is not automatically an optimization. More parallel jobs can increase CPU and memory pressure without removing the actual bottleneck. The concurrency limit should be changed only after observing how parsing, model calls, and database work behave.

Performance tuning should therefore answer a sequence of questions:

1. How much time is spent in storage upload?
2. How much time is spent parsing?
3. How much time is spent extracting metadata?
4. How much time is spent generating embeddings?
5. How much time is spent writing records?
6. Which stage uses the most CPU, RAM, GPU, or network capacity?
7. Does increasing concurrency improve total throughput or only increase pressure?

Batch tests should use realistic documents, not only tiny fixtures. The pending and processing states should make the controlled queue visible while resource use is inspected.

Local parsing can be moved to a more capable server if the application host is constrained. Cloud parsing or OCR services are alternatives, but they introduce provider cost and data-handling considerations. The architecture should keep the parser behind an interface so the implementation can change without rewriting the rest of ingestion.

Practical knowledge-maintenance rules:

• Hash every file before expensive processing.
• Skip identical content.
• Prefer full replacement when incremental chunk comparison could leave stale evidence.
• Validate metadata against an explicit schema.
• Keep metadata fields configurable but controlled.
• Apply filters conservatively and retry without them when appropriate.
• Treat parsing as a resource-intensive service.
• Bound concurrency and measure every stage.
• Test with batches of realistic documents.
• Optimize the actual bottleneck rather than increasing parallelism blindly.