Turn Raw Data Into Searchable, Structured Memory

A memory layer begins with ingestion, but ingestion is more than copying documents into storage. The raw data must be registered, transformed, and connected into a representation that agents can search.

One workflow starts with an add operation. Adding registers the source data with the system. The data may come from relational records, unstructured text, support tickets, contracts, billing records, invoices, PDFs, operational databases, or other source systems. The sources do not need to share the same format because the memory layer is responsible for producing a common representation.

The next step is a transformation operation called cognify. Cognify converts the registered data into a combined graph-vector representation. A language model is used during this process to extract entities, concepts, entity types, and relationships. The result is not only a set of embedded chunks, but a knowledge graph connected to those embeddings.

A higher-level remember operation provides a simple way to add text to memory. Text is passed to the memory system, and the configured ingestion layer uses a model to extract the relevant structure. The extraction behavior can be adjusted. In a production system, the prompts, schemas, and rules used to identify entities and relationships can be tuned for the domain rather than left at their default settings.

This tunability is important. A general-purpose extraction process may identify broad topics, while a domain-specific system may need to distinguish contract terms from billing events, entitlement flags from account states, or scientific entities from possible future states. The structure of the graph determines what relationships can later be traversed, so ingestion quality directly affects retrieval quality.

The graph can grow incrementally. A first document may introduce a concept. A second document may connect a new entity to that concept. A third may add a regulation, event, or dependency. In an agent or chat application, new knowledge can be processed as it arrives, allowing the graph to evolve rather than being rebuilt as a one-time static artifact.

After transformation, the data should be checked rather than assumed to be correct. A search can confirm that the expected information is retrievable. A graph visualization can show which entities were extracted and how they were linked. These checks help reveal missing concepts, incorrect entity types, isolated nodes, or relationships that were not captured.

The language model is part of the ingestion pipeline, so model configuration matters operationally. A hosted model can be used, or a local model endpoint can be supplied. If the selected setup requires an API key, ingestion will fail when that key is missing because the model is needed to perform extraction. Different models can also produce different entity and relationship structures, so the chosen model is not merely a response generator; it participates in memory construction.

For larger ingestion workloads, processing can be parallelized when the model and infrastructure allow it. The main constraint is often the language-model call rather than the database write. The ingestion architecture should therefore be designed with the cost and latency of extraction in mind.

The output of this lesson is a memory representation that contains four linked elements:

1. The original source and its provenance.
2. Vector embeddings for semantic retrieval.
3. Entities, types, and relationships for graph reasoning.
4. A structure that can be extended when new information arrives.

That combination is what turns raw data into agent memory rather than a passive document index.