Why we store agent memories in a knowledge graph, not a vector database

What vector databases get wrong about memory

Vector databases model the world as a bag of similar things. You embed a query, find the nearest neighbors, and hope the answer is in there somewhere. This works for document search. It breaks for agent memory because memory isn’t a similarity problem—it’s a relationship problem.

Consider a simple question an agent needs to answer: “Who worked on the billing rewrite last quarter?” A vector search might return chunks mentioning “billing” and “Q1” and “engineer.” But it won’t tell you that Alice led the project, Bob reviewed every PR, and the integration tests live in a specific repo. Those connections exist as implicit relationships in the text—but a vector database can’t traverse them.

The result is an agent that can retrieve documents but can’t reason across them. It can find the billing spec and find Alice’s commit log, but it can’t connect them without consuming both in context and hoping the LLM infers the relationship. This gets expensive and unreliable fast.

Knowledge graphs: memory with structure

A knowledge graph stores entities and their relationships explicitly. Instead of “here are 20 chunks that might be relevant,” the system answers: “Alice (person) worked_on billing-rewrite (project) in Q1-2026 (time_period), reviewed_by Bob (person), documented_in billing-spec.md (document).”

When an agent queries for context, the graph traversal returns structured facts—not raw text chunks. The agent knows Alice led it, Bob reviewed it, and there’s a spec document. It can then decide which details to pull in rather than dumping a pile of similarity-ranked chunks into context.

The key difference: vector search gives you proximity. Graph traversal gives you relationships. For agent memory, relationships matter more than proximity.

Why Postgres, not a graph DB

If you need a graph, why not use Neo4j or ArangoDB? Two reasons:

1. Postgres is source of truth for everything else. Your users, workspaces, billing, and API keys already live in Postgres. Adding a separate graph database means a second source of truth, a second consistency model, and a distributed transaction problem you don’t want. Our graph tables (hyobjects, relations, canon_relations, evidence) live in the same database, within the same transactions, under the same RLS policies.
2. pgvector handles the embedding layer. We use pgvector for similarity search on chunks when retrieval needs it. But the embeddings are one retrieval signal among many—not the primary data model. The graph is the truth. Embeddings are an index.

This architecture gives you the best of both: vector similarity for initial retrieval, graph traversal for relationship-aware context assembly, all within a single database with a single consistency model.

The provenance advantage

Vector databases store embeddings as opaque floating-point arrays. If a retrieved result is wrong, you can’t easily answer why. Was the input bad? The embedding model? The similarity threshold? You have to debug by feel.

With a knowledge graph, every fact carries a provenance chain. You can call brain.why on any node and trace it back to the source document, the extraction run, and the confidence score that promoted it to durable memory. If a fact is wrong, you know exactly how it got there—and you can fix the pipeline rather than re-embedding your corpus.

This matters for production systems. When your agent gives a client incorrect information, the first question is “where did that come from?” A vector search can’t answer that. A provenance graph can.

Schema as a forcing function

The less obvious benefit of a relational knowledge graph: schema forces discipline. Our core tables—hyobjects, chunks, relations, canon_relations, evidence—enforce that every fact has a type, every relationship has a predicate, and every claim has evidence.

An LLM can’t just dump unstructured text into a vector store and call it a memory. It has to propose structured facts that conform to the schema. If the proposal doesn’t meet the schema contract, it doesn’t get written. This constraint eliminates the “bag of loosely-related embeddings” problem at the architectural level.

We measured this: schematized ingestion reduces malformed extractions from ~25% of all writes to less than 1%. The schema is the gatekeeper, and the LLM learns to play within its rules.

When you should still use vectors

Vectors aren’t wrong. They’re wrong as the primary memory model for agents. We still use pgvector heavily for:

• Initial chunk retrieval — find relevant source material before graph resolution
• Memory recall — brain.recall_memory uses embeddings for fast candidate retrieval, then graph traversal for precise context assembly
• Context packing — brain.context_pack blends vector results with graph results into a single optimized response

The difference is that vectors are one signal in a multi-signal system. They don’t run the show. The graph does.