Context Lineage for Evolving AI Agents: A Formal Framework for Versioned Knowledge Persistence

The emergence of AI agents as primary information intermediaries represents a fundamental shift in how knowledge is consumed. Unlike search engines, which index and link to source material, LLMs internalize information during training, creating compressed representations that are inherently lossy [1]. When an LLM is retrained or updated, previously-learned context about specific entities can be overwritten, diluted, or lost entirely -- a phenomenon we term context decay.

The llms.txt standard [2] was proposed as a human- and machine-readable file format for providing context about a website or data source to LLMs. However, the current specification treats context as a static snapshot. This paper argues that context must be treated as a versioned, living artifact with full provenance lineage -- analogous to how Git tracks source code history or how data engineering pipelines maintain data lineage.

We make the following contributions:

1. A formal model of context decay quantifying information loss across LLM generations (Section 2)
2. A mathematical framework for context lineage using DAGs with temporal annotations (Section 3-4)
3. A composable, information-theoretic scoring function for LLM readability (Section 5)
4. An open system (AgentIndexc) implementing this framework across 1,400+ data sources (Section 6)
5. Empirical validation showing 3.2x citation improvement for high-LFS entities (Section 7)

Consider an LLM trained on corpus at time . The model encodes a compressed representation of each entity in the training data. Let denote the model's internal representation of entity at time , and let denote the ground-truth state of at time .

Under reasonable assumptions (sub-Gaussian innovations in entity state), we can bound the expected decay:

This theorem has a critical practical implication: the information an LLM holds about any entity degrades at a rate proportional to the square root of the time since its training cut. For fast-moving entities (high ), such as startups, financial APIs, or actively-developed open source projects, the decay is rapid. This motivates the need for a persistent, versioned context layer that exists outside the model.

We model the space of context documents as a normed vector space where each document maps to a point in a high-dimensional semantic embedding. Let denote the set of all context documents (llms.txt files, API specs, MCP configs, etc.).

The distance between two context states captures how much an entity's machine-readable representation has changed:

The core innovation of this work is the context lineage model, which tracks the full provenance chain of context documents through time.

The lineage graph enables two critical operations:

We define the LLM Friendliness Score (LFS) as a weighted composition of five sub-scores, each grounded in measurable properties of the context document and its source.

AgentIndexc implements the above framework as a production system processing 1,400+ data sources across four categories:

We evaluate the framework across the AgentIndexc corpus of 1,412 indexed entities.

llms.txt Specification. The llms.txt standard [2] defines a Markdown-based file format for providing website context to LLMs. Our work extends this by adding versioning semantics and a formal quality metric.

Retrieval-Augmented Generation (RAG). RAG systems [4] address knowledge currency by retrieving external documents at inference time. Our approach is complementary: rather than relying on the user to provide retrieval infrastructure, we ensure the source documents themselves are optimized for LLM consumption.

Data Lineage. Data lineage tracking in data engineering [5, 6] maintains provenance graphs for datasets flowing through ETL pipelines. We adapt this concept to the domain of machine-readable context documents.

Model Context Protocol (MCP). Anthropic's MCP [7] provides a protocol for AI agents to dynamically connect to external tools and data. AgentIndexc publishes MCP-compatible discovery endpoints, and our directory includes indexed MCP servers, bridging the static (llms.txt) and dynamic (MCP) approaches.

Knowledge Graphs. Systems like Wikidata [8] and Google Knowledge Graph maintain structured entity representations. Our framework operates at a different level of abstraction -- we optimize the source documents rather than the knowledge graph itself, making the approach compatible with any downstream consumer.

We have presented a formal framework for understanding and combating context decay in AI agent systems. The key insight is that as AI agents evolve through retraining cycles, the entities they represent need a persistent, versioned context layer that transcends any single model generation. Our context lineage model provides the mathematical foundation for this layer, and our LLM Friendliness Score gives practitioners an actionable metric for optimizing their data sources.

AgentIndexc implements this framework at scale, maintaining versioned llms.txt files for over 1,400 data sources. Our empirical results demonstrate that the framework delivers measurable improvements in AI agent recall and citation accuracy. As the number of AI agents and their retraining frequency increases, the value of maintained context lineage will only grow.

We release the LFS scoring algorithm and context lineage specification as open standards, and invite the community to contribute to the indexed corpus through the AgentIndexc platform.