"Long-horizon" agents: one word, several different properties
Oct 25, 2026
12 min read
3 views
Glasp Research note. Sources read 2026-10-25 and 2026-10-26.
Disclosure. This note is published by Glasp. Kazuki Nakayashiki, a Glasp team member, is the sole author of the two papers discussed in the section "Inherited memory that is no longer valid". No independent replication of either paper was identified as of October 2026.
"Long-horizon" is used for agents that run overnight, for models with very large context windows, for benchmarks whose tasks take a person many hours, and for memory systems that carry information across sessions. These are different properties. They are measured by different benchmarks, they fail in different ways, and a result on one says little about another. This note separates them, places the main public evidence against each, and then describes where the two Glasp papers fit, alongside closely related work by others.
Six properties that get bundled together
Task size, measured in human time. How much skilled human work the task needs. METR measures this directly: the task length, in human time, at which an agent succeeds half the time. Its paper reported that this length doubled roughly every seven months from 2019 [1]. This is a property of the task, not of how long the agent runs.
Step count. How many actions or tool calls one trajectory needs, so that small per-step errors can compound. OSWorld (369 computer tasks) reported humans above 72% and the best model at release near 12% [2]. tau-bench added pass^k, which asks whether an agent succeeds on every one of k repeated tries; GPT-4o-class agents solved under half of tasks, and retail pass^8 fell below 25% [3].
Context length. How many tokens the model must read or keep. RULER (v3) evaluated 17 long-context models that all claim 32K tokens or more and found only about half kept satisfactory performance at 32K [4]. A large window and effective use of it are different things.
Execution duration. Elapsed wall-clock time while the agent is working.
Persistence across interruptions. Whether work and state survive restarts, crashes, handoffs, or a new session.
State freshness. Whether information the agent carries, or inherits from an earlier session, is still valid, or has been superseded by a newer record.
These properties are correlated but not interchangeable. A one-million-token window (3) does not make a task smaller (1). For example (an illustrative case, not a measurement), a task measured as eight hours of human work might take an agent much less wall-clock time (4). And a fact can go stale (6) without any long execution at all: it only needs the world to change between the moment it was written and the moment it is used.
Several well-known benchmarks are easy to file under the wrong property:
Vending-Bench has an agent run a simulated vending business over long trajectories (more than 20 million tokens per run). All tested models had runs that derailed, and the authors found no clear correlation between failures and the point at which the context window became full [5]. This is evidence about long trajectories (2, 3), not about real hours of execution or recovery from restarts.
LongMemEval tests memory over simulated multi-session chat histories, including a "knowledge updates" ability in which an old fact has been replaced by a newer one. It reported a roughly 30% accuracy drop for commercial assistants and long-context models [6]. This bears on persistence across sessions (5, simulated) and freshness (6), not on durable execution over real time.
Durable execution and sandbox products (Temporal, E2B, Modal, Daytona and others) address (4) and (5). The available descriptions of suspend, resume, and session limits come from the vendors themselves.
What vendors describe shipping
Anthropic's context-engineering guide (2025-09-29) describes compaction (summarize and continue), clearing old tool output, and notes kept outside the context window, alongside a memory tool [7]. OpenAI's Codex CLI added persisted /goal workflows in release 0.128.0 (2026-04-30), which keep an agent working on an objective across turns, and made them on by default and no longer experimental in 0.133.0 (2026-05-21) [8]. These are the vendors' own descriptions of features. Public evidence that agents reliably run for long wall-clock periods is, in the sources reviewed here, mostly vendor-published demonstrations and customer statements rather than measurements.
How strong is the evidence?
The judgments below are selection judgments over the sources reviewed here, not established findings. The criteria are: whether the evidence comes from a party other than the vendor of the system being measured, whether data or tasks are public, and whether a result has been audited or replicated.
Benchmark audits provide the clearest evidence, and they cut against the benchmarks themselves. SWE-bench Verified, a widely used human-validated subset of 500 SWE-bench tasks, was audited by OpenAI. According to Epoch AI's review (2026-09-03), OpenAI audited 27.6% of the tasks, a selected subset rather than a random sample, and found that 59.4% of those had flawed test cases that rejected functionally correct submissions; Epoch rates the benchmark "Flawed" [9]. The 59.4% applies to the audited subset, not to all 500 tasks.
OpenAI stopped reporting SWE-bench Verified in February 2026, citing flawed tests and contamination [24].
SWE-bench Pro was introduced by Scale AI (v1 2025-09-21, v2 2025-11-14): 1,865 problems from 41 repositories, split into public, held-out and commercial sets. Scale designed it to reduce contamination by keeping the held-out and commercial (proprietary) problems out of public reach, and describes it as "contamination-resistant" [10]. The v1 abstract reported a best score of 23.3% (GPT-5); the v2 abstract gives no score.
On July 8, 2026, OpenAI reported an audit of SWE-bench Pro that found roughly 30% of its tasks broken, and retracted its earlier recommendation to adopt it [25]. Both Verified and Pro, then, later received material audits.
The lesson is not that audited benchmarks can be trusted. It is that every benchmark score is provisional until its tasks have been audited, and audits can arrive years after release.
METR's time-horizon series is a widely cited measurement of task size, with caveats from METR itself. METR's Time Horizon 1.1 update (2026-01-29) moved to a larger task suite and gave shorter doubling times than the original paper: 131 days for post-2023 models and 89 days since 2024 [11]. METR also states that "measurements above 16 hrs are unreliable with our current task suite" [12]. The trend direction is better supported than any single model's hour figure.
Memory benchmarks show weaknesses on their tested tasks; vendor comparisons conflict. Letta reports 74.0% on the LoCoMo benchmark by storing conversation history in files, and argues that current memory benchmarks may not be very meaningful [13]. Mem0's paper reports a 26% relative improvement over OpenAI's memory on LoCoMo under an LLM-as-judge metric, and 91% lower p95 latency than a full-context method [14]. Both are vendor-run. LongMemEval, which is not tied to either vendor, shows substantial weaknesses on its long-term-memory tasks [6]. No controlled head-to-head comparison of Letta and Mem0 was found.
Compaction, memory tools, and durable execution rest mostly on vendor descriptions. No independent controlled study of compaction's effect on task success was found. That is a statement about the evidence located, not a finding that compaction does not work. The cited deployment claims lack independent controlled validation.
Training for compact state has a narrow result. MEM1 (ICLR 2026) trains an agent with reinforcement learning to carry one compact internal state instead of its full history. Its abstract reports that MEM1-7B achieved 3.5 times the performance with 3.7 times less memory than Qwen2.5-14B-Instruct on a constructed 16-objective multi-hop question-answering task [15]. The comparison is a trained 7B method against a larger untrained instruction model on a constructed task; it is not evidence of a general 3.5x improvement for long-horizon agents. The wider literature on reinforcement learning for long agent tasks was not surveyed here.
Different failures, no shared mechanism claimed. These benchmarks expose several failure modes that context length alone does not explain: trajectory derailment (Vending-Bench), trial-to-trial inconsistency (tau-bench), and failure to apply updated information (LongMemEval's knowledge updates). They are different failures, measured differently, and nothing in the cited work shows a common cause. The last of them, state freshness, is the subject of the rest of this note.
Inherited memory that is no longer valid
An agent often starts from notes, summaries or instruction files written by an earlier session. If a constraint recorded in that memory has since been withdrawn by a newer record, and nothing in the memory says so, the question is whether the agent checks before acting.
Related work
A growing 2026 literature studies outdated memory in agents. The following are the most directly relevant papers found in a targeted search of arXiv; the list is not exhaustive.
STALE (Chao et al., v1 2026-05-07) introduces 400 expert-validated scenarios of "implicit conflict", where a later observation invalidates an earlier memory without explicitly negating it, probed along state resolution, premise resistance and implicit policy adaptation. It reports a gap between retrieving updated information and acting on it, with the best model at 55.2% overall [16].
When Memory Updates but Behavior Does Not (Sun and He, v1 2026-08-03) addresses agents that recognize stored state is outdated but still plan around the old value, and evaluates a provenance-and-chronology audit on STALE [17].
Supersede (Patel, v1 2026-06-25) uses LongMemEval's knowledge-update subset and reports that a bounded, self-maintained memory trails full context (about 77% against 92% with one model), and releases a reinforcement-learning environment for supersession [18].
Temporal Validity in Retrieval Memory (Yadav, v1 2026-06-25) shows that embedding similarity barely separates contradicted facts from duplicates and proposes a deterministic supersession ledger [19].
TEPA (Zhou et al., v2 2026-08-10) proposes revoking superseded memories under the same key while keeping them for audit [20].
The Memory Trust Gap (Hu and Ramachandran, v1 2026-09-01) studies when models trust a stale stored fact over a tool that always holds the current value, and how this changes with model size [21].
LongMemEval (2024) includes knowledge updates as one of five tested abilities [6].
Most of this work studies either whether the current value is retrieved and used, or how memory should be revised when new evidence is written. The Memory Trust Gap is closest in spirit, since an authoritative source is available to consult.
The two Glasp papers
The two papers by Kazuki Nakayashiki study a narrower question: how an agent allocates a fixed verification budget when inherited memory carries explicit links to source records, some of which have been withdrawn. This is cross-session state validity under constrained verification. The memories are a few lines long and the sessions are simulated; the papers are not evidence about long wall-clock execution, restarts, or context length. This note makes no claim that the setup is novel; a fuller comparison with the work above has not been done.
When Stale Constraints Go Unchecked (arXiv 2608.25553, figures from v3, 2026-08-28) [22]. An agent inherits a six-memory store in which one constraint's source record has been withdrawn by a newer record, and may check two records. Agents made decisions consistent with the stale constraint in 77.3%, 74.7% and 74.7% of episodes (primary run, replication, held-out domain). When the constraint was stated in memory, agents inspected its provenance path in 20.1% and 23.1% of episodes (primary run, replication); with the constraint removed from the same memory, 66.9% and 72.9%. Moving one verification slot to the critical record raised current-record-consistent decisions by 74.0, 72.7 and 61.3 points, but this relies on the experimenter knowing which record matters, and the paper states it is not a deployable policy. The paper lists its limits: six-memory stores, two scripted domains, one system prompt, schema and archive format, and nothing about how often constraints go stale in deployment.
Plan Pointers and Record-Directive Form (arXiv 2609.03450) [23] asks how the wording of a directive in memory (a bare record ID, a criterion identifying the record, or both) changes which record the agent requests. In v1 (2026-09-03, twelve registered studies, 14,760 attempts), a length-matched criterion beat a bare ID by 35.0 points on six direct-provider models but failed its registered superiority rule on a nine-model panel, and a one-character plan pointer shifted requests by 78.0 points (81.7 in a registered re-run). The current v2 (2026-09-21, sixteen registered studies, 179,352 attempts) repeats the six-model and nine-model result in its abstract without point figures and does not restate the plan-pointer figure there. Figures should be cited with the version.
Takeaways
Say which property "long-horizon" refers to: task size, step count, context length, execution duration, persistence, or state freshness.
Treat every benchmark as provisional. Audits of widely used coding benchmarks have found material problems after release.
Read memory, compaction and durable-execution claims as vendor descriptions until a controlled comparison exists.
Treat inherited state as possibly stale, and give agents a way, and some budget, to check the records it depends on.
Scope
Sources were primary papers and official pages, read on 2026-10-25 and 2026-10-26; each is listed in the references below with its date. Not covered: the agent product claims of Google DeepMind, xAI and Manus; the literature on reinforcement learning for coding agents; the self-correction literature; WebArena, MLE-bench, PaperBench and TheAgentCompany.
References
[1] METR (Kwa et al.), Measuring AI Ability to Complete Long Software Tasks, arXiv 2503.14499, v1 2025-03-18, v4 2026-07-10. https://arxiv.org/abs/2503.14499v4
[2] Xie et al., OSWorld, arXiv 2404.07972, 2024-04-11. https://arxiv.org/abs/2404.07972
[3] Yao et al. (Sierra), tau-bench, arXiv 2406.12045, 2024-06-17. https://arxiv.org/abs/2406.12045
[4] Hsieh et al. (NVIDIA), RULER: What's the Real Context Size of Your Long-Context Language Models?, arXiv 2404.06654 v3, 2024-08-06. https://arxiv.org/abs/2404.06654v3
[5] Backlund and Petersson (Andon Labs), Vending-Bench, arXiv 2502.15840, 2025-02-20. https://arxiv.org/abs/2502.15840
[6] Wu et al., LongMemEval, arXiv 2410.10813, 2024-10-14 (ICLR 2025). https://arxiv.org/abs/2410.10813
[7] Anthropic, Effective context engineering for AI agents, 2025-09-29 (vendor). https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
[8] OpenAI, Codex CLI release notes 0.128.0 (2026-04-30) and 0.133.0 (2026-05-21) (vendor). https://github.com/openai/codex/releases/tag/rust-v0.133.0
[9] Epoch AI, SWE-bench Verified review, reviewed 2026-09-03. https://epoch.ai/benchmarks/swe-bench-verified/review
[10] Deng et al. (Scale AI), SWE-Bench Pro, arXiv 2509.16941, v1 2025-09-21, v2 2025-11-14. https://arxiv.org/abs/2509.16941v2
[11] METR, Time Horizon 1.1, 2026-01-29. https://metr.substack.com/p/2026-1-29-time-horizon-1-1
[12] METR, Time horizons page (states it is no longer actively updated). https://metr.org/time-horizons/
[13] Letta, Benchmarking AI Agent Memory: Is a Filesystem All You Need?, 2025-08-12 (vendor). https://www.letta.com/blog/benchmarking-ai-agent-memory
[14] Chhikara et al. (Mem0), Mem0, arXiv 2504.19413, 2025-04-28 (vendor-authored). https://arxiv.org/abs/2504.19413
[15] Zhou et al., MEM1, arXiv 2506.15841, 2025-06-18 (ICLR 2026). https://arxiv.org/abs/2506.15841
[16] Chao, Bai, Sheng, Li and Sun, STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?, arXiv 2605.06527 v1, 2026-05-07. https://arxiv.org/abs/2605.06527
[17] Sun and He, When Memory Updates but Behavior Does Not: Repairing Implicit Stale Dependencies in Personalized Agent Responses, arXiv 2608.01619 v1, 2026-08-03. https://arxiv.org/abs/2608.01619
[18] Patel, Supersede: Diagnosing and Training the Memory-Update Gap in LLM Agents, arXiv 2606.27472 v1, 2026-06-25. https://arxiv.org/abs/2606.27472
[19] Yadav, Temporal Validity in Retrieval Memory: Eliminating Stale-Fact Errors for AI Agents over Evolving Knowledge, arXiv 2606.26511 v1, 2026-06-25. https://arxiv.org/abs/2606.26511
[20] Zhou, Ouyang, Zheng and Xiang, TEPA: Revoking Stale Memories for Conflict-Robust Language Agents, arXiv 2608.07429 v2, 2026-08-10. https://arxiv.org/abs/2608.07429
[21] Hu and Ramachandran, The Memory Trust Gap: Capability-Dependent Failures in Persistent-Memory Agents, arXiv 2609.01852 v1, 2026-09-01 (workshop submission under review). https://arxiv.org/abs/2609.01852
[22] Nakayashiki, When Stale Constraints Go Unchecked: Budgeted Verification Failures in Inherited Agent Memory, arXiv 2608.25553 v3, 2026-08-28. https://arxiv.org/abs/2608.25553v3
[23] Nakayashiki, Plan Pointers and Record-Directive Form in Budgeted Verification of Inherited Agent Memory, arXiv 2609.03450, v1 2026-09-03, v2 2026-09-21. https://arxiv.org/abs/2609.03450v1 https://arxiv.org/abs/2609.03450v2
[24] OpenAI, post on why it no longer reports SWE-bench Verified, 2026-02-23 (vendor). https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/
[25] OpenAI, Separating signal from noise in coding evaluations, 2026-07-08 (vendor; audit of another party's benchmark). https://openai.com/index/separating-signal-from-noise-coding-evaluations/