Zero-LLM-Cost Event-Sourced Tool Failure Memory for Desktop Agents
面向桌面代理的零 LLM 成本事件驱动型工具故障记忆系统
Abstract
Desktop LLM agents repeatedly encounter the same low-level tool failures—API timeouts, encoding errors, permission denials, and tool-specific quirks—yet current frameworks either discard these lessons at session end or pay a recurring price to extract them: reflection-based learners (ERL, ExpeL) require additional LLM calls per task, while general memory layers (Mem0) assume embedding infrastructure and perform poorly on Chinese tool-error text. We present EBEAC (Event-Based Experience Auto-Capture), a zero-additional-LLM-cost mechanism that harvests tool-level trial-and-error knowledge directly from the runtime event stream. EBEAC detects “consecutive failure → retry success” sequences via the EventBus, abstracts them into structured experiences (trigger, diagnosis, fix) with a keyword template generator, and retrieves them through a three-layer filter (outcome matching, pattern deduplication, time-decay weighting) with a 2000-character injection budget. On a production-log replay of 196 real experiences from ten weeks of live usage (chronological 137/59 train/test split), EBEAC achieves 71.19% recall@1 and 89.36% precision at 0.97 ms retrieval latency with only 3.39% false positives—recall@3 and the cross-session prevention rate both reach 76.27%—a precision-centric alternative to lexical baselines that reach 83.05% recall at a five-fold higher false-positive rate (16.95%). A capture-side analysis shows the 196 experiences collapse into only 23 abstract patterns (88.3% collapse ratio), and 16.9% of test failures involve patterns never seen in training; the residual recall gap is therefore dominated by capture-side coverage rather than retrieval-side quality (91.8% of covered events are retrieved). EBEAC is positioned as a cost-efficient niche memory mechanism for operational, high-frequency tool quirks, complementary to—not competing with—strategic reflection-based learning.
摘要
桌面端大语言模型代理频繁遭遇相同的底层工具故障——API 超时、编码错误、权限拒绝以及工具特有的异常现象——然而现有框架要么在会话结束时丢弃这些经验，要么为提取这些经验付出持续代价：基于反思的学习器（ERL 、ExpeL）每个任务需要额外调用大语言模型，而通用记忆层（Mem0）依赖嵌入基础设施，在处理中文工具错误文本时表现不佳。我们提出了 EBEAC（基于事件的体验自动捕获），这是一种无需额外大语言模型成本的机制，直接从运行时事件流中采集工具层面的试错知识。 EBEAC 通过事件总线检测“连续失败 → 重试成功”序列，利用关键词模板生成器将其抽象为结构化经验（触发、诊断、修复），并通过三层过滤器（结果匹配、模式去重、时间衰减加权）检索这些经验，同时限制注入预算为 2000 个字符。在从十周实际运行中采集的 196 个真实经验（按时间顺序分为 137 个训练集与 59 个测试集）的生产日志回放中， EBEAC 在检索延迟仅为 0.97 毫秒时，实现了 71.19% 的 recall@1 和 89.36% 的 precision，且误报率仅为 3.39%——recall@3 与跨会话预防率均达到 76.27%——相较于基于词汇的基准方法， EBEAC 提供了一种以精确度为导向的替代方案，后者在误报率高出五倍（16.95%）的情况下仍能达到 83.05% 的 recall。捕获端分析显示，196 个经验案例仅归纳为 23 个抽象模式（归纳率为 88.3%），而 16.9% 的测试失败案例涉及训练过程中从未见过的模式；因此，残余召回差距主要由捕获端覆盖率而非检索端质量决定（91.8% 的被覆盖事件均能被检索到）。 EBEAC 定位为一种成本效益高的专用记忆机制，适用于操作中的高频工具特性，与基于策略性反思的学习相辅相成——而非与之竞争。
关键词：LLM agents · tool failure memory · event sourcing · zero-cost experience capture · desktop agents
浏览 5 · 下载 40 · 归档于 Zenodo