--- pageType: source id: source.bridge.workspace-4c6d747f.memory-2026-08-27-619ff128 title: "Memory Bridge (main, coder-agent, evaluator-agent, humanizer-agent, setup-agent): 2026-08-27" sourceType: memory-bridge sourcePath: /home/owner/.openclaw/workspace/memory/2026-08-27.md bridgeRelativePath: memory/2026-08-27.md bridgeWorkspaceDir: /home/owner/.openclaw/workspace bridgeAgentIds: - main - coder-agent - evaluator-agent - humanizer-agent - setup-agent status: active updatedAt: 2026-08-27T20:37:52.618Z --- # Memory Bridge (main, coder-agent, evaluator-agent, humanizer-agent, setup-agent): 2026-08-27 ## Bridge Source - Workspace: `/home/owner/.openclaw/workspace` - Relative path: `memory/2026-08-27.md` - Kind: `markdown` - Agents: main, coder-agent, evaluator-agent, humanizer-agent, setup-agent - Updated: 2026-08-27T20:37:52.618Z ## Content ```markdown # 2026-08-27 ## Forensics: dreaming-sweep abort 27.08 03:00 MSK (read-only, session fbb58067) ### Timeline (MSK) - 03:00:00 — cron `dreaming-sweep` start (job id 3ee48d4c-d561-4129-9749-c90231ac8d15, isolated session, timeout 900s) - 03:00:02 — session start, prompt submitted (model qwen3.8-27b-q5xl, warm/keep_alive 12h) - 5 model calls: ~58s / 34s / 21s / 156s / 189s = ~458s total (~54% of run). Prefill grows with context (input 192 040 / output 22 807 tokens cumulative; last call 53 897 input) - 14 tool calls: 11 completed fast (batches 1–4, 0.2–0.5s each, ~80.5KB ≈ 20k tokens — not the problem) - Final batch at 03:07:41.664Z: exec + wiki_search + exec — **never committed** (batch-commit semantics; wiki_search hung, exec companions unpersisted) - 03:09:57 — first `stalled session` diagnostic (tool age 135s) - 03:13:57.273Z — watchdog `abort_embedded_run` (tool age ~375s) - cron: `AbortError: agent run aborted`, durationMs=839 392 ### Key measurements - wiki_search (1 call, query "VRAM RTX 5090 resident model mutually exclusive"): hung ≥375.6s, returned 0 bytes / 0 tokens — the fatal bottleneck - Job start → abort: 839.4s. The 900s cron timeout never fired because the stuck-session watchdog aborted first ### Watchdog explanation (standard OpenClaw, NOT the cron timeout) - `diagnostics` config absent → defaults: warn=120s, abort = max(300s, 3×120s) = **360s of no progress**, measured from last-progress/active-tool age, NOT session start - batch 5 started 03:07:41 → +360s → 30s diagnostic tick at 03:13:57 saw tool age 375s ≥ 360s → abort - The "600–840s" window = model time before the hang (~458s) + 360s+ hang. Earlier hang start → earlier abort (parallel wiki-ingest-event eaa637d7 aborted at ~645s, same mechanism) ### Root cause (minimal) **wiki_search → shared memory-search backend → bge-m3 embeddings on Dell (192.168.1.200:11434) → endpoint unresponsive.** - Live read-only probe: POST /api/embed bge-m3 → 120s hang, http_code=000. /api/ps + /api/tags respond; only 27b loaded (~28GB VRAM), bge-m3 not loaded - Gateway logs: since 26.08 08:41 MSK continuous `memory embeddings retryable error` and `memory embeddings batch timed out after 600s` - On 26.08 wiki_search at least failed fast (`memory embeddings query timed out after 60s`, twice); on 27.08 no such line — hang deeper than 375s inside embedding path - Successful sweeps 17–22.08 never called wiki_search (cron durations 209–298s). 27.08 = first sweep where model spontaneously used wiki_search and hit dead bge-m3 - Conclusion: NOT "27B + 131k + wiki_search" combo; the minimal cause is the dead bge-m3 endpoint. Slow 27B prefill is secondary (fits within 900s on its own) ### Parallel wiki-ingest-event abort (00:02 MSK, session eaa637d7) - Same mechanism: wiki_search hung (query "VRAM RTX 5090 27b residen..."), watchdog abort at session age ~645s; lane error durationMs=648 958; `embedded abort settle timed out ... timeoutMs=2000` - The event was later processed normally at 08:33 — no data loss ### Trajectory file limits (learned) - `.trajectory.jsonl` has only 7 event types (session.started, trace.metadata, context.compiled, prompt.submitted, model.completed, trace.artifacts, session.ended) — no per-tool call/result/timing details - 21–22.08 OK runs have trajectory only → per-sweep tool detail not extractable; 18–20.08 jsonl files span multi-day main sessions (comparison is approximate) ### Status / decisions - Report delivered in chat (RU). User: NO fixes for now (no bge-m3 repair, no timeout changes, no backfill 26–27.08, wiki-ingest abort left as-is). Awaiting user decision on minimal recommendation: restore bge-m3 endpoint responsiveness on Dell (infra → setup-agent, needs user OK). - Live /api/embed probe was one-time read-only; bge-m3 did NOT get loaded (post-probe /api/ps unchanged: only 27b). ### Artifacts - Failed sweep: ~/.openclaw/agents/main/sessions/fbb58067-f86f-40a2-b2be-e758ebddbad7{.jsonl,.trajectory.jsonl} - Ingest abort: ~/.openclaw/agents/main/sessions/eaa637d7-8d78-4d3a-8994-9f850d7d71f4{.jsonl,.trajectory.jsonl} - 26.08 isolated failed runs (jsonl+trajectory): 22efd928, d45f8e8c, fa0b2b81, fa20c5e0 - Logs: /tmp/openclaw/openclaw-2026-08-26.log, openclaw-2026-08-27.log (MSK timestamps) ``` ## Notes ## Related - No related pages yet.