obsidian-vault/sources/bridge-workspace-4c6d747f-memory-2026-08-27-619ff128.md
2026-11-03 23:00:01 +00:00

5 KiB
Raw Blame History

pageType id title sourceType sourcePath bridgeRelativePath bridgeWorkspaceDir bridgeAgentIds status updatedAt
source source.bridge.workspace-4c6d747f.memory-2026-08-27-619ff128 Memory Bridge (main): 2026-08-27 memory-bridge /home/owner/.openclaw/workspace/memory/2026-08-27.md memory/2026-08-27.md /home/owner/.openclaw/workspace
main
active 2026-08-27T20:37:52.618Z

Memory Bridge (main): 2026-08-27

Bridge Source

  • Workspace: /home/owner/.openclaw/workspace
  • Relative path: memory/2026-08-27.md
  • Kind: markdown
  • Agents: main
  • Updated: 2026-08-27T20:37:52.618Z

Content

# 2026-08-27

## Forensics: dreaming-sweep abort 27.08 03:00 MSK (read-only, session fbb58067)

### Timeline (MSK)
- 03:00:00 — cron `dreaming-sweep` start (job id 3ee48d4c-d561-4129-9749-c90231ac8d15, isolated session, timeout 900s)
- 03:00:02 — session start, prompt submitted (model qwen3.8-27b-q5xl, warm/keep_alive 12h)
- 5 model calls: ~58s / 34s / 21s / 156s / 189s = ~458s total (~54% of run). Prefill grows with context (input 192 040 / output 22 807 tokens cumulative; last call 53 897 input)
- 14 tool calls: 11 completed fast (batches 1–4, 0.2–0.5s each, ~80.5KB ≈ 20k tokens — not the problem)
- Final batch at 03:07:41.664Z: exec + wiki_search + exec — **never committed** (batch-commit semantics; wiki_search hung, exec companions unpersisted)
- 03:09:57 — first `stalled session` diagnostic (tool age 135s)
- 03:13:57.273Z — watchdog `abort_embedded_run` (tool age ~375s)
- cron: `AbortError: agent run aborted`, durationMs=839 392

### Key measurements
- wiki_search (1 call, query "VRAM RTX 5090 resident model mutually exclusive"): hung ≥375.6s, returned 0 bytes / 0 tokens — the fatal bottleneck
- Job start → abort: 839.4s. The 900s cron timeout never fired because the stuck-session watchdog aborted first

### Watchdog explanation (standard OpenClaw, NOT the cron timeout)
- `diagnostics` config absent → defaults: warn=120s, abort = max(300s, 3×120s) = **360s of no progress**, measured from last-progress/active-tool age, NOT session start
- batch 5 started 03:07:41 → +360s → 30s diagnostic tick at 03:13:57 saw tool age 375s ≥ 360s → abort
- The "600–840s" window = model time before the hang (~458s) + 360s+ hang. Earlier hang start → earlier abort (parallel wiki-ingest-event eaa637d7 aborted at ~645s, same mechanism)

### Root cause (minimal)
**wiki_search → shared memory-search backend → bge-m3 embeddings on Dell (192.168.1.200:11434) → endpoint unresponsive.**
- Live read-only probe: POST /api/embed bge-m3 → 120s hang, http_code=000. /api/ps + /api/tags respond; only 27b loaded (~28GB VRAM), bge-m3 not loaded
- Gateway logs: since 26.08 08:41 MSK continuous `memory embeddings retryable error` and `memory embeddings batch timed out after 600s`
- On 26.08 wiki_search at least failed fast (`memory embeddings query timed out after 60s`, twice); on 27.08 no such line — hang deeper than 375s inside embedding path
- Successful sweeps 17–22.08 never called wiki_search (cron durations 209–298s). 27.08 = first sweep where model spontaneously used wiki_search and hit dead bge-m3
- Conclusion: NOT "27B + 131k + wiki_search" combo; the minimal cause is the dead bge-m3 endpoint. Slow 27B prefill is secondary (fits within 900s on its own)

### Parallel wiki-ingest-event abort (00:02 MSK, session eaa637d7)
- Same mechanism: wiki_search hung (query "VRAM RTX 5090 27b residen..."), watchdog abort at session age ~645s; lane error durationMs=648 958; `embedded abort settle timed out ... timeoutMs=2000`
- The event was later processed normally at 08:33 — no data loss

### Trajectory file limits (learned)
- `.trajectory.jsonl` has only 7 event types (session.started, trace.metadata, context.compiled, prompt.submitted, model.completed, trace.artifacts, session.ended) — no per-tool call/result/timing details
- 21–22.08 OK runs have trajectory only → per-sweep tool detail not extractable; 18–20.08 jsonl files span multi-day main sessions (comparison is approximate)

### Status / decisions
- Report delivered in chat (RU). User: NO fixes for now (no bge-m3 repair, no timeout changes, no backfill 26–27.08, wiki-ingest abort left as-is). Awaiting user decision on minimal recommendation: restore bge-m3 endpoint responsiveness on Dell (infra → setup-agent, needs user OK).
- Live /api/embed probe was one-time read-only; bge-m3 did NOT get loaded (post-probe /api/ps unchanged: only 27b).

### Artifacts
- Failed sweep: ~/.openclaw/agents/main/sessions/fbb58067-f86f-40a2-b2be-e758ebddbad7{.jsonl,.trajectory.jsonl}
- Ingest abort: ~/.openclaw/agents/main/sessions/eaa637d7-8d78-4d3a-8994-9f850d7d71f4{.jsonl,.trajectory.jsonl}
- 26.08 isolated failed runs (jsonl+trajectory): 22efd928, d45f8e8c, fa0b2b81, fa20c5e0
- Logs: /tmp/openclaw/openclaw-2026-08-26.log, openclaw-2026-08-27.log (MSK timestamps)

Notes

  • No related pages yet.