obsidian-vault/sources/bridge-workspace-4c6d747f-memory-2026-08-27-619ff128.md
2026-10-17 19:00:02 +00:00

89 lines
5.2 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

---
pageType: source
id: source.bridge.workspace-4c6d747f.memory-2026-08-27-619ff128
title: "Memory Bridge (main, coder-agent, evaluator-agent, humanizer-agent,
setup-agent): 2026-08-27"
sourceType: memory-bridge
sourcePath: /home/owner/.openclaw/workspace/memory/2026-08-27.md
bridgeRelativePath: memory/2026-08-27.md
bridgeWorkspaceDir: /home/owner/.openclaw/workspace
bridgeAgentIds:
- main
- coder-agent
- evaluator-agent
- humanizer-agent
- setup-agent
status: active
updatedAt: 2026-08-27T20:37:52.618Z
---
# Memory Bridge (main, coder-agent, evaluator-agent, humanizer-agent, setup-agent): 2026-08-27
## Bridge Source
- Workspace: `/home/owner/.openclaw/workspace`
- Relative path: `memory/2026-08-27.md`
- Kind: `markdown`
- Agents: main, coder-agent, evaluator-agent, humanizer-agent, setup-agent
- Updated: 2026-08-27T20:37:52.618Z
## Content
```markdown
# 2026-08-27
## Forensics: dreaming-sweep abort 27.08 03:00 MSK (read-only, session fbb58067)
### Timeline (MSK)
- 03:00:00 — cron `dreaming-sweep` start (job id 3ee48d4c-d561-4129-9749-c90231ac8d15, isolated session, timeout 900s)
- 03:00:02 — session start, prompt submitted (model qwen3.8-27b-q5xl, warm/keep_alive 12h)
- 5 model calls: ~58s / 34s / 21s / 156s / 189s = ~458s total (~54% of run). Prefill grows with context (input 192 040 / output 22 807 tokens cumulative; last call 53 897 input)
- 14 tool calls: 11 completed fast (batches 1–4, 0.2–0.5s each, ~80.5KB ≈ 20k tokens — not the problem)
- Final batch at 03:07:41.664Z: exec + wiki_search + exec — **never committed** (batch-commit semantics; wiki_search hung, exec companions unpersisted)
- 03:09:57 — first `stalled session` diagnostic (tool age 135s)
- 03:13:57.273Z — watchdog `abort_embedded_run` (tool age ~375s)
- cron: `AbortError: agent run aborted`, durationMs=839 392
### Key measurements
- wiki_search (1 call, query "VRAM RTX 5090 resident model mutually exclusive"): hung ≥375.6s, returned 0 bytes / 0 tokens — the fatal bottleneck
- Job start → abort: 839.4s. The 900s cron timeout never fired because the stuck-session watchdog aborted first
### Watchdog explanation (standard OpenClaw, NOT the cron timeout)
- `diagnostics` config absent → defaults: warn=120s, abort = max(300s, 3×120s) = **360s of no progress**, measured from last-progress/active-tool age, NOT session start
- batch 5 started 03:07:41 → +360s → 30s diagnostic tick at 03:13:57 saw tool age 375s ≥ 360s → abort
- The "600–840s" window = model time before the hang (~458s) + 360s+ hang. Earlier hang start → earlier abort (parallel wiki-ingest-event eaa637d7 aborted at ~645s, same mechanism)
### Root cause (minimal)
**wiki_search → shared memory-search backend → bge-m3 embeddings on Dell (192.168.1.200:11434) → endpoint unresponsive.**
- Live read-only probe: POST /api/embed bge-m3 → 120s hang, http_code=000. /api/ps + /api/tags respond; only 27b loaded (~28GB VRAM), bge-m3 not loaded
- Gateway logs: since 26.08 08:41 MSK continuous `memory embeddings retryable error` and `memory embeddings batch timed out after 600s`
- On 26.08 wiki_search at least failed fast (`memory embeddings query timed out after 60s`, twice); on 27.08 no such line — hang deeper than 375s inside embedding path
- Successful sweeps 17–22.08 never called wiki_search (cron durations 209–298s). 27.08 = first sweep where model spontaneously used wiki_search and hit dead bge-m3
- Conclusion: NOT "27B + 131k + wiki_search" combo; the minimal cause is the dead bge-m3 endpoint. Slow 27B prefill is secondary (fits within 900s on its own)
### Parallel wiki-ingest-event abort (00:02 MSK, session eaa637d7)
- Same mechanism: wiki_search hung (query "VRAM RTX 5090 27b residen..."), watchdog abort at session age ~645s; lane error durationMs=648 958; `embedded abort settle timed out ... timeoutMs=2000`
- The event was later processed normally at 08:33 — no data loss
### Trajectory file limits (learned)
- `.trajectory.jsonl` has only 7 event types (session.started, trace.metadata, context.compiled, prompt.submitted, model.completed, trace.artifacts, session.ended) — no per-tool call/result/timing details
- 21–22.08 OK runs have trajectory only → per-sweep tool detail not extractable; 18–20.08 jsonl files span multi-day main sessions (comparison is approximate)
### Status / decisions
- Report delivered in chat (RU). User: NO fixes for now (no bge-m3 repair, no timeout changes, no backfill 26–27.08, wiki-ingest abort left as-is). Awaiting user decision on minimal recommendation: restore bge-m3 endpoint responsiveness on Dell (infra → setup-agent, needs user OK).
- Live /api/embed probe was one-time read-only; bge-m3 did NOT get loaded (post-probe /api/ps unchanged: only 27b).
### Artifacts
- Failed sweep: ~/.openclaw/agents/main/sessions/fbb58067-f86f-40a2-b2be-e758ebddbad7{.jsonl,.trajectory.jsonl}
- Ingest abort: ~/.openclaw/agents/main/sessions/eaa637d7-8d78-4d3a-8994-9f850d7d71f4{.jsonl,.trajectory.jsonl}
- 26.08 isolated failed runs (jsonl+trajectory): 22efd928, d45f8e8c, fa0b2b81, fa20c5e0
- Logs: /tmp/openclaw/openclaw-2026-08-26.log, openclaw-2026-08-27.log (MSK timestamps)
```
## Notes
<!-- openclaw:human:start -->
<!-- openclaw:human:end -->
## Related
<!-- openclaw:wiki:related:start -->
- No related pages yet.
<!-- openclaw:wiki:related:end -->