Observer wedges permanently on a stale `seven_day` quota entry that no incoming event can refresh

## Summary The observer's rate-limit cache is keyed by the event's top-level `rateLimitType` and has no TTL. The Claude Code CLI now reports per-window utilisation nested under `rate_limit_info.unifiedWindows`, and only names *one* window at the top level — the binding one. `unifiedWindows` is never read by claude-mem (the string does not appear in `worker-service.cjs` at all). Consequence: once a `seven_day` entry is cached at a high utilisation, nothing can ever overwrite it, because subsequent events are keyed `five_hour`. The quota guard iterates **all** windows on **every** event, so each fresh `five_hour` event re-reads the stale `seven_day` value and aborts the session. That arms a 30-minute cooldown; the probe re-aborts on the same entry; repeat forever. In my case the observer was wedged for ~41 hours (2026-10-21T21:17:46Z → manual intervention 2026-10-23T14:57Z) while actual seven-day utilisation was **16%**. It would not have self-cleared until `resetsAt` (2026-10-26) or a worker restart. ## Impact - Memory capture stops completely and does not recover on its own. - The failure is **more likely the longer the worker runs** — mine had 167h uptime, which I don't think is unusual. - The user-facing message is actively misleading (see below), so the natural response is to go buy capacity or switch providers, neither of which helps. ## The message is wrong, which is most of the damage The outage banner and `observer-health.json` both say: > Provider reported the inference allowance exhausted The provider reported no such thing. The cached entry that triggers the abort has: ```json "status": "allowed_warning" ``` This is a **locally inferred** threshold abort (`utilization >= 0.93`), not a provider rejection — the code path for a real rejection is a separate branch (`status === "rejected"`). The banner also states: > Restarting will NOT help here, and it clears the backoff that is currently keeping claude-mem from hammering the provider For this failure mode that is inverted: restarting is the **only** fix, and the backoff is throttling traffic that is entirely within quota. I'd suggest the wording distinguish a provider rejection from a local threshold abort, and only claim "allowance exhausted" for the former. ## Evidence `GET http://127.0.0.1:37777/api/health` → `rateLimits`, taken while wedged: ```json { "five_hour": { "status": "allowed", "rateLimitType": "five_hour", "resetsAt": 1792731000, "unifiedWindows": { "five_hour": { "utilization": 0.26, "resetsAt": 1792731000 }, "seven_day": { "utilization": 0.16, "resetsAt": 1793034000 } }, "observedAt": 1792727400073 }, "seven_day": { "status": "allowed_warning", "rateLimitType": "seven_day", "utilization": 0.97, "resetsAt": 1793034000, "surpassedThreshold": 0.75, "unifiedWindows": { "five_hour": { "utilization": 0, "resetsAt": 1792635000 }, "seven_day": { "utilization": 0.97, "resetsAt": 1793034000 } }, "observedAt": 1792617466708 } } ``` Two things stand out: 1. The `seven_day` entry's `observedAt` is **2026-10-21T21:17:46.708Z** — frozen at the exact second capture stopped, two days earlier. Its `resetsAt` is still in the future, so the staleness check never drops it. 2. The **fresh** `five_hour` entry carries the correct current value — `unifiedWindows.seven_day.utilization: 0.16` — in the very same object. The right answer is already in hand and simply isn't read. Resulting log, repeating every 30 minutes across days: ``` [2026-10-23 14:19:32.982] [WARN] [SDK] Aborting session for quota guard: quota:seven_day utilization 97.0% >= 93% {window=seven_day, authMethod=Claude Code OAuth token (read from system keychain at spawn)} [2026-10-23 14:19:41.275] [WARN] [SESSION] Generator paused for quota; preserving buffered work {pendingCount=318} [2026-10-23 14:32:00.675] [WARN] [SESSION] Skipping generator start while the provider quota cooldown is active {window=seven_day, retryInMs=1060585} ``` Independent ground truth — spawning the same CLI by hand and reading the raw event: ```json {"type":"rate_limit_event","rate_limit_info":{ "status":"allowed","rateLimitType":"five_hour","resetsAt":1792731000, "unifiedWindows":{ "five_hour":{"utilization":0.22,"resetsAt":1792731000}, "seven_day":{"utilization":0.16,"resetsAt":1793034000}}}} ``` No top-level `utilization` field at all, and `seven_day` appears only inside `unifiedWindows`. ## Root cause Three behaviours combine: 1. **Cache keyed by top-level `rateLimitType`, no TTL.** The store writes `entries.set(info.rateLimitType ?? "default", {...info, observedAt: Date.now()})`. `observedAt` is recorded but never consulted. 2. **The guard iterates every window on every event.** It walks `["five_hour","seven_day_opus","seven_day_sonnet","seven_day","overage"]` and returns on the first breach. Its only staleness defence is `if (resetsAt !== undefined && resetsAt <= now) continue` — useless for a long window whose reset is days away. (If `resetsAt` were ever absent, the entry would be immortal.) 3. **`unifiedWindows` is never parsed.** Grepping the shipped bundle for `unifiedWindows` returns no match, so non-binding windows can only be written when the CLI happens to make them binding. So `seven_day` is written exactly once — during a genuine spike — and is then pinned at that value for the life of the process. ## Reproduction 1. Drive seven-day utilisation above 93% so the CLI emits a top-level `rateLimitType: "seven_day"` event. The observer correctly aborts here. 2. Let utilisation fall well below the threshold (new 5-hour windows, lighter use). The CLI now emits `rateLimitType: "five_hour"` with the true `seven_day` figure nested in `unifiedWindows`. 3. Observe the observer stay wedged indefinitely: every event re-trips the guard on the stale entry. It recovers only on worker restart or when `seven_day.resetsAt` passes. Steps 2–3 are what I observed directly over ~41h. Step 1 is reconstructed from the logs and the cached entry's `observedAt` rather than deliberately re-driven, so treat the trigger as well-evidenced but not re-created from scratch. ## Suggested fixes 1. **Parse `unifiedWindows` and upsert every window it contains, not just `rateLimitType`.** This alone resolves it — each `five_hour` event carries a current `seven_day` figure, so the stale entry is overwritten within seconds. Highest value, smallest change. 2. **Use the `observedAt` you already record.** Treat an entry older than some TTL (a few minutes) as unknown rather than authoritative. Good defence in depth if the payload shape shifts again. 3. **Clear cached state on a successful probe.** Right now a probe that succeeds doesn't invalidate the entry that caused the cooldown, so recovery depends entirely on the stale value disappearing by itself. 4. **Fix the messaging** (see above): don't report "Provider reported the inference allowance exhausted" for a locally inferred threshold abort, and don't tell users a restart won't help when it's the only remedy. A smaller related observation: `overageStatus: "rejected"` with `overageDisabledReason: "org_level_disabled"` is permanent for accounts with extra usage switched off. The guard aborts on `overageStatus === "rejected"` when the window key is `overage`; if the CLI ever emits a top-level `rateLimitType: "overage"` for such an account, that would wedge the same way and never clear. I did not reproduce this one. ## Workaround **Order matters.** The worker loads `quota-cooldown.json` into memory once at boot, so deleting it afterwards has no effect — my first restart inherited the cooldown and stayed stuck: ```bash rm -f ~/.claude-mem/quota-cooldown.json curl -X POST http://127.0.0.1:37777/api/admin/restart ``` Verify with `STORED | obsIds=[...]` in `~/.claude-mem/logs/`, not with the banner. Note `observer-health.json` keeps showing `cooldownActive: true` until the cooldown map next changes, so it reads stale even after a successful recovery. The SQLite-backed queue survives the restart. ## Environment | | | |---|---| | claude-mem | 13.28.0 | | Claude Code CLI | 2.1.296 | | OS | Windows 11 Pro 26300 | | Worker | bun, `managed: false`, `hasIpc: false` | | Provider | `claude`, `CLAUDE_MEM_CLAUDE_AUTH_METHOD: subscription` (Pro, `default_claude_ai`) | | Worker uptime at failure | 167h | | Wedged for | ~41h, until manual restart |