Axiom Observability Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. Independent runtime integrations use the dispatching-parallel-agents skill after the shared protocol is fixed.

Goal: Add selected diagnostic logs for both SeekAlgo repositories without log-spending ceilings.

Architecture: An independent authenticated collector validates and aggregates events into a SQLite-backed spool, then batches to one Axiom dataset. Producers report operation boundaries and counter deltas without changing business behavior. Delivery failures remain separate from product failures.

Tech Stack: Python/FastAPI/SQLite, existing Next.js/TypeScript and Node runner; existing HTTP clients.

Spec: SEEKALGO_AXIOM_OBSERVABILITY_SPEC.md (copied into docs/observability/ during implementation). Shared wire contract: downloader observability/schema.py and the typed runtime constructors. See docs/observability/runbook.md and the companion verification record for evidence and limitations.

Global Constraints

• No daily/monthly spending ceiling, byte-admission ledger, usage-triggered cutoff or sampling change.
• Fixed allowlisted metadata; no prompts, source, arbitrary errors, raw provider or financial payloads.
• Target <1 KiB, maximum 2 KiB per event; queues bound 128 events / 256 KiB; gateway batches 50 events / 64 KiB.
• 15-minute summaries; first repeated failure per 10-minute fingerprint plus suppressed counts and recovery.
• SQLite spool initially 64 MiB / 24h; adjustable physical resources, never spending controls.
• Preserve API statuses, scheduling, provider requests, storage semantics, engine isolation and trading decisions.
• Opt-in configuration; no production account changes, token provisioning or deployment in this PR task.

Review Focus

• Telemetry outage never changes streaming completion or masks primary errors.
• Browser cannot assert trusted identity or inject arbitrary content.
• Accepted/rejected/ambiguous Axiom batches never retry accepted events blindly.
• Aggregation preserves recovery/counts through restart and distinguishes absent owner from idle/complete owner.
• Traffic at 1,000 DAU and missing usage observations cannot cause a logging cutoff.

Task1: Shared collector and delivery

Files: create observability/{schema,store,gateway,exporter}.py and tests/test_observability_gateway.py in downloader. Interfaces: authenticated POST/v1/events consumes flat events and counter deltas; durable 202; exporter consumes persisted records and Axiom acceptance counts.

• Write failing behavior tests for privacy, auth, persistence/restart, dedupe, coalescing/recovery, partial acceptance, retry, expiry/overflow and continued delivery at high accumulated bytes.
• Run pytest tests/test_observability_gateway.py and observe missing behavior.
• Implement schema, spool, collector and exporter; run fixtures against actual SQLite and local HTTP fake Axiom.
• Verify tests pass and commit collector.

Task2: Python downloader integration

Files: pipeline/observability.py; scheduler/downloader/provider/storage/stream boundaries; api middleware/backfill manager; operational script terminals; tests/test_observability_producer.py. Interfaces: emit_event(event_name,**allowed_fields), count_outcome(event_name,**counter_fields), lifecycle summary from existing counters; shared POST/v1/events.

• Write/run failing tests for safe bounded nonblocking delivery, counter summaries, partial publish/bookkeeping, ASGI late failure and health exclusions.
• Instrument spec event boundaries; preserve domain outcomes.
• Run pytest tests plus existing focused/full offline suite and commit changes.

Task3: Node runner integration

Files: live-runner/lib/observability.js, runner.js, queue/pool boundaries and live-runner/test/observability.test.js. Interfaces: safe emit/count/summary helpers; same wire contract; no per-bot successful polls.

• Write/run failing tests for redaction, queue bounds, HTTP failure isolation, duplicate failure aggregation and timeout/discovery/reconciliation outcomes.
• Instrument startup/dependency/queue/engine/persistence and meaningful intent transitions.
• Run npm test and commit changes.

Task4: Next.js and browser integration

Files: src/lib/observability/**; /api/observability; builder/backtest/MCP/paper/auth/data boundaries; browser parent workers/data/error boundaries; scripts/test-observability.ts. Interfaces: safe operation events/counter deltas; completion hook registered in request context; restricted same-origin browser relay.

• Write/run failing tests for HTTP200 semantic errors, stream terminal ordering, safe browser validation, save failure, metadata privacy, timeout isolation and no usage cutoff.
• Add typed constructors/relay/instrumentation with deterministic 1% routine success sampling.
• Run focused scripts, typecheck and lint; commit changes. Attempt the production build without deployment migrations and record its local tooling stall.
• Verify the production build in Next CI; actual Edge/browser import-boundary fixtures pass.
• During activation, verify a real browser workflow with logging enabled.

Task5: Deployment and cross-system verification

Files: downloader Dockerfile.observability, Compose/env.example, docs/observability/{spec,runbook,monitors}; Next env/runbook; host-lifecycle probe. Interfaces: opt-in profile, separate service credentials, one dataset-scoped token only at collector; expected-owner roster and three monitor queries.

• Test local end-to-end producer->collector->fakeAxiom including CORS/range/streaming IDs and replay after restart.
• Run 1,000 DAU workload fixture and report fanout/serialized bytes/queue behavior; storage/query costs remain production rollout checks.
• Validate Compose/config, review both diffs, fix blockers with reproducing tests.
• Commit, push both feature branches, create linked PRs and attach both to chat: Next #69 and downloader #74.

Implementation checks completed locally and in CI, including the Next production build and downloader disposable-PostgreSQL cases. A real browser workflow and live Axiom receipt/readback remain activation checks as documented in the runbooks. No production deployment migrations were run.