evidence / Engineering depth
Unattended Systems That Fail Visibly -- the reliability stack behind an overnight agentic build
This platform is built to run unattended for days at a time under an agent fleet. The engineering that makes that safe is not the absence of failures -- it is the machinery that makes a failure loud. The single truth-source for any figure below is docs/JOB_EVIDENCE_PACKET.md (sections B and G).
strongest single receipt
the claim
A system that runs overnight with no human watching has exactly one unacceptable failure mode: failing silently. A wedged loop, a full disk, a dropped alert, a corrupted ledger, or a dashboard that stays green over a dead subsystem are all worse than a crash, because a crash at least announces itself. So the reliability work on this platform is organized around a single principle -- an unattended run must fail LOUDLY -- and the sharpest proof of that principle is a health readout that was caught rolling up green over a genuinely-down subsystem, fixed, and now reports OVERALL: RED with the specific broken subsystem named. On this page a red status, honestly reported, is a feature receipt, not an embarrassment. That is the whole point of the stack: it is the difference between "looked fine" and "is fine."
receipts
| claim | committed artifact |
|---|---|
| Watchdog-supervised 9-daemon fleet | scripts/daemon_watchdog.py, scripts/daemon_registry.json |
| Alerting: token-bucket + dead-letter + circuit breaker | scripts/execute_loop/L22_alerting.py |
| Transactional units ledger (locking, stale-lock recovery, atomic writes) | src/betting/pnl_ledger.py; race test scripts/platformkit/test_clv_ledger_io.py |
| Ops sentinels (disk / exception-burst / heartbeat / tamper-evidence / wedge-restart) | scripts/platformkit/ops_sentinel/ |
| Serving layer + kill switch + ops/health dashboard | api/main.py (/api/risk/status, /health/ops) |
| One-command liveness harness (reports RED honestly) | scripts/platformkit/proof_harness/system_proof.py (fix 2eedc37e) |
| Sanctioned single-PID restart, verified twice | scripts/daemon_watchdog.py; runtime data/frontend/ops/post_restart_verify.json (local-only) |
why this matters
For a platform, infrastructure, or SRE-adjacent role, the interesting question is never "did it work in the demo" -- it is "what happens at 3am when it doesn't, and no one is watching." This stack is the answer: supervised processes that relaunch, an alert path hardened against its own failure modes, a ledger that stays intact under concurrent writers, sentinels that watch the watchers, and a single health command whose defining feature is that it will tell you the truth when things are broken. The load-bearing decision was to treat a decorative green as a bug and an honest red as correct behavior -- and to ship the commit that enforces it. That instinct, that an unattended system's first job is to fail visibly, is the reliability skill the rest of the stack is built to demonstrate.
reproduce
# One-command health readout: prints OVERALL RED/GREEN with the failing subsystems named
python -m scripts.platformkit.proof_harness.system_proof
# Sentinel tests (each watcher has its own test file)
python -m pytest scripts/platformkit/ops_sentinel/test_heartbeat_coverage.py -q
python -m pytest scripts/platformkit/ops_sentinel/test_guard_integrity.py -q
python -m pytest scripts/platformkit/ops_sentinel/test_disk_space.py -q
# Alerting subsystem: rate-limit, dead-letter queue, circuit breaker
python -m pytest scripts/execute_loop/tests/test_L22_alerting.py -q
# Ledger concurrency: the two-OS-subprocess race that proves the lock actually closes
python -m pytest scripts/platformkit/test_clv_ledger_io.py -qThis site reports calibration and sharpness only, never a dollar edge, ROI, or bankroll result. An honest null is a success.