Operator Playbook: Monitoring, Client Diversity, and Incident Response for AVSs

Why this playbook (and what “good” looks like)

AVSs expand what blockchains can do, but they also multiply slashable surfaces: signing malformed commitments, equivocation, late attestations, censorship alarms, invalid results, and more. In restaking, the same operator often serves multiple AVSs so a common-mode fault can trigger correlated slashing. “Good” operations therefore means three things:

1. Predictability: explicit SLOs, measurable performance, and clear change control.
2. Defense-in-depth: client diversity, key isolation, version pinning, staged rollouts, auto-remediation.
3. Fast, documented response: IR drills, slash-evidence capture, and transparent postmortems.

Done right, you reduce slash probability and severity, shrink mean time to recovery (MTTR), and earn delegator trust—often translating to better fees and stickier stake.

Operator principles (how to make fewer mistakes)

• Prefer boring infra: stable kernels, conservative OS updates, deterministic builds; keep “new and shiny” at the edge.
• Automate first, watch always: automation handles 90% of incidents; humans supervise and decide when to stop automation.
• Inversion of control: scripts should include disable flags; IR leads can freeze pipelines with one command.
• Single-responsibility nodes: minimize multi-AVS colocation to reduce correlated failure; if colocation is required, isolate with separate signers and clients.
• Rehearse reality: drills beat documents; measure the gap between runbook theory and on-call reality.

Define SLOs & error budgets

Service-level objectives convert risk into targets. Tie them to slash conditions or their precursors.

Error budgets (allowable failure minutes) inform change velocity: if you burn 60% of budget mid-month, freeze risky releases and focus on stability.

Observability stack (logs, metrics, traces, evidence)

Your stack should be dual-purpose: run the service and defend against slashing. Recommended architecture:

[Node & Clients]  →  [Exporter/Sidecar]  →  [Metrics DB]  →  [Dashboards]
   |                         |                |
   +→ [Structured Logs] -----+                +→ [Alertmanager/Pager]
   |                                          |
   +→ [Evidence Vault: signed msgs, headers, pcap, key audit logs]

• Metrics: time-series DB (Prometheus/VM); scrape at 5–10s for latency-sensitive AVSs.
• Logs: structured JSON; retain hot logs 7–14 days, cold storage ≥ 90 days (to dispute slash accusations).
• Traces: for complex pipelines (builders, bundle routers), propagate IDs to correlate slow paths.
• Evidence vault: append-only store of signed messages, monotonic counters, and packet captures around incidents (small rolling window) with chain-of-custody.

Golden signals & alerting rules (copy/paste templates)

Golden signals: latency, throughput, errors, saturation, plus slash precursors (equivocation risk, missed deadlines). Below are example alert rules (pseudo-YAML) to adapt.

groups:
- name: avs-operators
  rules:
  - alert: AVSDeadlineMissRateHigh
    expr: (sum(rate(avs_missed_deadlines_total[5m])) by (avs) /
           sum(rate(avs_required_attestations_total[5m])) by (avs)) > 0.001
    for: 5m
    labels: {severity: page}
    annotations:
      summary: "Missed deadlines >0.1% on {{ $labels.avs }}"
      runbook: "https://runbooks.example/avs-deadline-miss"

  - alert: EquivocationRiskSpike
    expr: increase(avs_conflicting_signatures_seen_total[1m]) > 0
    for: 1m
    labels: {severity: page}
    annotations:
      summary: "Conflicting signatures observed"
      action: "Freeze signer; switch to backup; start evidence capture."

  - alert: PreconfirmationLatencySLO
    expr: histogram_quantile(0.999, sum(rate(avs_preconf_latency_ms_bucket[2m])) by (le, region)) > 300
    for: 2m
    labels: {severity: ticket}
    annotations:
      summary: "Preconfirmation p99.9 > 300ms"
      action: "Route traffic regionally; check peer backlog."

Add blackhole alerts: if the exporter stops reporting, page within 2–3 minutes. Missing telemetry is itself an incident.

Client diversity strategy (and how to enforce it)

Client monoculture is the fastest path to correlated slashing. Your goal is to keep any single client, version, or automation stack below policy caps and to ensure rollbacks are quick.

Measure effective diversity—not just counts. Ten nodes on the same cloud/AZ are not diverse. Enforce geographic and provider spread.

Release management, canaries, and rollbacks

Most major incidents trace back to rushed upgrades. Adopt a staged rollout with canary, cohort, fleet phases and pre-approved rollback buttons.

1. Canary (1–5% nodes): run for 2–6 hours; watch error rate, CPU, mem, equivocation flags.
2. Cohort (25–40%): expand to low-risk regions; continue monitoring.
3. Fleet (100%): only after error budget status is green for the month.

Keys, HSMs, and signer security (no shared hot keys)

Key compromise is catastrophic—often slash-max. Minimum standards:

• Per-AVS keys: never reuse a signing key across AVSs; distinct key paths and labels.
• HSM/TEE-backed signers: hardware-backed or enclave-backed signers with rate limits (e.g., max X signatures per window) and pause controls.
• Air-gapped backups: for emergency rotations; test restoration quarterly.
• MPC/threshold for admin keys: distribute trust; geofenced signers across cloud providers.

Log every signature request and decision: digest, AVS, monotonic counter, result. These logs are your innocence proof in slash disputes.

Network topology & capacity planning

Sequencers, builders, and data networks are latency-sensitive. Design for regional proximity, dedicated egress, and DDoS damping.

• Regional cells: self-contained deployments (ingress, peer manager, signer, observer) per region; no single region can starve the fleet.
• Peering policies: throttle unknown peers; priority lanes for known relays; circuit breakers for traffic floods.
• Capacity margin: keep 30–50% headroom; autoscale read-only components; pin signer capacity.
• Time sync: NTP/PTP hardened; drift alarms at 100 ms; clock skew can cause deadline misses.

Chaos drills & game days (practice the bad paths)

Run short, sharp drills that simulate real slash precursors:

• Equivocation canary: inject near-conflict inputs in a sandbox; confirm detectors fire and signers pause.
• Deadline spike: burst traffic to saturate IO; validate autoscaling and backpressure work.
• Signer freeze: simulate HSM unavailability; ensure failover and queue draining are clean.
• Rollback race: revert client version under load; measure recovery time and error spikes.

Record every drill: time to detect, time to mitigate, artifacts captured, policy gaps. Convert gaps to tickets with owners and deadlines.

Incident response (IR) playbook that actually works

When seconds matter, ambiguity kills. Standardize on SEV levels and roles.

SEV-1 quick actions (scriptable)

1. Freeze: disable signing on suspect nodes (feature flag/API) while keeping telemetry alive.
2. Isolate: quarantine canaries; route traffic to known-good cohort.
3. Rollback: revert to last-good client; reload configuration hashes.
4. Capture: rotate logs, export last 10–15 minutes of signed messages, attach to case.
5. Communicate: status page + AVS operator channel with facts, not speculation.

Evidence & slash defense (trust, but verify and archive)

Slash defense succeeds when you can prove your node behaved or that an external fault occurred. Keep a minimal but strong evidence pack schema:

• Signed message digests with timestamps and nonces
• Client version/build hash, config checksum
• Peer view (peer IDs, latency, partition hints)
• Network captures around the incident window (rolling buffer)
• Signer audit logs (HSM decisions, rate limits triggered)

Store evidence immutably (WORM or hash-anchored). Document who can access and how to export for AVS arbitration.

Postmortems & learning loops (reduce recurrence)

Every SEV-1/2 gets a blameless postmortem within 5 business days. Structure:

1. What happened: timeline with raw metrics and screenshots.
2. Why it happened: 5-Whys that stops at policy or system causes (not people).
3. Where the sensors failed: which alerts were late or noisy.
4. Fixes: specific, dated, owners; link to PRs and runbook changes.
5. Follow-up drill: run the reproduction and demonstrate improvement.

Share summaries with delegators and AVS partners. Transparency buys patience when the next gray-swan hits.

Practical checklists (print these)

Frequently Asked Questions

Isn’t client diversity expensive to maintain?

It costs less than a slash. You can reduce overhead with standardized exporters, config generators, and CI that builds multiple clients from a single manifest. The key is policy + automation: target shares and automated admission control.

What’s the most important alert?

Anything that predicts a slash: conflicting signatures seen, rapid clock skew, missed-deadline ratio spikes, signer rate-limit saturation, DA posting failures (for sequencers). Blackhole alerts (no metrics) are next if you’re blind, you’re already late.

How do we balance speed vs. safety on releases?

Use error budgets. If you’re green, allow normal velocity with canaries. If you’ve burned most of the budget, freeze risky upgrades, focus on stability, and regain headroom before shipping features.

Should we colocate multiple AVSs on the same bare metal?

Prefer separation. If you must colocate, isolate with different signers/HSM partitions, cgroups, network namespaces, and per-AVS rate limits. Treat the host as failure domain that can take down multiple services document the blast radius.

What evidence convinces an AVS arbitration panel?

Signed message timelines, client/version hashes, monotonic counters, and independent network captures showing partition or upstream invalid data. Keep data tamper-evident and exportable under time pressure.