Before the LLM work and the solo products, I carried a pager for crypto trading infrastructure — first at ApexCrypto, then at Bakkt. Regulated finance adds a layer on top of normal SRE: every incident has a compliance shadow, and "we'll fix it in the morning" is rarely an option when money is moving.

None of this is crypto-specific advice. It is reliability engineering with the safety margins turned up.

Ambiguity is the enemy

In consumer web, a fuzzy error message is annoying. In trading, ambiguity is how you lose trust — with users, with auditors, and with your own team.

We learned to be precise everywhere:

• Idempotency keys on anything that could retry
• Explicit state machines for orders and settlements — no "probably filled"
• Runbooks that name the rollback, not just the forward fix

If two on-call engineers would interpret an alert differently, the alert is wrong.

Change management is a feature

Fast deploys are good. Unreviewed deploys during market hours are a different sport.

The habit that saved us: small changes, feature flags where possible, and a default posture of "freeze during high-volatility windows" unless the change reduces risk. Boring Tuesday deploys beat heroic Friday saves.

Observability before optimization

Latency tuning is seductive. Knowing which path an order took is prerequisite.

We invested early in:

• End-to-end traces across matching, risk checks, and settlement
• Business-level SLIs — not just "service up" but "order acknowledged within X ms"
• Dashboards that executives and engineers could read without translation

When an incident started, the first question was never "what broke?" — it was "what is the customer impact right now?"

Incidents are communications events

Technical resolution is half the job. The other half:

• Clear internal timeline — who knew what, when
• External messaging that matches reality — no "all systems operational" while reconciliation is still running
• Postmortems that assign systems fixes, not blame

Regulated environments teach you that trust compounds slowly and burns fast.

What I took with me

The same primitives show up in LLM pipelines and edge-hosted side projects:

• Fail closed, not open
• Make retries safe
• Measure outcomes, not just uptime
• Write the runbook before you need it

Crypto trading and model inference look unrelated. The pager teaches the same lesson: your users experience the worst case, not your median.

For the client-side half of that story — backoff, jitter, circuit breakers on flaky endpoints — see the Retry & Backoff Playground.