fix(blue-green-deployment): correct contradictory rollback guidance, complete SKILL.md, harden audit trail
references/standards.md instructed the reader that "Rollback is a simple pointer
swap" and mandated reversion in under 1ms. That is the exact defect the
implementation's primary safeguard exists to prevent: rollback reconciles state
backwards from the live slot and refuses when reconciliation fails. It also
justified state continuity with "Blue remains actively updating its state from
market data" -- market data does not carry the live instance's fills.
SKILL.md carried placeholder text ("Documentation for When to Use.") in six of
the seven required sections; the validator only checks that headers exist, so
this passed CI.
Documentation:
- Rewrite SKILL.md with real When to Use / When NOT to Use, workflow decision
points, and specific pitfalls, following sibling deployment-ops conventions.
- Rewrite standards.md: correct the rollback and hot-standby claims, separate
regulatory obligations from engineering practice, and scope each to its
jurisdiction. Add verified RTS 6 Art. 8 (controlled deployment) and Art. 11
(management of material changes), the two most on-point articles, which were
absent; add SEC Rule 15c3-5 pre-trade control placement.
- Extend workflows.md with the slot lifecycle, cutover-refusal classification
(a failed state sync is not retryable), and the kill-switch escalation.
- Extend checklist.md with authoriser sign-off, rollback-viability and audit
export gates.
- Fix frontmatter: subdomain general -> deployment-ops, meaningful tags,
discovery-oriented description; regenerate index.json.
Implementation:
- Add is_authorised_to_route(); the docstring told the execution layer to
consult the deployer before every submission but exposed no API for it,
leaving integrators to read mutable slot state outside the lock.
- Record an INITIALIZE baseline; initial_version was documented as recorded in
deployment_history but nothing was written at construction.
- Flag forced rollbacks. The [FORCED] marker was gated on force and not
sync_ok, so overriding the target-eligibility guard while the sync happened
to succeed was recorded as an ordinary successful rollback.
- Close an ABA hole in the post-warmup guard: identifying the slot by
(state, version) let a decommission plus same-version redeploy during warmup
receive the previous attempt's stale health verdict. Use a generation token.
- Accept authorised_by on decommission_standby().
Tests: 25 -> 34. The two regression tests were verified to fail against the
pre-fix code and pass after. All API changes are additive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoqCTfNGdU91RQWxC3Jwyv
This commit is contained in:
Claude
+10
-11
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"version": "1.0.0",
|
||||
"generated_at": "2026-08-13T12:02:26.458024+00:00",
|
||||
"generated_at": "2026-08-13T16:41:01.504301+00:00",
|
||||
"repository": "https://github.com/HimanshuJ16/Algo-Trading-Skills",
|
||||
"domain": "algorithmic-trading",
|
||||
"total_skills": 504,
|
||||
@@ -1087,19 +1087,18 @@
|
||||
},
|
||||
{
|
||||
"name": "blue-green-deployment-for-live-strategy-updates",
|
||||
"description": "Institutional quant standards for zero-downtime, state-synchronized blue-green deployments for live trading systems.",
|
||||
"description": "Use when updating a trading strategy that is already live and holding positions \u2014 staging the new version on a standby slot, health-checking it, synchronizing portfolio and open-order state, cutting over order routing atomically, and keeping a guarded rollback path \u2014 without a gap in market coverage or duplicate order risk.",
|
||||
"domain": "algorithmic-trading",
|
||||
"subdomain": "general",
|
||||
"subdomain": "deployment-ops",
|
||||
"tags": [
|
||||
"trading",
|
||||
"algo",
|
||||
"skill"
|
||||
"deployment-ops",
|
||||
"blue-green",
|
||||
"zero-downtime-cutover",
|
||||
"rollback",
|
||||
"state-synchronization"
|
||||
],
|
||||
"brokers_frameworks": [
|
||||
"Python",
|
||||
"Dataclasses"
|
||||
],
|
||||
"version": "1.0.0",
|
||||
"brokers_frameworks": [],
|
||||
"version": "1.1.0",
|
||||
"author": "algo-trading-skills-contributors",
|
||||
"license": "Apache-2.0",
|
||||
"path": "skills/blue-green-deployment-for-live-strategy-updates",
|
||||
|
||||
@@ -1,68 +1,170 @@
|
||||
---
|
||||
name: blue-green-deployment-for-live-strategy-updates
|
||||
description: Institutional quant standards for zero-downtime, state-synchronized blue-green
|
||||
deployments for live trading systems.
|
||||
description: Use when updating a trading strategy that is already live and holding
|
||||
positions — staging the new version on a standby slot, health-checking it, synchronizing
|
||||
portfolio and open-order state, cutting over order routing atomically, and keeping a
|
||||
guarded rollback path — without a gap in market coverage or duplicate order risk.
|
||||
domain: algorithmic-trading
|
||||
subdomain: general
|
||||
subdomain: deployment-ops
|
||||
tags:
|
||||
- trading
|
||||
- algo
|
||||
- skill
|
||||
brokers_frameworks:
|
||||
- Python
|
||||
- Dataclasses
|
||||
version: "1.0.0"
|
||||
- deployment-ops
|
||||
- blue-green
|
||||
- zero-downtime-cutover
|
||||
- rollback
|
||||
- state-synchronization
|
||||
brokers_frameworks: []
|
||||
version: "1.1.0"
|
||||
author: algo-trading-skills-contributors
|
||||
license: Apache-2.0
|
||||
---
|
||||
|
||||
# Blue-Green Deployment for Live Strategy Updates
|
||||
|
||||
## Overview
|
||||
This skill provides robust, institutional-grade standards and implementation patterns for deploying quantitative trading strategies via blue-green deployment. This approach guarantees zero-downtime cutovers, continuous market data ingestion, and rapid, state-aware rollbacks.
|
||||
|
||||
## Objectives
|
||||
- **Zero Market Disruption:** Ensure continuous trading presence and uninterrupted market data processing.
|
||||
- **Risk Mitigation:** Run extensive risk and health checks on new strategy instances (Green) before transferring live order routing.
|
||||
- **Atomic Cutovers:** Transfer live portfolio state and order routing instantaneously to the new version.
|
||||
- **Immediate Rollback:** Retain the prior version (Blue) in a standby/draining state to allow sub-second rollback if anomalies are detected.
|
||||
|
||||
## Structure
|
||||
- `scripts/blue_green_deployer.py`: The robust implementation with state syncing and thread safety.
|
||||
- `scripts/test_blue_green_deployer.py`: Thorough unit tests validating cutover and rollback paths.
|
||||
- `references/workflows.md`: Detailed CI/CD to live trading deployment pipelines.
|
||||
- `references/standards.md`: Quant-specific engineering requirements for zero-gap state transitions.
|
||||
- `assets/checklist.md`: The critical path checklist for trading operations teams.
|
||||
|
||||
## Usage
|
||||
Consult the `references` for architectural standards, and utilize `scripts/blue_green_deployer.py` to manage active vs inactive slots in your quantitative execution engine.
|
||||
|
||||
|
||||
## When to Use
|
||||
|
||||
Documentation for When to Use.
|
||||
Invoke this skill when a strategy that is **currently routing live orders and holding
|
||||
positions** must be replaced with a new build, and stopping it until the next maintenance
|
||||
window is not acceptable. Two instances (BLUE and GREEN) run side by side; exactly one of
|
||||
them holds routing authority at any instant, and the handover carries the book with it.
|
||||
|
||||
Use it when: the strategy holds overnight or intraday inventory that cannot be flattened
|
||||
for the deploy; market coverage gaps cost money (market making, hedging, continuous
|
||||
execution); or you need a rehearsed, sub-minute path back to the previous version.
|
||||
|
||||
Do **not** use it when:
|
||||
|
||||
- The book is flat and the venue is closed. A plain restart is simpler, and simpler is
|
||||
safer — blue-green exists to solve a problem you do not have in that window.
|
||||
- The change is configuration or a risk-parameter update that your system can reload
|
||||
without a new process. Do not spin up a second instance to change a number.
|
||||
- You cannot actually afford two live instances. Two copies mean double the market-data
|
||||
subscriptions, double the hardware, and — during the drain window — double the licence
|
||||
and colocation footprint.
|
||||
- Your broker or venue session does not permit it. Many retail broker APIs authenticate
|
||||
one session per account, or allow only one order-placing connection at a time; verify
|
||||
your broker's concurrency rules before assuming two instances can coexist, because a
|
||||
second login that silently invalidates the first one turns a zero-downtime deploy into
|
||||
an outage with an open position.
|
||||
- What you actually need is to *stop*. If the running strategy is misbehaving right now,
|
||||
the primitive is the kill switch (`kill-switch-and-drawdown-circuit-breakers`), not a
|
||||
deployment.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Documentation for Prerequisites.
|
||||
|
||||
- Two isolated strategy instances, both able to consume market data independently, with
|
||||
resource isolation so GREEN's startup does not degrade BLUE's live trading.
|
||||
- A **health check that proves fitness to trade**, not just that a process is running:
|
||||
market data flowing and current, order gateway reachable, risk limits loaded, signal
|
||||
output within sane bounds.
|
||||
- A **state synchronization routine** that transfers positions, open/working orders and
|
||||
live alpha state from the outgoing instance to the incoming one, and reports failure
|
||||
honestly rather than partially succeeding in silence.
|
||||
- Idempotent client order IDs (`order-placement-idempotency`). A cutover is exactly the
|
||||
moment two instances can both believe they own an in-flight order.
|
||||
- A working kill switch (`kill-switch-and-drawdown-circuit-breakers`). It is the correct
|
||||
fallback whenever rollback is refused, and this skill's workflow depends on it existing.
|
||||
- An execution layer that gates **every** submission on the routing pointer, by calling
|
||||
`BlueGreenDeployer.is_authorised_to_route(color)` — not a cached boolean, not a direct
|
||||
read of the `slots` dict.
|
||||
- A named person authorizing the deployment. For EU investment firms this is mandatory,
|
||||
not good practice — see `references/standards.md`.
|
||||
|
||||
## Workflow
|
||||
|
||||
Documentation for Workflow.
|
||||
1. **Record the baseline.** Construct the deployer with `initial_version` set to the
|
||||
version already live. A history that starts at the first deployment cannot say what
|
||||
was running before it, which makes the pre-existing version unidentifiable in a
|
||||
post-incident review.
|
||||
2. **Stage the new version on the standby slot** with `deploy_to_inactive(version,
|
||||
authorised_by=...)`. It refuses to write over a `DRAINING` slot — that slot is the
|
||||
last-known-good rollback target, and overwriting it is how teams discover mid-incident
|
||||
that they have nothing to roll back to.
|
||||
3. **Shadow-run and health-check.** GREEN consumes live market data and generates signals
|
||||
but emits no orders. Warmup (JIT, risk-model compilation, cache fill) runs outside the
|
||||
deployer's lock so an emergency rollback of the live slot is never blocked behind it.
|
||||
A failed health check is a normal outcome: the slot lands in `FAILED`, which is
|
||||
redeployable, and the pipeline is not stuck.
|
||||
4. **Cut over** with `cutover(authorised_by=...)`. Health is re-checked *at the instant of
|
||||
the swap*, then state is synchronized, then the pointer moves — all in one critical
|
||||
section, so there is no window where both slots or neither slot is authorized.
|
||||
5. **If cutover is refused, classify before retrying.** A failed pre-cutover health check
|
||||
means the staged instance degraded while parked; investigate the instance. A failed
|
||||
*state sync* is different: the sync may have partially applied, so the slot is
|
||||
quarantined as `FAILED` deliberately. Do not retry the cutover onto a possibly torn
|
||||
book — redeploy the version cleanly instead.
|
||||
6. **Observe GREEN under real flow** before touching anything else. BLUE stays `DRAINING`:
|
||||
no signal generation, but still holding last-known-good state as the rollback target.
|
||||
7. **If GREEN misbehaves, roll back** with `rollback(authorised_by=...)`. This is not a
|
||||
pointer swap: BLUE's book is stale by however long GREEN was live, so state is
|
||||
reconciled *backwards* from GREEN to BLUE first, and the rollback is refused if that
|
||||
reconciliation fails or if BLUE is not a viable target.
|
||||
8. **If rollback is refused, use the kill switch — do not reach for `force=True`.**
|
||||
Refusal means the target's book cannot be trusted; routing live flow to a strategy
|
||||
with a wrong position does more damage than staying put. Halt, cancel working orders,
|
||||
reconcile against the broker by hand, and only then consider a forced rollback. Every
|
||||
guard a forced rollback overrides is named in the audit record.
|
||||
9. **Decommission only after a full observation window** with `decommission_standby(
|
||||
authorised_by=...)`. This is the deliberate end of the drain window: after it returns
|
||||
there is no rollback target until a new version has been deployed and cut over. Weigh
|
||||
it against reclaiming the hardware, and record who accepted that trade.
|
||||
|
||||
> Full step-by-step procedure, including the shadow-mode and state-transfer mechanics:
|
||||
> see `references/workflows.md`.
|
||||
> Regulatory touchpoints and engineering standards: see `references/standards.md`.
|
||||
> Printable cutover sign-off checklist: see `assets/checklist.md`.
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
Documentation for Common Pitfalls.
|
||||
|
||||
- **Treating rollback as a bare pointer swap.** Once GREEN has traded, BLUE's positions
|
||||
and open orders are stale. Swapping back without reconciling hands the market to a
|
||||
strategy that believes it holds a position it no longer has — and it will "correct"
|
||||
that phantom position with real orders.
|
||||
- **Assuming a hot standby has a current book because it consumes market data.** Market
|
||||
data does not carry the live instance's fills. A standby that is not also consuming the
|
||||
execution/drop-copy stream has a stale position the moment the live instance trades.
|
||||
- **Trusting a `READY` stamp from minutes ago.** Between staging and cutover the staged
|
||||
instance can lose its market data feed, drift out of risk limits, or die outright.
|
||||
Re-check health inside the same critical section as the swap.
|
||||
- **Retrying a cutover after a failed state sync.** The failure may have left the target
|
||||
with a partially applied book; retrying cuts over onto it.
|
||||
- **Deploying over the `DRAINING` slot to "save a step".** That slot is the rollback
|
||||
target. Releasing it must be an explicit, attributable act.
|
||||
- **Decommissioning as soon as the cutover succeeds.** The failure modes that matter —
|
||||
a bad fill model, a signal that only misbehaves at the close, a slow memory leak —
|
||||
show up later in the session, after you have thrown away the way back.
|
||||
- **Cutting over with a large working-order book.** Orders BLUE placed are live at the
|
||||
venue; if the sync does not hand them over precisely, GREEN will not manage them and
|
||||
they become orphans nobody cancels. Prefer a quiet moment with few resting orders.
|
||||
- **Letting the execution layer cache "am I live?".** Both instances then believe they
|
||||
are authorized across the swap and duplicate submissions. Gate every submission on
|
||||
`is_authorised_to_route()`.
|
||||
- **Unbounded health or sync callbacks.** `cutover()` and `rollback()` hold the lock
|
||||
across those callbacks so the swap is atomic — which means a hung callback stalls
|
||||
order submission itself, not just deployment. Bound both with your own timeout.
|
||||
- **Using `force=True` routinely.** It exists for the case where a human has already
|
||||
reconciled the book by other means. Habitual use converts every guard in this skill
|
||||
into decoration.
|
||||
|
||||
## Verification
|
||||
|
||||
Documentation for Verification.
|
||||
|
||||
- Run the unit suite and confirm every test passes:
|
||||
`python -m unittest discover -s skills/blue-green-deployment-for-live-strategy-updates/scripts`
|
||||
- Assert the safety invariant continuously in staging: at every instant, exactly one slot
|
||||
returns `True` from `is_authorised_to_route()` — never zero, never two — including
|
||||
while a cutover is in flight.
|
||||
- Run fault-injection drills against a paper/simulated venue, not just the happy path:
|
||||
kill the staged instance between `READY` and cutover (cutover must abort); fail the
|
||||
state sync (slot must quarantine, routing must not move); fail the backward sync during
|
||||
rollback (rollback must be refused and point you at the kill switch).
|
||||
- Reconcile positions against the broker immediately after every cutover and rollback.
|
||||
The deployer moves routing; it cannot confirm the venue agrees with your book.
|
||||
- Export `deployment_history` after each drill and confirm each entry carries an
|
||||
authorizer, that refused operations are present (not just successful ones), and that
|
||||
any forced rollback is flagged `forced` with the overridden guard named.
|
||||
|
||||
## Related Skills
|
||||
|
||||
Documentation for Related Skills.
|
||||
- `kill-switch-and-drawdown-circuit-breakers`
|
||||
- `order-placement-idempotency`
|
||||
- `paper-to-live-promotion-checklist`
|
||||
- `multi-region-failover-for-broker-connectivity`
|
||||
- `structured-logging-for-post-incident-forensics`
|
||||
- `mifid-ii-algo-trading-compliance-eu`
|
||||
|
||||
@@ -1,24 +1,65 @@
|
||||
# Blue-Green Deployment Checklist
|
||||
# Blue-Green Cutover Sign-Off Checklist
|
||||
|
||||
## Pre-Deployment (Staging)
|
||||
- [ ] Verify target binary / strategy version passes offline backtest reconciliation.
|
||||
- [ ] Confirm sufficient hardware capacity on target trading servers (NUMA cores, memory).
|
||||
- [ ] Lock risk parameters for the new deployment.
|
||||
Deployment ID: ____________ Date/Time (UTC): ____________
|
||||
Outgoing version: ____________ Incoming version: ____________
|
||||
Authorized by (named person): ____________
|
||||
|
||||
## Shadow Deployment (Green)
|
||||
- [ ] Spin up Green instance in shadow mode.
|
||||
- [ ] Confirm Green is successfully consuming market data feeds.
|
||||
- [ ] Verify zero outgoing FIX/binary messages from Green.
|
||||
- [ ] Wait for Green JIT warmup and initial signal generation cycles.
|
||||
- [ ] Green instance passes real-time risk checks.
|
||||
## Pre-deployment (staging)
|
||||
|
||||
## Synchronization & Cutover
|
||||
- [ ] Trigger atomic portfolio and state sync from Blue to Green.
|
||||
- [ ] Validate Green's internalized portfolio matches Blue's exactly.
|
||||
- [ ] Execute execution gateway pointer swap (Blue -> Green).
|
||||
- [ ] Monitor first 5 minutes of Green's order submissions.
|
||||
- [ ] Named authorizer recorded above and passed as `authorised_by`. **EU/UK investment
|
||||
firms: this is mandatory (RTS 6 Art. 5(2)), not a formality.**
|
||||
- [ ] Incoming build passes offline backtest reconciliation against the outgoing build.
|
||||
- [ ] Broker/venue session concurrency confirmed: two simultaneous instances will not
|
||||
invalidate each other's session or order-placing connection.
|
||||
- [ ] Sufficient hardware capacity confirmed (cores, NUMA placement, memory, feed
|
||||
handler capacity) to run both instances without degrading the live one.
|
||||
- [ ] Risk parameters for the incoming version locked and reviewed.
|
||||
- [ ] Kill switch tested and confirmed available for this session — it is the fallback if
|
||||
rollback is refused.
|
||||
- [ ] Rollback decision criteria written down *before* cutover (what specifically
|
||||
triggers a rollback, and who decides).
|
||||
|
||||
## Post-Cutover
|
||||
- [ ] If issues detected: TRIGGER EMERGENCY ROLLBACK.
|
||||
- [ ] If stable for 30 minutes: Decommission Blue instance.
|
||||
- [ ] Document deployment artifacts and performance metrics.
|
||||
## Shadow deployment (green)
|
||||
|
||||
- [ ] Green instance running in shadow mode.
|
||||
- [ ] **Zero outgoing order messages from Green verified at the gateway**, not only in
|
||||
application config.
|
||||
- [ ] Green consuming market data: flowing, current, no unrecovered sequence gaps.
|
||||
- [ ] JIT warmup and initial signal-generation cycles complete.
|
||||
- [ ] Green passes real-time risk checks against the *current live* portfolio, not an
|
||||
empty book.
|
||||
- [ ] Ingress-to-signal latency measured against SLA for this build.
|
||||
- [ ] `deploy_to_inactive()` returned `READY` (not `FAILED`).
|
||||
|
||||
## Synchronization and cutover
|
||||
|
||||
- [ ] Working-order book reviewed; cutover timed for a quiet moment where practical.
|
||||
- [ ] Outgoing instance quiesced per the defined procedure before snapshot.
|
||||
- [ ] State sync transfers positions, working orders (with venue order IDs) and dynamic
|
||||
alpha state; receiving instance **acknowledges ingestion**.
|
||||
- [ ] Green's internalized portfolio validated against Blue's exactly before routing moves.
|
||||
- [ ] Health check and state sync both bounded by an explicit timeout.
|
||||
- [ ] `cutover()` returned success. If refused, refusal classified (not `READY` /
|
||||
health re-check failed / **state sync failed → redeploy, do not retry cutover**).
|
||||
- [ ] Exactly one slot returns `True` from `is_authorised_to_route()` after the swap.
|
||||
- [ ] Positions reconciled against the broker immediately post-cutover.
|
||||
- [ ] Controlled-deployment limits applied for the observation window (cautious caps on
|
||||
instruments, order value/count, positions) and monitoring intensified — EU/UK firms:
|
||||
RTS 6 Art. 8.
|
||||
- [ ] First minutes of Green's order submissions monitored against expected behaviour.
|
||||
|
||||
## Post-cutover
|
||||
|
||||
- [ ] Blue confirmed `DRAINING` and still viable as a rollback target.
|
||||
- [ ] Observation window length chosen from when this strategy's failure modes actually
|
||||
surface (not a default five minutes), and end-of-window review scheduled.
|
||||
- [ ] If issues detected: **rollback**. If rollback is refused → kill switch, cancel
|
||||
working orders, reconcile manually. Do **not** reach for `force=True` first.
|
||||
- [ ] Any forced rollback: overridden guard(s) named in the audit record, second person
|
||||
informed, incident review opened.
|
||||
- [ ] Positions reconciled against the broker before ending the observation window.
|
||||
- [ ] `decommission_standby(authorised_by=...)` executed only after the window closed,
|
||||
with the loss of rollback capability explicitly accepted and attributed.
|
||||
- [ ] `deployment_history` exported and retained: authorizer present on each action,
|
||||
refused operations included, forced overrides flagged.
|
||||
- [ ] Deployment artifacts and performance metrics documented.
|
||||
|
||||
@@ -1,17 +1,126 @@
|
||||
# Institutional Quant Standards for Blue-Green Deployments
|
||||
# Standards for Blue-Green Deployments of Live Trading Strategies
|
||||
|
||||
## 1. Zero-Gap Data Continuity
|
||||
- **Mandate:** Strategy instances MUST NOT drop a single packet of market data during cutover.
|
||||
- **Implementation:** Both Blue and Green must process multicast A/B feeds concurrently prior to cutover. State sync handles the delta between Blue's processed state and Green's initialization.
|
||||
## 0. How to read this document
|
||||
|
||||
## 2. Order Safety & idempotency
|
||||
- **Mandate:** Cutover must not result in duplicate orders or dropped fills.
|
||||
- **Implementation:** The execution gateway must enforce order ID idempotency. Any in-flight orders generated by Blue just prior to cutover must have their fills correctly routed back to Green's position manager.
|
||||
Sections 1-2 are **regulatory touchpoints**: obligations that exist in law for firms in
|
||||
scope, with the jurisdiction stated. Sections 3-7 are **engineering standards**: they are
|
||||
this repository's recommended practice, not legal requirements, and are labelled as such
|
||||
so an agent does not present them to an operator as compliance mandates.
|
||||
|
||||
## 3. Instantaneous Reversibility
|
||||
- **Mandate:** If Green triggers a risk limit breach or generates malformed predictions, the system must revert to Blue in under 1 millisecond.
|
||||
- **Implementation:** Blue remains actively updating its state from market data, merely ignoring its own signal generation output. Rollback is a simple pointer swap.
|
||||
Nothing here is a substitute for your own compliance function's determination of which
|
||||
regime applies to you.
|
||||
|
||||
## 4. Hardware Resource Isolation
|
||||
- **Mandate:** Green's initialization must not cause CPU cache thrashing or network congestion that impacts Blue's live trading performance.
|
||||
- **Implementation:** Deploy Green to dedicated pinned CPU cores on the NUMA node separate from Blue, mapping only required shared memory.
|
||||
## 1. EU / MiFID II — Commission Delegated Regulation (EU) 2017/589 ("RTS 6")
|
||||
|
||||
**Applicability:** investment firms engaged in algorithmic trading that are authorized
|
||||
under MiFID II (Directive 2014/65/EU). It does **not** apply to a US-only broker-dealer,
|
||||
to an unregulated proprietary trader outside the EU, or to a retail individual trading
|
||||
their own capital. The UK operates a materially equivalent onshored version of RTS 6,
|
||||
supervised by the FCA. Do not apply these obligations to a firm outside their scope.
|
||||
|
||||
| RTS 6 Article | Title | Relevance to this skill |
|
||||
|---|---|---|
|
||||
| Art. 5(2) | General methodology | A person designated by the firm's senior management must authorize the deployment or substantial update of an algorithmic trading system, trading algorithm or algorithmic trading strategy. This is what `authorised_by` on `deploy_to_inactive()` and `cutover()` exists to capture — an unattributed deployment does not satisfy it. |
|
||||
| Art. 5(7) | General methodology | The firm must keep records of any material change made to the software used for algorithmic trading, allowing it to determine: when the change was made, the person that made it, the person that approved it, and the nature of the change. `deployment_history` is structured to answer exactly those four questions — hence `timestamp`, `authorised_by`, `action`/`detail`, and the `forced` flag that distinguishes an overridden rollback from a clean one. |
|
||||
| Art. 8 | Controlled deployment of algorithms | Introducing a new or materially modified algorithm into production must be controlled, including cautious limits (instruments traded, price, order value and count, positions, markets involved) and more intensive monitoring of the algorithm's activity. A blue-green cutover is a controlled deployment mechanism, but the *pointer swap alone does not satisfy this article* — the limits and heightened monitoring during the drain window are a separate obligation. |
|
||||
| Art. 11 | Management of material changes | Post-deployment management of material changes to trading systems and algorithms. A strategy version bump routed to live capital is the material change this article governs. |
|
||||
| Art. 12 | Kill functionality | The firm must be able to cancel immediately, as an emergency measure, any or all of its unexecuted orders at any or all venues it is connected to, and must be able to attribute each order to the responsible algorithm, trader, desk or client. This is the mandated fallback when `rollback()` refuses — halting is a regulated capability; a forced rollback onto an unreconciled book is not. |
|
||||
|
||||
Primary text: Commission Delegated Regulation (EU) 2017/589, EUR-Lex ELI
|
||||
<https://eur-lex.europa.eu/eli/reg_del/2017/589/oj/eng>. The UK onshored text is at
|
||||
<https://www.legislation.gov.uk/eur/2017/589> and in the FCA Handbook technical standards.
|
||||
|
||||
## 2. US — SEC Rule 15c3-5 (Market Access Rule)
|
||||
|
||||
**Applicability:** broker-dealers with market access, including those providing market
|
||||
access to customers. It binds the *broker-dealer*, not the strategy author — but it
|
||||
constrains any deployment that touches the order path.
|
||||
|
||||
- Financial and regulatory risk-management controls must be applied on an **automated,
|
||||
pre-trade** basis, and must be under the **direct and exclusive control** of the
|
||||
broker-dealer with market access.
|
||||
- Consequence for this skill: a cutover must never move order flow onto a path where
|
||||
those controls are absent, weaker, or owned by the strategy instance rather than the
|
||||
broker-dealer. "The new version has its own risk checks" does not satisfy the rule.
|
||||
Treat the pre-trade control layer as *outside* the blue-green swap, not as part of the
|
||||
strategy artifact being swapped.
|
||||
- The rule also requires at least annual review of the effectiveness of these controls,
|
||||
with certification by the CEO or equivalent officer.
|
||||
|
||||
Primary text: 17 CFR 240.15c3-5; adopting release SEC Rel. No. 34-63241
|
||||
<https://www.sec.gov/files/rules/final/2010/34-63241.pdf>. Staff FAQs:
|
||||
<https://www.sec.gov/rules-regulations/staff-guidance/trading-markets-frequently-asked-questions/divisionsmarketregfaq-0>.
|
||||
|
||||
## 3. Engineering standard — market data continuity
|
||||
|
||||
*Recommended practice, not a regulatory requirement.*
|
||||
|
||||
- The incoming instance should be consuming and processing live market data, and be
|
||||
caught up, **before** it is eligible for cutover — not started at the moment routing
|
||||
moves. Where the venue publishes redundant A/B multicast feeds, both instances should
|
||||
arbitrate across them independently.
|
||||
- Gap detection belongs in the health check: an instance whose sequence numbers show an
|
||||
unrecovered gap is not fit to trade regardless of process liveness.
|
||||
|
||||
## 4. Engineering standard — order safety across the handover
|
||||
|
||||
*Recommended practice, not a regulatory requirement.*
|
||||
|
||||
- Client order IDs must be idempotent across the cutover so that an order in flight when
|
||||
routing moves cannot be submitted twice under two different identities. See
|
||||
`order-placement-idempotency`.
|
||||
- Orders BLUE placed remain live at the venue after the swap. Ownership of those working
|
||||
orders must transfer explicitly as part of state synchronization; anything the incoming
|
||||
instance does not know about becomes an orphan that nobody cancels or manages.
|
||||
- Fills for BLUE-originated orders can arrive *after* the swap. The execution/drop-copy
|
||||
consumer must route them into the now-live instance's position manager, keyed by order
|
||||
ID, rather than to whichever instance submitted them.
|
||||
|
||||
## 5. Engineering standard — reversibility
|
||||
|
||||
*Recommended practice, not a regulatory requirement.*
|
||||
|
||||
- **Rollback is not a pointer swap.** The moment GREEN sends its first order, BLUE's book
|
||||
is stale. Restoring routing to BLUE without reconciling the delta gives control to a
|
||||
strategy holding a wrong position, which it will then trade against. `rollback()`
|
||||
therefore reconciles state backwards from the live slot to the target and refuses the
|
||||
rollback when that reconciliation fails.
|
||||
- **A hot standby is not automatically a current standby.** Keeping BLUE subscribed to
|
||||
market data while suppressing its signal output keeps its *market* view fresh; it does
|
||||
nothing for its *position* view. Fills are not in the market data feed. If you want a
|
||||
standby whose book stays current, it must consume the execution report / drop-copy
|
||||
stream as well — and that is a design decision with its own reconciliation failure
|
||||
modes, not a free property of running two instances.
|
||||
- **Do not publish a rollback latency target that the reconciliation step cannot meet.**
|
||||
Rollback time is dominated by state reconciliation and by the venue round trip for any
|
||||
corrective orders, not by the pointer write. Measure it in your own environment before
|
||||
committing to a number in a runbook or an SLA.
|
||||
- **When rollback is refused, halt — do not force.** The regulated, rehearsed primitive
|
||||
for "stop trading now" is the kill switch (EU firms: RTS 6 Art. 12). Forcing routing
|
||||
onto a slot with an unreconciled book is strictly worse than being flat.
|
||||
|
||||
## 6. Engineering standard — resource isolation
|
||||
|
||||
*Recommended practice, not a regulatory requirement.*
|
||||
|
||||
- The incoming instance's initialization (JIT compilation, cache warming, risk-model
|
||||
construction, historical replay) is CPU- and memory-intensive and must not degrade the
|
||||
live instance. On latency-sensitive systems this usually means dedicated pinned cores,
|
||||
ideally on a separate NUMA node, with only the intended shared-memory regions mapped.
|
||||
- Quantify the isolation before relying on it. "Separate process" is not isolation if the
|
||||
two instances contend for the same cores, the same NIC queues, or the same feed handler.
|
||||
|
||||
## 7. Engineering standard — auditability
|
||||
|
||||
*Recommended practice, which for in-scope EU firms also carries the Art. 5(7) obligation
|
||||
in Section 1.*
|
||||
|
||||
- Record refused operations, not only successful ones. A rollback that was blocked is
|
||||
precisely the event a post-incident review needs to find.
|
||||
- Record overrides distinguishably. A forced rollback that reads like a clean one in the
|
||||
log defeats the purpose of having the guard.
|
||||
- Record ordering: the deployment history's list order is authoritative. Wall-clock
|
||||
timestamps are for human reading and can move backwards across an NTP correction.
|
||||
- Retention is set by your applicable regime (EU firms should consult their MiFID II
|
||||
record-keeping retention requirements with their compliance function); this skill does
|
||||
not assert a retention period.
|
||||
|
||||
@@ -1,24 +1,139 @@
|
||||
# Blue-Green Deployment Workflows for Live Trading
|
||||
|
||||
## 1. Strategy Version Staging (Green Deployment)
|
||||
- **Deployment:** The new version of the strategy (Green) is instantiated in an isolated memory space.
|
||||
- **Warmup:** Green subscribes to live market data (multicast feeds) but operates in *shadow mode* (no outgoing FIX/binary order messages).
|
||||
- **Compilation:** JIT caches are warmed, risk models are instantiated.
|
||||
The deployer in `scripts/blue_green_deployer.py` owns exactly one thing: **which slot is
|
||||
authorized to route orders**. It does not start processes, move positions, or cancel
|
||||
orders. Everything below describes the surrounding procedure that the two callbacks —
|
||||
`health_check_fn` and `state_sync_fn` — must actually implement.
|
||||
|
||||
## 2. Validation & Health Checks
|
||||
- **Risk Verification:** In-memory risk constraints are run against current portfolio state.
|
||||
- **Latency Testing:** Market data ingress-to-signal latency is measured to ensure the new build meets microsecond SLA.
|
||||
- **Dependency Checks:** Connectivity to order gateways and reference data APIs is confirmed.
|
||||
## 0. Slot lifecycle
|
||||
|
||||
## 3. State Synchronization (The Critical Path)
|
||||
- Active (Blue) strategy initiates a momentary order pause or snapshot sequence.
|
||||
- Live position state, open order books, and dynamic alpha features are synchronized to Green via zero-copy shared memory or extremely fast IPC.
|
||||
- Green acknowledges full state ingestion.
|
||||
```
|
||||
IDLE ──deploy──> DEPLOYING ──health ok──> READY ──cutover──> LIVE
|
||||
^ │ │ │
|
||||
│ └──health fail──> FAILED <──sync fail────┘
|
||||
│ │ (quarantine)
|
||||
└────decommission──── DRAINING <────cutover (outgoing slot)
|
||||
```
|
||||
|
||||
## 4. Atomic Cutover
|
||||
- Execution engine atomically swaps the routing pointer from Blue to Green.
|
||||
- Green transitions from *shadow* to *live* order generation.
|
||||
- Blue transitions to *draining*, pausing new signal generation but maintaining data ingestion to serve as an instant rollback target.
|
||||
- `DRAINING` is the last-known-good rollback target. `deploy_to_inactive()` refuses to
|
||||
overwrite it; releasing it requires an explicit `decommission_standby()`.
|
||||
- `FAILED` is redeployable but is **never** a rollback target.
|
||||
- `READY` is also refused for redeployment, so an un-cut-over deployment is not silently
|
||||
replaced by another.
|
||||
|
||||
## 5. Drain & Decommission
|
||||
- Once Green establishes stability (e.g., 5-30 minutes), Blue is fully spun down to reclaim hardware resources.
|
||||
## 1. Strategy version staging (green deployment)
|
||||
|
||||
- Instantiate the new version in an isolated memory space, on isolated cores (see
|
||||
`references/standards.md` §6).
|
||||
- GREEN subscribes to live market data and runs in **shadow mode**: it computes signals
|
||||
and maintains its own view, but emits no outgoing FIX/binary order messages. Verify the
|
||||
absence of outgoing order messages at the gateway, not just in application config — a
|
||||
shadow instance that is actually wired to the venue is the worst possible outcome.
|
||||
- Warm JIT caches, instantiate risk models, and replay whatever history the strategy
|
||||
needs to have a meaningful state.
|
||||
- Call `deploy_to_inactive(version, authorised_by=...)`. Warmup and the health check
|
||||
deliberately run **outside** the deployer's lock, so an emergency rollback of the
|
||||
currently live slot is never queued behind a slow warmup.
|
||||
|
||||
## 2. Validation and health checks
|
||||
|
||||
`health_check_fn(color)` must answer "is this instance fit to trade *right now*", and it
|
||||
is called twice: once after warmup, and again inside the cutover critical section. It
|
||||
should verify at minimum:
|
||||
|
||||
- **Market data**: subscribed, flowing, no unrecovered sequence gaps, timestamps current.
|
||||
- **Risk**: in-memory risk constraints loaded and evaluated against the *current* live
|
||||
portfolio state, not an empty book.
|
||||
- **Latency**: market-data-ingress-to-signal latency measured against whatever SLA the
|
||||
strategy is held to. Measure it; do not assume the new build kept the old build's
|
||||
profile.
|
||||
- **Dependencies**: order gateway session established, reference data loaded, clock in
|
||||
sync.
|
||||
|
||||
An exception raised inside the callback is treated as a failed check, never as a pass.
|
||||
The slot lands in `FAILED`, which is redeployable — a raising health check does not strand
|
||||
the pipeline.
|
||||
|
||||
## 3. State synchronization (the critical path)
|
||||
|
||||
`state_sync_fn(source, target)` must transfer, and the receiving instance must
|
||||
acknowledge:
|
||||
|
||||
- **Positions** per instrument, with the source's own view of average cost.
|
||||
- **Open/working orders**, including venue order IDs, so the incoming instance can manage
|
||||
and cancel orders it did not place.
|
||||
- **Dynamic alpha/working state** — rolling features, inventory targets, schedule
|
||||
progress for a parent order — anything whose loss would make the new instance restart a
|
||||
execution schedule from scratch.
|
||||
|
||||
Practical requirements:
|
||||
|
||||
- Quiesce briefly and deliberately: the outgoing instance should stop *originating* new
|
||||
orders and take a consistent snapshot, rather than snapshotting a book that is moving
|
||||
underneath it. Define what "quiesce" means concretely in your system (typically: stop
|
||||
new order origination, let in-flight acknowledgements settle to a known state, then
|
||||
snapshot) and bound how long it may last.
|
||||
- Transfer over shared memory or fast IPC as appropriate, but treat **acknowledged
|
||||
ingestion** — not transmission — as success.
|
||||
- Return `False` on partial application. Do not report success on a half-transferred book;
|
||||
the deployer's guarantee is only as good as this return value.
|
||||
- The sync runs **inside** the deployer's lock, so it also blocks
|
||||
`is_authorised_to_route()` and therefore order submission. That is intended — no order
|
||||
should be sent while the book is mid-transfer — but it means an unbounded sync stalls
|
||||
trading. Bound it with your own timeout.
|
||||
|
||||
## 4. Atomic cutover
|
||||
|
||||
`cutover(authorised_by=...)` performs, in one critical section:
|
||||
|
||||
1. Confirm the standby is `READY`.
|
||||
2. **Re-run the health check.** The `READY` stamp may be minutes old.
|
||||
3. Run `state_sync_fn(live, standby)`.
|
||||
4. Swap the routing pointer; standby becomes `LIVE`, the outgoing slot becomes `DRAINING`.
|
||||
|
||||
Because all four happen under one lock, there is no interval in which both slots or
|
||||
neither slot is authorized.
|
||||
|
||||
**If it is refused, classify before acting:**
|
||||
|
||||
| Refusal | Meaning | Correct next step |
|
||||
|---|---|---|
|
||||
| Slot is not `READY` | Nothing staged, or a previous attempt quarantined it | Deploy the version again |
|
||||
| Pre-cutover health re-check failed | The staged instance degraded while parked | Investigate the instance; the slot is now `FAILED` and redeployable |
|
||||
| State sync failed | The transfer may be **partially applied** | Do **not** retry the cutover. The slot is quarantined as `FAILED` by design; redeploy cleanly |
|
||||
|
||||
After a successful cutover, apply RTS 6 Art. 8-style controlled-deployment limits if in
|
||||
scope: cautious caps on instruments, order value/count and positions, plus heightened
|
||||
monitoring for the observation window. The pointer swap is not the whole obligation.
|
||||
|
||||
## 5. Rollback
|
||||
|
||||
`rollback(authorised_by=...)` is **not** the inverse pointer swap of a cutover.
|
||||
|
||||
1. Confirm the target is a viable slot (`DRAINING` or `READY`). `IDLE` (never deployed or
|
||||
decommissioned) and `FAILED` (rejected) are refused.
|
||||
2. Run `state_sync_fn(live, target)` — **backwards**, from the currently live instance to
|
||||
the rollback target, because the target's book is stale by the length of the live
|
||||
window.
|
||||
3. Only then move routing. The outgoing (bad) slot becomes `FAILED`, so a second rollback
|
||||
cannot ping-pong routing back into the version just rejected.
|
||||
|
||||
**If rollback is refused, escalate to the kill switch, not to `force=True`.** Cancel
|
||||
working orders, stop submission, reconcile against the broker by hand. `force=True` is
|
||||
reserved for the case where a human has *already* confirmed the target's state by other
|
||||
means; whichever guards it overrides are named in the audit record and logged at CRITICAL.
|
||||
|
||||
Rollback restores *routing*, not *executions*. Fills the bad version already booked at the
|
||||
venue are real, and reconciling them is the state sync's job.
|
||||
|
||||
## 6. Drain and decommission
|
||||
|
||||
- BLUE remains `DRAINING` through the observation window: no signal generation, still
|
||||
holding last-known-good state.
|
||||
- Choose the window length from when your failure modes actually surface — a bad fill
|
||||
model or a close-specific signal bug will not appear in the first five minutes — and
|
||||
reconcile positions against the broker before ending it.
|
||||
- `decommission_standby(authorised_by=...)` is the deliberate end of that window. After it
|
||||
returns there is no rollback target until a new version has been deployed and cut over.
|
||||
It is refused for a `LIVE` slot, and it records who accepted the loss of rollback
|
||||
capability.
|
||||
|
||||
+108
-22
@@ -37,16 +37,21 @@ Limitations the caller must handle:
|
||||
- The deployer cannot interrupt ``health_check_fn`` or ``state_sync_fn``.
|
||||
``cutover()`` and ``rollback()`` hold the internal lock while those callbacks
|
||||
run so the swap is atomic, which means a hung callback blocks every other
|
||||
deployer operation. Bound them with your own timeout.
|
||||
deployer operation — including ``is_authorised_to_route()``, and therefore
|
||||
order submission itself. Blocking submission for the duration of the handover
|
||||
is the intended semantics (no order may be sent while the book is mid-transfer),
|
||||
but it only stays safe if you bound both callbacks with your own timeout.
|
||||
- Enforcement of "only the LIVE slot may route" belongs to the execution layer:
|
||||
it must consult the deployer before every submission and reject orders from a
|
||||
slot that is not ``SlotState.LIVE``.
|
||||
it must call ``is_authorised_to_route(color)`` before every submission and
|
||||
reject orders from a slot that is not authorised. Do not read ``slots``
|
||||
directly for this — that dict is mutable shared state and reading it outside
|
||||
the lock can observe a half-completed cutover.
|
||||
- Rollback restores *routing*, not *executions*. Fills GREEN already booked at
|
||||
the venue are real; reconciliation is ``state_sync_fn``'s job.
|
||||
|
||||
See ``references/standards.md`` for the MiFID II RTS 6 authorisation and
|
||||
change-record obligations that ``authorised_by`` and ``deployment_history``
|
||||
are designed to satisfy.
|
||||
are designed to satisfy, and for the jurisdictional limits of those obligations.
|
||||
"""
|
||||
import dataclasses
|
||||
import logging
|
||||
@@ -105,6 +110,12 @@ class DeploymentSlot:
|
||||
version: str = ""
|
||||
health_ok: bool = False
|
||||
deploy_time: float = 0.0
|
||||
# Monotonic per-deployer counter identifying *which* deployment attempt owns
|
||||
# this slot. (state, version) alone cannot: a slot decommissioned and then
|
||||
# redeployed with the same version during a warmup window looks unchanged to
|
||||
# the thread that is still warming up the previous instance, which would then
|
||||
# stamp its stale health result onto the new one.
|
||||
deploy_generation: int = 0
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -151,15 +162,20 @@ class BlueGreenDeployer:
|
||||
standby) and on rollback (live -> rollback target). An exception is
|
||||
treated as a failed sync.
|
||||
initial_version: Version already running in the initially-live BLUE
|
||||
slot, recorded in ``deployment_history`` so a rollback to it is
|
||||
identifiable in an audit trail.
|
||||
slot. Written as the opening ``INITIALIZE`` record of
|
||||
``deployment_history`` so the audit trail states which version was
|
||||
live at t0 instead of beginning mid-stream, and so a later rollback
|
||||
to it is identifiable.
|
||||
"""
|
||||
if not isinstance(warmup_seconds, (int, float)) or isinstance(warmup_seconds, bool):
|
||||
raise ValueError("warmup_seconds must be a number")
|
||||
if not math.isfinite(warmup_seconds) or warmup_seconds < 0:
|
||||
raise ValueError(f"warmup_seconds must be finite and >= 0, got {warmup_seconds!r}")
|
||||
if not isinstance(initial_version, str):
|
||||
raise ValueError(f"initial_version must be a string, got {type(initial_version).__name__}")
|
||||
|
||||
self._lock = threading.RLock()
|
||||
self._deploy_seq = 0
|
||||
self.slots: Dict[SlotColor, DeploymentSlot] = {
|
||||
SlotColor.BLUE: DeploymentSlot(
|
||||
color=SlotColor.BLUE,
|
||||
@@ -174,6 +190,16 @@ class BlueGreenDeployer:
|
||||
self.state_sync_fn = state_sync_fn or (lambda _, __: True)
|
||||
self.warmup_seconds = warmup_seconds
|
||||
self.deployment_history: List[Dict[str, object]] = []
|
||||
self._record(
|
||||
"INITIALIZE", SlotColor.BLUE, SlotColor.BLUE, initial_version, True, "",
|
||||
"initial live slot at construction"
|
||||
+ ("" if initial_version else " (no initial_version supplied)"),
|
||||
)
|
||||
if not initial_version:
|
||||
logger.warning(
|
||||
"BlueGreenDeployer constructed without initial_version; the audit trail "
|
||||
"cannot identify which strategy version was live before the first cutover."
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Inspection
|
||||
@@ -195,6 +221,30 @@ class BlueGreenDeployer:
|
||||
with self._lock:
|
||||
return dataclasses.replace(self.slots[self._inactive_color()])
|
||||
|
||||
def is_authorised_to_route(self, color: SlotColor) -> bool:
|
||||
"""
|
||||
Whether ``color`` is the slot currently authorised to submit orders.
|
||||
|
||||
This is the check the execution layer must make before *every* order
|
||||
submission — the deployer only owns the routing pointer, so nothing else
|
||||
stops a draining or failed instance from sending to the venue.
|
||||
|
||||
It deliberately takes the lock rather than reading ``active_slot``
|
||||
unsynchronised: during a cutover or rollback the pointer and the two slot
|
||||
states are updated together, and an unsynchronised read can land between
|
||||
those writes and see either no live slot or two.
|
||||
|
||||
A consequence worth designing for: while ``cutover()`` or ``rollback()``
|
||||
is running its ``state_sync_fn``, this call blocks. That is the intended
|
||||
behaviour — orders must not be submitted while positions and open orders
|
||||
are being handed between instances — but it means an unbounded sync
|
||||
callback stalls order submission, not just deployment.
|
||||
"""
|
||||
if not isinstance(color, SlotColor):
|
||||
raise TypeError(f"color must be a SlotColor, got {type(color).__name__}")
|
||||
with self._lock:
|
||||
return self.active_slot == color and self.slots[color].state is SlotState.LIVE
|
||||
|
||||
def _inactive_color(self) -> SlotColor:
|
||||
with self._lock:
|
||||
return SlotColor.GREEN if self.active_slot == SlotColor.BLUE else SlotColor.BLUE
|
||||
@@ -208,15 +258,25 @@ class BlueGreenDeployer:
|
||||
success: bool,
|
||||
authorised_by: str,
|
||||
detail: str,
|
||||
forced: bool = False,
|
||||
) -> None:
|
||||
"""
|
||||
Append one immutable change record.
|
||||
|
||||
MiFID II RTS 6 Art. 5(7) requires records of material changes to
|
||||
algorithmic trading software identifying when the change was made, who
|
||||
made it, who approved it and its nature. Refused operations are recorded
|
||||
too: an attempted rollback that was blocked is exactly the event a
|
||||
post-incident review needs to see.
|
||||
For EU investment firms, MiFID II RTS 6 Art. 5(7) requires records of
|
||||
material changes to algorithmic trading software identifying when the
|
||||
change was made, who made it, who approved it and its nature. Refused
|
||||
operations are recorded too: an attempted rollback that was blocked is
|
||||
exactly the event a post-incident review needs to see.
|
||||
|
||||
``forced`` records that an operator overrode a safety guard. Without it a
|
||||
forced rollback is indistinguishable from a clean one in the history,
|
||||
which is precisely the "nature of the change" a reviewer must be able to
|
||||
determine.
|
||||
|
||||
``timestamp`` is wall-clock (``time.time()``), not monotonic, because an
|
||||
audit record needs a real date; list order, not the timestamp, is the
|
||||
authoritative ordering of events.
|
||||
"""
|
||||
with self._lock:
|
||||
self.deployment_history.append({
|
||||
@@ -228,6 +288,7 @@ class BlueGreenDeployer:
|
||||
"success": success,
|
||||
"authorised_by": authorised_by,
|
||||
"detail": detail,
|
||||
"forced": forced,
|
||||
})
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
@@ -270,10 +331,13 @@ class BlueGreenDeployer:
|
||||
)
|
||||
|
||||
live_color = self.active_slot
|
||||
self._deploy_seq += 1
|
||||
generation = self._deploy_seq
|
||||
slot.state = SlotState.DEPLOYING
|
||||
slot.version = version
|
||||
slot.health_ok = False
|
||||
slot.deploy_time = time.time()
|
||||
slot.deploy_generation = generation
|
||||
logger.info(f"Deploying quant version '{version}' to {color.value} slot.")
|
||||
|
||||
# Warmup and the health check run OUTSIDE the lock: warmup is unbounded
|
||||
@@ -290,15 +354,19 @@ class BlueGreenDeployer:
|
||||
|
||||
with self._lock:
|
||||
slot = self.slots[color]
|
||||
# Routing may have moved while the lock was released. Writing READY or
|
||||
# FAILED onto a slot that has since gone LIVE would mislabel the
|
||||
# running strategy and could disarm the rollback guard.
|
||||
if (slot.state is not SlotState.DEPLOYING
|
||||
or slot.version != version
|
||||
# Routing may have moved, or the slot may have been released and
|
||||
# redeployed, while the lock was open. Writing READY or FAILED onto a
|
||||
# slot this attempt no longer owns would mislabel the running strategy
|
||||
# and could disarm the rollback guard. The generation check is what
|
||||
# makes this sound: a slot decommissioned and redeployed with the same
|
||||
# version is indistinguishable by (state, version) alone.
|
||||
if (slot.deploy_generation != generation
|
||||
or slot.state is not SlotState.DEPLOYING
|
||||
or color == self.active_slot):
|
||||
logger.warning(
|
||||
f"Deployment of '{version}' to {color.value} abandoned: slot changed during warmup "
|
||||
f"(state={slot.state.value}, version='{slot.version}', active={self.active_slot.value})."
|
||||
f"(state={slot.state.value}, version='{slot.version}', active={self.active_slot.value}, "
|
||||
f"generation={slot.deploy_generation} != {generation})."
|
||||
)
|
||||
self._record("DEPLOY", live_color, color, version, False, authorised_by,
|
||||
"abandoned: slot changed during warmup")
|
||||
@@ -314,7 +382,7 @@ class BlueGreenDeployer:
|
||||
"health check passed" if healthy else "health check failed")
|
||||
return dataclasses.replace(slot)
|
||||
|
||||
def decommission_standby(self) -> DeploymentSlot:
|
||||
def decommission_standby(self, authorised_by: str = "") -> DeploymentSlot:
|
||||
"""
|
||||
Release the standby slot back to ``IDLE``, freeing it for a new version.
|
||||
|
||||
@@ -322,6 +390,12 @@ class BlueGreenDeployer:
|
||||
no rollback target: ``rollback()`` will refuse until a new version has
|
||||
been deployed and cut over. Call it only once the live version has proven
|
||||
stable over a full observation window.
|
||||
|
||||
Args:
|
||||
authorised_by: Person accepting the loss of rollback capability.
|
||||
Recorded in ``deployment_history``; giving up the last-known-good
|
||||
target is an operational decision a post-incident review needs to
|
||||
be able to attribute.
|
||||
"""
|
||||
with self._lock:
|
||||
color = self._inactive_color()
|
||||
@@ -335,6 +409,7 @@ class BlueGreenDeployer:
|
||||
slot.version = ""
|
||||
slot.health_ok = False
|
||||
slot.deploy_time = 0.0
|
||||
slot.deploy_generation = 0
|
||||
|
||||
if previous_state is SlotState.DRAINING:
|
||||
logger.warning(
|
||||
@@ -345,7 +420,7 @@ class BlueGreenDeployer:
|
||||
logger.info(
|
||||
f"Decommissioned {color.value} (was {previous_state.value}, ver '{released_version}')."
|
||||
)
|
||||
self._record("DECOMMISSION", self.active_slot, color, released_version, True, "",
|
||||
self._record("DECOMMISSION", self.active_slot, color, released_version, True, authorised_by,
|
||||
f"released from {previous_state.value}")
|
||||
return dataclasses.replace(slot)
|
||||
|
||||
@@ -436,18 +511,26 @@ class BlueGreenDeployer:
|
||||
authorised_by: Person authorising the rollback.
|
||||
force: Override both guards. The rollback proceeds with an
|
||||
unreconciled book and is logged at CRITICAL. Use only when a
|
||||
human has confirmed the target's state by other means.
|
||||
human has confirmed the target's state by other means. Whichever
|
||||
guards were actually overridden are named in the history record's
|
||||
``detail`` and flagged by its ``forced`` field, so a forced
|
||||
rollback is never indistinguishable from a clean one.
|
||||
"""
|
||||
with self._lock:
|
||||
target_color = self._inactive_color()
|
||||
target = self.slots[target_color]
|
||||
live_color = self.active_slot
|
||||
# Which safety guards this call actually overrode. force=True alone is
|
||||
# not the fact a reviewer needs — an operator may pass it defensively
|
||||
# without any guard having tripped.
|
||||
overridden: List[str] = []
|
||||
|
||||
if target.state not in ROLLBACK_ELIGIBLE_STATES:
|
||||
msg = (f"Rollback refused: {target_color.value} is {target.state.value} and is not a viable "
|
||||
f"rollback target (never deployed, already failed, or decommissioned). "
|
||||
f"Use the kill switch to stop trading, then redeploy.")
|
||||
if force:
|
||||
overridden.append(f"target-eligibility ({target.state.value})")
|
||||
logger.critical(f"FORCED rollback onto non-viable slot. {msg}")
|
||||
else:
|
||||
logger.critical(msg)
|
||||
@@ -470,6 +553,7 @@ class BlueGreenDeployer:
|
||||
logger.critical(msg)
|
||||
self._record("ROLLBACK", live_color, target_color, target.version, False, authorised_by, msg)
|
||||
return CutoverResult(False, live_color, live_color, msg)
|
||||
overridden.append("state-reconciliation")
|
||||
logger.critical(
|
||||
f"FORCED rollback to {target_color.value} with UNRECONCILED state — "
|
||||
f"the restored strategy's book may not match the broker's."
|
||||
@@ -479,8 +563,10 @@ class BlueGreenDeployer:
|
||||
target.state = SlotState.LIVE
|
||||
self.slots[live_color].state = SlotState.FAILED
|
||||
|
||||
suffix = f" [FORCED, overrode: {', '.join(overridden)}]" if overridden else ""
|
||||
msg = (f"EMERGENCY ROLLBACK EXECUTED: Restored to {target_color.value} slot "
|
||||
f"(ver '{target.version}'){' [FORCED]' if force and not sync_ok else ''}.")
|
||||
f"(ver '{target.version}'){suffix}.")
|
||||
logger.warning(msg)
|
||||
self._record("ROLLBACK", live_color, target_color, target.version, True, authorised_by, msg)
|
||||
self._record("ROLLBACK", live_color, target_color, target.version, True, authorised_by, msg,
|
||||
forced=bool(overridden))
|
||||
return CutoverResult(True, target_color, live_color, msg)
|
||||
|
||||
+165
@@ -1,3 +1,4 @@
|
||||
import itertools
|
||||
import threading
|
||||
import unittest
|
||||
|
||||
@@ -74,6 +75,44 @@ class TestDeployment(unittest.TestCase):
|
||||
self.assertEqual(deployer.slots[SlotColor.GREEN].version, "")
|
||||
|
||||
|
||||
class TestRoutingAuthority(unittest.TestCase):
|
||||
"""
|
||||
The execution layer gates every order submission on is_authorised_to_route.
|
||||
If it ever authorises two slots, or none, live flow is either duplicated or
|
||||
dropped.
|
||||
"""
|
||||
|
||||
def test_exactly_one_slot_is_authorised_before_and_after_cutover(self):
|
||||
deployer = BlueGreenDeployer(warmup_seconds=0.0, initial_version="v1.0")
|
||||
self.assertTrue(deployer.is_authorised_to_route(SlotColor.BLUE))
|
||||
self.assertFalse(deployer.is_authorised_to_route(SlotColor.GREEN))
|
||||
|
||||
deployer.deploy_to_inactive("v2.0", authorised_by="ops")
|
||||
# Staging a version must not confer routing authority on its own.
|
||||
self.assertFalse(deployer.is_authorised_to_route(SlotColor.GREEN))
|
||||
|
||||
deployer.cutover(authorised_by="ops")
|
||||
self.assertTrue(deployer.is_authorised_to_route(SlotColor.GREEN))
|
||||
# BLUE is DRAINING: it holds state for rollback but must not send orders.
|
||||
self.assertFalse(deployer.is_authorised_to_route(SlotColor.BLUE))
|
||||
|
||||
def test_refused_cutover_leaves_routing_authority_with_the_live_slot(self):
|
||||
deployer = BlueGreenDeployer(
|
||||
state_sync_fn=lambda a, b: False, warmup_seconds=0.0, initial_version="v1.0"
|
||||
)
|
||||
deployer.deploy_to_inactive("v2.0", authorised_by="ops")
|
||||
self.assertFalse(deployer.cutover(authorised_by="ops").success)
|
||||
|
||||
self.assertTrue(deployer.is_authorised_to_route(SlotColor.BLUE))
|
||||
self.assertFalse(deployer.is_authorised_to_route(SlotColor.GREEN))
|
||||
|
||||
def test_rejects_non_slotcolor_argument(self):
|
||||
"""A truthy string must not be mistaken for an authorised slot."""
|
||||
deployer = BlueGreenDeployer(warmup_seconds=0.0, initial_version="v1.0")
|
||||
with self.assertRaises(TypeError):
|
||||
deployer.is_authorised_to_route("BLUE")
|
||||
|
||||
|
||||
class TestCutover(unittest.TestCase):
|
||||
def test_state_sync_failure_blocks_cutover(self):
|
||||
deployer = BlueGreenDeployer(state_sync_fn=lambda a, b: False, warmup_seconds=0.0)
|
||||
@@ -261,6 +300,76 @@ class TestAuditTrail(unittest.TestCase):
|
||||
for record in deployer.deployment_history:
|
||||
self.assertGreater(record["timestamp"], 0)
|
||||
|
||||
def test_history_opens_with_the_version_that_was_already_live(self):
|
||||
"""
|
||||
A history that begins at the first deployment cannot say what was running
|
||||
before it, so a rollback to the pre-existing version is unidentifiable.
|
||||
"""
|
||||
deployer = BlueGreenDeployer(warmup_seconds=0.0, initial_version="v1.0")
|
||||
|
||||
first = deployer.deployment_history[0]
|
||||
self.assertEqual(first["action"], "INITIALIZE")
|
||||
self.assertEqual(first["version"], "v1.0")
|
||||
self.assertEqual(first["to"], "BLUE")
|
||||
self.assertFalse(first["forced"])
|
||||
|
||||
def test_clean_rollback_is_not_flagged_as_forced(self):
|
||||
deployer = BlueGreenDeployer(warmup_seconds=0.0, initial_version="v1.0")
|
||||
deployer.deploy_to_inactive("v2.0", authorised_by="ops")
|
||||
deployer.cutover(authorised_by="ops")
|
||||
|
||||
self.assertTrue(deployer.rollback(authorised_by="risk-officer").success)
|
||||
|
||||
rollbacks = [r for r in deployer.deployment_history if r["action"] == "ROLLBACK"]
|
||||
self.assertEqual(len(rollbacks), 1)
|
||||
self.assertFalse(rollbacks[0]["forced"])
|
||||
|
||||
def test_forced_rollback_over_ineligible_target_is_marked_in_the_audit_trail(self):
|
||||
"""
|
||||
Regression: a forced rollback whose state sync happened to succeed was
|
||||
recorded as an ordinary successful rollback, so the override of the
|
||||
target-eligibility guard left no trace for a post-incident review.
|
||||
"""
|
||||
deployer = BlueGreenDeployer(warmup_seconds=0.0, initial_version="v1.0")
|
||||
self.assertEqual(deployer.slots[SlotColor.GREEN].state, SlotState.IDLE)
|
||||
|
||||
result = deployer.rollback(authorised_by="head-of-trading", force=True)
|
||||
|
||||
self.assertTrue(result.success)
|
||||
self.assertIn("FORCED", result.message)
|
||||
record = [r for r in deployer.deployment_history if r["action"] == "ROLLBACK"][-1]
|
||||
self.assertTrue(record["forced"])
|
||||
self.assertIn("target-eligibility", str(record["detail"]))
|
||||
self.assertEqual(record["authorised_by"], "head-of-trading")
|
||||
|
||||
def test_forced_rollback_over_failed_sync_names_the_overridden_guard(self):
|
||||
sync_ok = {"value": True}
|
||||
deployer = BlueGreenDeployer(
|
||||
state_sync_fn=lambda a, b: sync_ok["value"],
|
||||
warmup_seconds=0.0,
|
||||
initial_version="v1.0",
|
||||
)
|
||||
deployer.deploy_to_inactive("v2.0", authorised_by="ops")
|
||||
deployer.cutover(authorised_by="ops")
|
||||
|
||||
sync_ok["value"] = False
|
||||
self.assertTrue(deployer.rollback(authorised_by="risk-officer", force=True).success)
|
||||
|
||||
record = [r for r in deployer.deployment_history if r["action"] == "ROLLBACK"][-1]
|
||||
self.assertTrue(record["forced"])
|
||||
self.assertIn("state-reconciliation", str(record["detail"]))
|
||||
|
||||
def test_decommission_records_who_released_rollback_capability(self):
|
||||
deployer = BlueGreenDeployer(warmup_seconds=0.0, initial_version="v1.0")
|
||||
deployer.deploy_to_inactive("v2.0", authorised_by="ops")
|
||||
deployer.cutover(authorised_by="ops")
|
||||
|
||||
deployer.decommission_standby(authorised_by="head-of-trading")
|
||||
|
||||
record = [r for r in deployer.deployment_history if r["action"] == "DECOMMISSION"][-1]
|
||||
self.assertEqual(record["authorised_by"], "head-of-trading")
|
||||
self.assertEqual(record["version"], "v1.0")
|
||||
|
||||
def test_history_records_refused_operations(self):
|
||||
"""A blocked rollback is exactly the event a post-incident review needs."""
|
||||
deployer = BlueGreenDeployer(warmup_seconds=0.0)
|
||||
@@ -333,6 +442,62 @@ class TestConcurrency(unittest.TestCase):
|
||||
self.assertEqual(len(abandoned), 1)
|
||||
self.assertIn("abandoned", str(abandoned[0]["detail"]))
|
||||
|
||||
def test_stale_health_result_cannot_land_on_a_redeployed_slot(self):
|
||||
"""
|
||||
Regression: the post-warmup guard identified the slot by (state, version).
|
||||
Decommissioning the standby and redeploying the *same* version while the
|
||||
first attempt was still warming up produced two indistinguishable
|
||||
deployments, letting the first attempt's stale (unhealthy) verdict be
|
||||
stamped onto the second attempt's instance.
|
||||
"""
|
||||
entered = [threading.Event(), threading.Event()]
|
||||
release = [threading.Event(), threading.Event()]
|
||||
counter = itertools.count()
|
||||
counter_lock = threading.Lock()
|
||||
|
||||
def health(_):
|
||||
with counter_lock:
|
||||
i = next(counter)
|
||||
if i > 1:
|
||||
return True
|
||||
entered[i].set()
|
||||
release[i].wait(timeout=5)
|
||||
return i == 1 # first attempt unhealthy, second healthy
|
||||
|
||||
deployer = BlueGreenDeployer(
|
||||
health_check_fn=health, warmup_seconds=0.0, initial_version="v1.0"
|
||||
)
|
||||
|
||||
first = threading.Thread(target=deployer.deploy_to_inactive, args=("v2.0",))
|
||||
first.start()
|
||||
self.assertTrue(entered[0].wait(timeout=5))
|
||||
|
||||
# Operator releases the standby and redeploys the same version while the
|
||||
# first attempt is still blocked in its health check.
|
||||
deployer.decommission_standby(authorised_by="ops")
|
||||
second_result = {}
|
||||
|
||||
def redeploy():
|
||||
second_result["slot"] = deployer.deploy_to_inactive("v2.0")
|
||||
|
||||
second = threading.Thread(target=redeploy)
|
||||
second.start()
|
||||
self.assertTrue(entered[1].wait(timeout=5))
|
||||
|
||||
release[0].set() # stale attempt finishes first, carrying health=False
|
||||
first.join(timeout=5)
|
||||
release[1].set()
|
||||
second.join(timeout=5)
|
||||
|
||||
self.assertEqual(deployer.slots[SlotColor.GREEN].state, SlotState.READY)
|
||||
self.assertTrue(deployer.slots[SlotColor.GREEN].health_ok)
|
||||
self.assertEqual(second_result["slot"].state, SlotState.READY)
|
||||
|
||||
abandoned = [r for r in deployer.deployment_history
|
||||
if r["action"] == "DEPLOY" and not r["success"]]
|
||||
self.assertEqual(len(abandoned), 1)
|
||||
self.assertIn("abandoned", str(abandoned[0]["detail"]))
|
||||
|
||||
def test_deploy_abandons_slot_decommissioned_during_warmup(self):
|
||||
entered = threading.Event()
|
||||
release = threading.Event()
|
||||
|
||||
Reference in New Issue
Block a user