Deployment
Release trains, rollback, and the fleet-wide freeze.
Release train
Fleet software ships on a weekly train, cut Tuesday 14:00 UTC from main. The train rolls to the staging fleet immediately, to 5% of production robots after 24 hours of clean staging, and to 100% after a further 48 hours.
Rollback
- Announce in
#fleet-opswith the version you are rolling back from and to. - Run
fleetctl rollout undo --stage=all. This is safe to run twice. - Watch the mission-success dashboard for fifteen minutes. If it does not recover, page Platform.
Fleet-wide freeze
A freeze halts all rollouts, including hotfixes. Only the on-call lead may declare one. To declare:
fleetctl freeze set --reason="<incident id>" --ttl=24h
Freezes expire automatically. Extending one requires a second lead's approval.
On-call expectations
- Acknowledge pages within ten minutes.
- If you are going to be unreachable for more than an hour, hand off explicitly.
- Every page gets a post-incident note, even if the resolution was "flapped and recovered".
Comments
Comments are reviewed by the Platform team before they appear on the page.