Some changes have to touch every customer exactly once, and the touch itself is the risky part. Our client runs an email-security platform that protects Outlook mailboxes, and we needed to migrate every mailbox's Microsoft Graph change-subscription to a new shape. Graph offers no in-place edit for that: each subscription had to be deleted and recreated, one round trip per mailbox, across a few thousand mailboxes spread over about 70 customer tenants. A boolean feature flag would have started all of it the moment the deploy landed. That is the shape of change that turns a migration into an incident, either by tripping a third-party rate limit or by breaking the same thing for everyone simultaneously. What we wanted was a dial: enable a slice of the fleet, watch it, widen, repeat.
The obvious first answer is an allowlist of tenant IDs in the flag's value, and it works for about two steps. By the time you are halfway through the fleet you are hand-editing an environment variable containing thirty-odd UUIDs, and every edit is a chance to accidentally drop a tenant that has already migrated. For a one-directional change, that means migrating it back and then forward again. Sampling randomly per call is worse, because the decision is re-rolled on every invocation and tenants flicker in and out of the cohort continuously. Building a real rate limiter is genuine infrastructure, with its own state and its own failure modes, commissioned for a job that runs once and is then deleted.
What we used instead was a hash. Every tenant already has a stable UUID, so we derived a bucket from it with crc32(id) % 100, and made the flag's value a threshold against that bucket. Our feature flags resolve from environment variables, and the resolver already understood two value forms: a boolean, and a comma-separated allowlist. We added a third. (Our codebase calls a customer a client; one client maps to one Microsoft tenant.)
// A flag's value comes from an env var. The resolver accepts:
// "true" / "false" -> on / off for everyone
// "<uuid>,<uuid>" -> an explicit allowlist
// "25%" / "25% -<uuid>" -> a hash-bucket cohort, optionally minus named clients
private function clientIsInPercentageRollout(string $clientId, string $value): bool
{
if (!preg_match('/^\s*(\d+)\s*%\s*(?:-\s*(.*))?$/', $value, $matches)) {
return false; // an unparseable value enables nobody: a typo must fail closed
}
// Split on commas, not on the hyphen: UUIDs are full of hyphens.
$excluded = array_filter(array_map('trim', explode(',', $matches[2] ?? '')));
if (in_array($clientId, $excluded, true)) {
return false;
}
return crc32($clientId) % 100 < (int) $matches[1];
}
Three properties make this better than it first looks. It is deterministic: there is no enrolment table, no assignment step, and no state that can drift between two workers or be lost in a rollback. The cohort is recomputed from the tenant's own identifier every time anyone asks. It is monotonic: raising the threshold only ever adds tenants, so a tenant enabled at 10% is still enabled at 25%. That mattered more than it sounds, because the desired state was derived from the flag. Had a tenant fallen out of the cohort when the dial moved, it would have migrated backwards and then forwards again, paying the risky delete-and-recreate three times instead of once. And it is previewable: MySQL's CRC32() returns byte-identical values to PHP's crc32(), so the exact membership of any future step can be queried in SQL before it is ever deployed. Bucketing on the tenant rather than on the individual mailbox was deliberate too, since each customer is a separate Microsoft tenant with its own API throttle budget and is therefore the real failure domain. Hashing per mailbox would have spread every step thinly across all seventy tenants: the maximum blast radius per step rather than the minimum.
The ramp we ran was 10% → 25% → 35% → 65% → 100%, holding each step for about twenty-four hours. That was long enough for the renewal window to convert that cohort and for the error checkpoints to stay clean before widening. Those odd-looking thresholds are the most transferable thing we learned. A percentage rollout sounds like it gives you proportional control over blast radius, and it does not. Tenant sizes span two orders of magnitude: the step that opened ten buckets brought in a quarter of the fleet, while the step that opened thirty brought in a fifth. Percent-of-tenants is simply not percent-of-work. So we stopped picking round numbers and started picking thresholds from the data, grouping buckets into 5% bands and running a cumulative total over measured mailbox counts. The first step was deliberately tiny: some 3% of the fleet across four tenants, enough to prove the mechanism end to end and nothing more. The four remaining steps then took roughly 28%, 26%, 20% and 23% of the fleet, comparable workloads chosen from a query rather than from the shape of the number line.
Every step was judged against a stop signal fixed before the first one. It was a SQL count of mailboxes left without a working subscription, scoped to the live cohort and compared against a baseline measured before any of it started. That way widening was a decision with a number behind it, rather than an impression that the error tracker looked quiet. It worked. The whole fleet migrated, and no ramp step stranded a mailbox. The one that did get stuck during the rollout was stranded by an unrelated transient Graph fault on the renewal path, and we cleared it by hand. The exclude list earned its keep as well. One tenant turned out to have a broken OAuth consent entirely unrelated to the migration, and rather than dial back and stall everyone else, we parked that single UUID after the threshold and kept ramping.
Two caveats before you copy this. It is a poor fit for A/B experiments needing balanced arms, because a CRC32 over UUIDs balances tenant count and nothing else; as our own numbers show, it does not balance weight. And if your change is genuinely free to reverse, the monotonicity that makes this valuable is machinery you do not need. This technique is for the changes that are expensive to do twice.
