Beyond the Boolean Switch: Dynamic Configuration for Infrastructure Operations

Every infrastructure engineer has a story like this one.

It’s 2:14 AM. You’ve been up for a good 10 minutes, and the reason why is getting clearer every minute (even without the caffeine): a third-party dependency is timing out. The service happy path is burning through its thread pool waiting on a provider that stopped answering 15 minutes ago. The fix is obvious: stop calling the dependency, serve the fallback path. Thankfully, the playbook says everything: an SSH session, a hand-edited config file, a process restart, and a prayer that you typed the right thing into the right box on the right host while fighting back sleep (still no caffeine).

Reflection kicks in later: the fix carried more risk than the actual outage. Discovery took a dozen minutes and required digging into dashboards and logs. The actual fix took 90 seconds – involving direct access and typing half-asleep.

This is the gap Ops Toggles close. Not feature flags for product teams, but a runtime control surface for infrastructure operators. Pete Hodgson’s toggle taxonomy names the category: Ops Toggles control operational aspects of system behavior, live longer than release toggles, and exist so that operators can change how the system behaves in production without touching the deployment pipeline or using direct access.

Most teams associate Ops Toggle with one pattern: the kill switch. On-off switch. This post builds out the rest of the playbook – and for each pattern, shows what the Layer 1/2 (Code/Deploy) equivalent costs you when minutes matter.

Pattern 1: Kill Switch (the one you know)

Engage the switch if something fails. The one of the canonical Ops Toggles:

def process_order(order, user):
  ctx = {"userId": user.id, "region": user.region}

  # Provider is down — route around it.
  if unleash.is_enabled("main-payment-provider-killswitch", ctx):
    return fallback_payment_flow(order)

  return primary_payment_flow(order)

Alternatives

The Layer 2 (Deploy) equivalent: update service configuration or ConfigMap, trigger a rolling restart, wait for services to cycle. This usually takes minutes, during which every pod that hasn’t restarted yet keeps calling the dead dependency.

The Layer 3 (Runtime) version: flag off, SDK propagation in seconds, every instance stops calling the dependency at nearly the same moment. No restart. No draining. No stragglers.

The kill switch is the foundation – well understood and relatively popular. The interesting patterns start to emerge when you realize the same primitive – externalized, instant, and with targeted activation – generalizes far beyond on-off.

Pattern 2: Graceful Degradation Ladders

A binary switch is a blunt instrument. Under load, what you usually want is less, not nothing – and you want to choose which less carefully.

The way of choosing the less part is called a degradation ladder. Intuitively, you can design it as an ordered set of Ops Toggles in the code, ranked by the cost of disabling each one:

# The ladder, from cheapest to most painful to disable:
# "Pain" is based on how negatively it affects customer experience.
DEGRADATION_LADDER = [
  "enable-recommendations",    # Rung 4: nobody's day is ruined.
  "enable-search-suggestions", # Rung 3: minor friction.
  "enable-avatars-and-media",  # Rung 2: visible (UI only).
  "enable-order-history",      # Rung 1: useful functionality.
  # Floor: Core checkout is never on the ladder.
]

def render_product_page(user, request):
    ctx = {
        "userId": user.id,
        "region": user.region,
        "userPlan": user.plan
    }

    # Always served.
    page = core_product_data(request)

    if unleash.is_enabled("enable-recommendations", ctx):
        page.recommendations = compute_recommendations(user)

    if unleash.is_enabled("enable-search-suggestions", ctx):
        page.suggestions = suggestion_index.query(request)

    if unleash.is_enabled("enable-order-history", ctx):
        page.order_history = fetch_user_order_history(user)

    return page

When saturation climbs, an operator – or an Unleash safeguard watching the saturation metric – descends the ladder rung by rung. Recommendations off. Suggestions off. Order history off – as a last resort. Each rung buys back precious resources (e.g., additional network requests, used memory, CPU cycles) and dependency capacity for the path that matters: checkout.

But there is one more flexibility point that is possible by activation strategies. Each ladder rung presented in the code above can be dialed inside Unleash using targeting rules. Each rule applies on each user’s context and gradual rollout criteria.

The example above reduces pressure only for users in one region, e.g., due to a regional dependency failure.

Alternatively, you might want to gracefully degrade the experience based on the user tiers. Above, we enable the feature for everyone, but not for users on a free plan (with the help of segments).

Alternatives

The Layer 2 (Deploy) equivalent: doesn’t exist. Typically, infrastructure doesn’t know which features are non-critical. Even in a microservice architecture, the smallest element you can disable is a microservice – which may be too coarse-grained. That knowledge lives in the application code, which makes the toggle points Layer 1 by nature.

The Layer 1 (Code) equivalent: the ladder is hardcoded in an if-else cascade with thresholds baked into the code. Changing the order, the thresholds, or which rung you’re on means a deployment – which might be challenging or not even possible during an overload event.

The graceful degradation ladder pattern also answers a question that pure kill switches can’t: what’s our shedding order? Deciding at 2 AM which feature to sacrifice is improvisation. Deciding it in a design review, encoding it as ordered flags, and documenting the cost of each rung – that’s the FeatureOps principle that resilience is designed, not improvised.

Pattern 3: Externally Controlled Circuit Breakers

Circuit breakers are a solved pattern at the library level – Resilience4j, Polly, Hystrix, Hyx, or Opossum’s descendants. What library-level breakers lack is operator control. The breaker trips on its own thresholds, recovers on its own schedule, and if its configuration is wrong for today’s incident, changing it is a deployment.

However, you can wrap that trusted library call in a flag-controlled breaker – to separate the mechanism (in code) from the policy (at runtime):

import httpx
from hyx.retry import retry

retrier = Retry(attempts=4, on=httpx.NetworkError)

async def call_inventory_service(request):
    ctx = {"dependency": "inventory-service"}
    breaker = unleash.get_variant("inventory-circuit-breaker", ctx)

    # Operator says: stop calling it.
    if breaker.name == "forced-open":       
        return cached_inventory_response(request)

    # Operator says: trust it, always call.
    if breaker.name == "forced-closed":     
        return inventory_service.call(request)

    # When "auto", we are dropping to library behavior.
    result = await retrier(inventory_service, request)

    return result

During an incident, the operator can force the breaker open before the automatic threshold trip – or force it closed to verify a dependency has recovered – without waiting for the half-open probe cycle. You can also enable the force-closed behavior for a specific tenant or for the operator’s user via targeting. Most importantly, you do not have to throw away the library and operational knowledge built around it. It keeps handling the common case, but the flag provides a manual override for the uncommon ones.

Pattern 4: Draining and Maintenance Mode

Not all operational responses are about failures. Some are about controlled, targeted unavailability: putting one tenant into read-only mode. At the same time, you repair their data, drain a region ahead of planned maintenance, and freeze writes for accounts under investigation.

The key word is targeted. Layer 2 (Deploy) maintenance mode is per-service or per-cluster – everyone in or everyone out. Works if you have single-tenancy at the infrastructure level. However, runtime-level maintenance mode (Layer 3) is more flexible – e.g., per-tenant:

def handle_write(user, request):
    ctx = {"tenantId": user.tenant, "region": user.region}

    # Constraint on the flag: tenantId IN ["example-corp"]
    # Strategy variant "read-only" with `string` payload:
    #     "writes resume at 04:00 UTC"
    variant = unleash.get_variant("maintenance-read-only", ctx)

    if variant["enabled"]:
        payload = variant.get("payload", {})
        details = payload.get("value", "writes resume shortly")
        response = f"Scheduled maintenance: {details}"
        return read_only_response(response)

    return process_write(request)

One tenant is in maintenance. Other tenants do not know maintenance exists. Now imagine trying to do that with an ingress rule on a heavily utilized multi-tenant cluster, where you want to enable/disable multiple tenants, across multiple endpoints, with overlapping maintenance windows.

Pattern 5: Capacity Management Flags

The next pattern shifts behavior based on what the fleet can afford right now: batch job concurrency, cache TTLs, sampling rates, queue consumer counts. These are classic tuning parameters that teams bake into environment variables – and then can’t touch without a rolling restart, which is the wrong move when the reason you’re tuning is that the system is already under pressure.

As flag variants delivering specific payloads, they become live dials – a truly dynamic configuration:

concurrency = unleash.get_variant("batch-worker-concurrency", ctx)
pool.resize(int(concurrency.payload.value))
# ... and it's applied at the next scheduling cycle.

Note the applied at the next scheduling cycle – this is the intent register pattern from the previous post. The flag changes instantly, and the pool resizes at its safe lifecycle boundary. Runtime doesn’t mean reckless.

Pattern 6: Runtime Verbosity Control for Operational Tools

A support ticket comes in: one tenant sees intermittent failures during order synchronization. Nobody else does. You need DEBUG logs for that tenant’s requests, and you need them while the problem is still happening.

The usual path is to raise LOG_LEVEL for the whole service. Now every tenant writes DEBUG lines into your log pipeline. You pay to ship, index, and store them. Then you spend the next hour writing queries to filter out everything you didn’t ask for. You turned up the volume for everyone to hear one conversation.

The same applies to profilers and tracing. Continuous profiling on every request costs CPU on the hot path. Sampling 100% of traces costs storage and makes the trace backend slower for everyone using it. So teams keep observability low by default and pay for it during incidents, when they need detail and don’t have it.

Let’s apply the learning from previous patterns and create a flag with targeting that lets you raise the detail for the requests you care about and leave everything else alone.

Log level per tenant

The flag returns a variant whose payload is the log level. Targeting decides who gets it:

import logging

LEVELS = {
    "debug": logging.DEBUG,
    "info": logging.INFO,
    "warning": logging.WARNING
}

def request_log_level(user):
    ctx = {
        "userId": user.id, 
        "tenantId": user.tenant, 
        "region": user.region
    }
    variant = unleash.get_variant("log-verbosity", ctx)

    # Default for everyone outside the targeting.
    if not variant.enabled:
        return logging.INFO

    return LEVELS.get(variant.payload.value, logging.INFO)

def handle_request(user, request):
    with request_logger(level=request_log_level(user)) as log:
        # Emitted only for targeted tenants.
        log.debug("order sync payload: %s", request.body)

        return sync_orders(request)

The service keeps logging at INFO. Requests from the targeted tenant log at DEBUG. When you have what you need, you turn the flag off, and the next request returns to INFO. No restart, so the failing process keeps running in the same state you were trying to observe.

Profiling only the requests you suspect

Profilers are expensive enough that most teams run them in short bursts or not at all in production. Why not gate capture with a flag?

def handle_checkout(user, request):
    ctx = {
        "userId": user.id, 
        "tenantId": user.tenant, 
        "region": user.region
    }
    if unleash.is_enabled("profile-checkout", ctx):
        profiling_ctx = {"tenant": user.tenant, "user": user.id}
        with profiler.capture(tags=profiling_ctx):
            return process_checkout(request)

    return process_checkout(request)

Target a single user who reported slowness, or 5% of requests in one region where p99 latency went up. You get profiles for the traffic that shows the problem and with no cost for the rest.

Trace sampling narrowed by context

The same idea works one level lower, in the tracing SDK. A sampler can ask the flag whether this trace should be kept at full rate:

# Conceptual sketch of an OpenTelemetry sampler 
# that consults a flag.
class FlagAwareSampler(Sampler):
    def __init__(self, unleash, default_ratio=0.01):
        self.unleash = unleash
        self.default = TraceIdRatioBased(default_ratio)

    def get_description(self):
        return "FlagAwareSampler"

    def should_sample(self, parent_context, trace_id, name,
                      kind=None, attributes=None, links=None):
        ctx = {"tenantId": (attributes or {}).get("tenant.id", "")}

        if self.unleash.is_enabled("full-trace-sampling", ctx):
            return SamplingResult(
                Decision.RECORD_AND_SAMPLE, 
                attributes
            )

        return self.default.should_sample(
            parent_context, trace_id, name, 
            kind, attributes, links
        )

Everyone stays at 1% sampling. The tenant under investigation gets every trace. Flag evaluation happens in the SDK from a local copy of the configuration, so the sampler doesn’t add a network call per span.

How does it look in Unleash?

Every tenant outside the constraint gets the default INFO level. To widen the scope, add another tenant to the constraint. To narrow it, remove one.

Also, when chasing those bugs, stickiness matters. With userId stickiness, the same 5% of users are profiled on every request, so you see full sessions instead of scattered single requests.

Last but not least: verbose observability is easy to switch on and forget. A safeguard on log or span volume (1) turns it off if someone widens the targeting by mistake or leaves it on over the weekend. You can also narrow capture to specific dates and hours via targeting (2).

Alternatives

The Layer 2 (Deploy) equivalent: change LOG_LEVEL or the sampling ratio via static artifact configuration or in a ConfigMap, and bounce the service. Minutes of wait time and it applies to every tenant.

The Layer 1 (Code) equivalent: add log statements or a profiler hook, build, and deploy. That’s the slowest path, and you’ll need another deployment to remove them afterwards.

In both layers there is one more subtlety. The restart also has one drawback: it may clear the in-memory state you were trying to see, so the bug may stop showing up the moment you start looking for it.

The cost argument is the one infrastructure teams feel most. With Layer 2 you pay for detail across the whole fleet and filter at query time. With a flag you capture only the detail you asked for, so there’s less to ship, store, and search.

Pattern 7: Targeted Load Shedding

When demand exceeds what the fleet can serve, something has to be dropped to avoid complete downtime. Layer 2 (Deploy) tools can drop traffic. Ingress rate limits, load balancer connection caps, and overload filters all reduce how much traffic reaches you. Not all of them can tell whose request they’re dropping. The enterprise customer and the free-tier scraper look the same to a connection counter, unless you deliberately blur the lines between infrastructure and runtime.

At runtime (Layer 3), the request carries its context, so you can choose who gets dropped. And by layering toggles on the same endpoint, you get multiple shedding modes:

def write_transaction(request):
    ctx = {
        "userId": request.user.id,
        "properties": {
            "userTier": request.user.tier,
            "region": request.region
        }
    }

    # Graceful shedding: a cheap, well-formed answer 
    # instead of the real work.
    if unleash.is_enabled("serve-static-response", ctx):
        return {"now": "STATIC", "static": True}
    
    # Hard shedding: hold the connection, then reset it.
    if unleash.is_enabled("drop-write-transaction-traffic", ctx):
        # Tarpit: slows the client's retries.
        time.sleep(1.0)
        return abort_connection(request)

    return process_transaction(request)

Graceful shedding returns a static response. The client gets an answer, and its retry logic stays calm. For POST traffic, this has to live in the application, because a write can’t be served from a cache.

Hard shedding drops the connection. The delay before the drop matters. If you reset the connection immediately, the client retries immediately, and your shedding creates more inbound traffic than it removes. The Google SRE book calls this tarpitting: hold the request for a while before you fail it. One second of delay caps each client’s retry rate. The companion demo also resets the connection with RST rather than closing it with FIN, so load generators and HTTP client pools don’t keep half-dead connections.

How does it look in Unleash?

The targeting is set in Unleash, not in code:

Free-tier users get the static response. Half of EU and EEA users get it too, and with stickiness on userId, the same half stays shed, so no client sees a random 50% failure rate. Premium traffic in every region stays live. In the demo’s Grafana dashboard, the FREE series switches from live to static while REGULAR and PREMIUM don’t move.

Alternatives

The Layer 2 (Deploy) equivalent: rate limits and connection caps at the ingress or load balancer. Fast and useful against a flood, but blind to tier, tenant, and region.

The Layer 1 (Code) equivalent: shedding rules and thresholds hardcoded in the service. Changing who gets shed means a deployment during an overload.

The two layers work together. Layer 2 caps the flood at the front door, before your process spends any CPU. Layer 3 decides who gets served among the traffic you can handle.

Run it yourself: the companion demo ships this scenario. Run make setup, then make load-test ENDPOINT=/write-transaction RPS=60, and flip the flags while you watch the per-tier dashboard.

What Those Patterns Require: Operational Maturity

Here’s the part that separates a toggle from an operational control plane.

Ops Toggle is a production change executed under pressure. That means it needs everything any production change needs: access control over who can flip it, an audit trail of who flipped what and when, an approval workflow for the changes that shouldn’t be one person’s 2:14 AM judgment call, and integration with the incident timeline so the postmortem can reconstruct what happened.

This is where homegrown systems often fail. A kill switch in a homegrown flag system usually means someone updates a database row and hopes the cache refreshes before the incident escalates. What does it miss?

• No RBAC: anyone with database access can change production behavior.
• No audit trail:  the postmortem says the flag was changed at some point by someone.
• No approval gates: the intern can flip the same switches as the incident commander.

Mercadona Tech’s engineering team described exactly this failure mode with their homegrown system: direct database access for toggling, vulnerabilities from unrestricted writes, and a total lack of traceability.

The governance framing we already discussed applies with extra force here: during an incident, what was activated, for whom, by whom, and why is not compliance paperwork – it’s the incident timeline. An OpsToggle without an audit trail is an untracked production change executed during the worst possible window.

Reversibility as a service, for operators

Step back and look at what all the above patterns have in common. Each one replaces a slow, risky, manual intervention – SSH sessions, hand-edited configs, emergency deploys, kubectl surgery – with a governed, instant, targeted, reversible control. And even if the screenshots show the Unleash Admin UI, you can manage all of the above via IaC and automate via API.

That’s reversibility as a service compatible with an infrastructure engineer’s mindset. It is delivered to the people who need it most acutely: operators mid-incident. The platform provides the evaluation infrastructure, the propagation speed, the targeting engine, and the governance. The operations team gets a control surface where every action is fast, scoped, audited, and – critically – undoable in seconds.

This is Post 6 of “The Runtime Control Layer” – a series on FeatureOps for infrastructure engineers.

Previously: Runtime control doesn’t have to be instant: you control “when”
Next: Blurred Lines: When Infrastructure Decisions Leak Into Runtime (and vice versa)