Build, Deploy, or Request: Where your configuration decisions actually belong

You’ve probably had this argument. Maybe not out loud, but in a PR review, a Slack thread, or a 2 AM incident where someone asks: why is this hardcoded?

The answer is usually some combination of it was easier, it never needed to change, or – the honest version – nobody thought about where this decision should live.

That’s the problem. Not the specific choice, but the absence of a framework for making it. We have mature, well-understood practices for getting code to production: CI/CD, GitOps, IaC, container orchestration. We have remarkably little shared vocabulary for deciding where a decision belongs in the first place.

This post introduces a battle-tested mental model that gives you a diagnostic for that question, structured in three layers. It doesn’t argue that one layer is better than another. It argues that misplacing a decision creates friction, and that most teams have a blind spot in exactly the same place.

The three layers

Every configuration decision in your system gets bound – committed, resolved, locked in – at a specific point in time. That binding time determines everything about how flexible, how fast, and how costly that decision is to change.

Layer 1: Code (Build Time)

Decisions baked into the artifact. If-else branches, compiled defaults, static config files bundled in Docker images, Helm values frozen at chart build. The decision is made when the code compiles or the image builds. Changing it means a new artifact, a new pipeline run, a new deployment.

# Layer 1: This value is baked into the image
ENV WORKER_THREADS=4
COPY config/production.json /app/config.json

// Layer 1: Compiled into the artifact
private static final int MAX_CONNECTIONS = 100;
private static final boolean FEATURE_NEW_CHECKOUT = false;

 

Properties: Deterministic. Reproducible. No runtime dependencies. If available, full compiler or linter verification. Referentially transparent, which is a fancy name for saying that you know exactly what you shipped because the decision is literally inside the artifact.

Trade-offs: Zero runtime flexibility. Every change – even toggling a boolean – requires going through the full CI/CD process. No ability to customize those values for specific contextual runtime attributes – like user or tenant IDs. Reaction time is measured in minutes to hours.

Reversal cost: hours, because – rebuild, re-test, redeploy.

Layer 2: Infrastructure (Deploy Time)

Decisions set when the artifact reaches the environment. Environment variables, Kubernetes ConfigMaps, mounted secrets, Terraform outputs, service mesh routing rules. The artifact is the same, the environment configures it differently.

For example, a mentioned ConfigMap:

# Layer 2: Set at deploy time, same artifact across environments
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  DATABASE_URL: "postgres://prod-primary:5432/app"
  LOG_LEVEL: "warn"
  CACHE_TTL: "300"

Or a Kubernetes Deployment fragment:

# Layer 2: GitOps reconciliation ensures the cluster matches this
apiVersion: apps/v1
kind: Deployment
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: api
          env:
            # "Feature flag", but really: a deploy-time constant
            - name: FEATURE_DARK_MODE
              value: "true" 

 

Properties: Declarative desired state. Drift detection (if you’re using something like GitOps). Audit trail via source control history. Support for environment-specific configuration.

Trade-offs: Contextual decisions are at most infrastructure related. The ConfigMap doesn’t know who’s making the request. If you want more, abstractions start to leak – and you drag down user specific context to the infrastructural level. Last, but not least: changing a value requires a restart, a rolling update, or at minimum: reconciliation cycle. You’re coordinating across topology, not across users.

Reversal cost: minutes, because – git revert + reconciliation loop, or kubectl rollout undo.

Layer 3: Runtime (Request Time)

Decisions evaluated per-request, with the full context of what users or tenants are interacting with the system, what they’re asking for, and what the system knows right now. Feature flags, contextual branching, dynamic targeting. The decision is externalized – the code defines the capability, but the external system provides value responsible for activation.

# Layer 3: Evaluated per-request with user context
from unleash_client import UnleashClient

client = UnleashClient(
  url="https://unleash.internal/api", 
  app_name="checkout-service"
)

def process_payment(user, order):
    context = {
      "userId": user.id, 
      "tenantId": user.tenant, 
      "region": user.region
    }
    
    if client.is_enabled("new-payment-flow", context):
        return new_payment_processor(order)
    else:
        return legacy_payment_processor(order)

 

Properties: The cost to add or remove a decision point is not higher than in other layers, but full application and request context is available. Per-user, per-tenant, per-region targeting – full flexibility. And yet it adds instant reaction – on or off in seconds, as a decision switch is independent from your software delivery pipeline.

Trade-offs: Additional dependency in the chain. With remote evaluation there is an overhead (though with local/edge evaluation, we’re talking sub-millisecond or milliseconds). Requires explicit instrumentation – you have to write the decision point (as in layer 1). And flags have lifecycles that need management (like infrastructure in layer 2 – pet vs. cattle analogy)

Reversal cost: seconds, because architects, engineers and product managers can toggle off the flag using the external system’s dashboard. No code change. No restart. No deployment.

The temporal axis: late binding as a design property

If the binding framing sounds unfamiliar, don’t worry – it is an analogy. The three layers map directly onto binding time – a concept you know from programming languages design:

• Layer 1 = early binding. Mental model analogy is compile time or static typing: you get guarantees, but you give up flexibility.
• Layer 2 = link binding. It’s deploy time binding or like dynamic linking: the same binary loads different libraries depending on the environment. But flexibility has its price, as DLL hell is real.
• Layer 3 = late binding. Runtime level, virtual dispatch or service discovery: the decision is deferred to the last responsible moment, when maximum context is available.

The common denominator is visible immediately: later binding layers are progressively more flexible. That’s not a value judgment – it’s a structural property. The more you can defer a decision, the more information you have, the more options you retain, and the faster you can change course.

But – and this matters – late binding also introduces variability. A statically compiled binary does the same thing every time (it’s referentially transparent). A runtime-evaluated flag might do different things for different users on different requests. That variability is the point, but it has its price – and that can be paid with an infrastructure that allows you to manage it safely.

The context axis: what you know at each layer

However, those layers don’t just differ in when decisions bind. They differ in what information is available to make the actual decision.

Layer 1: Code, Build time

What you know: Code structure, dependencies, compiler output.

What you don’t know: Environment, user, request, current system state.

Layer 2: Infrastructure, Deploy

What you know: Environment (prod / staging), topology (region, AZ, rack awareness), resource allocation.

What you don’t know: Which user, which request, current error rate, current load, business-specific metrics.

Layer 3: Runtime, Request Time

What you know: Runtime context flexibility: user identity, tenant, request attributes, session state, system metrics.

What you don’t know: Has the richest context by default, though not infinite – you are bound by application context.

Quick analysis

That axis makes clear why dragging Layer 3 decisions down to Layer 2 always creates friction. To make a per-user decision at the infrastructure layer, you’d need to propagate user context into your service mesh, your load balancer, or your Kubernetes routing rules. Teams do this (with yours truly guilty as charged) – header-based routing in Istio, weighted traffic splits with session affinity – but it’s architecturally awkward. It’s an abstraction leak.

At best, you’re working around a layer boundary, not working within it. At worst, you are breaking layers, turning them inside-out to achieve something that wasn’t possible by design. It’s a round peg in a square hole moment, and that friction is a signal: you’re solving a problem that requires information from Layer 3 at Layer 2, and the workaround is more complex than the solution it’s trying to avoid.

To make the point above complete and honest: you should be careful with the reverse as well, when you are moving Layer 2 concepts up to Layer 3. It may end up in the same class of abstraction leak as explained above. Imagine artificially adding rack awareness or some network topology details to the runtime. It is not just a bad taste, but it may cause unintended consequences constraining the application design (e.g., hindering horizontal scalability).

However, there are certain situations where such an upward move may be helpful – especially when the decision is runtime-specific and would benefit from having enhanced context. Are you tracking down a mysterious issue that happens only for a specific subset of tenants? Then you would benefit enormously with enabled detailed logging/tracing/profiling for that group only, to narrow down the performance overhead and lower down a potential blast radius when troubleshooting goes wrong. Such a tactic is a powerful tool in your operational toolbox – and that is called an OpsToggle (we will return to this in more detail later in the series).

The Diagnostic

Here’s the practical takeaway. Next time you’re deciding where a configuration value belongs, ask these questions:

Is this value the same for every request in every environment forever?

• If yes → Layer 1.  Bake it in, enjoy the determinism.

Does this value need to differ per-request, per-user/tenant, or is it contextual?

• If yes → Layer 3. Layer 2 can’t see request context without workarounds.

Does this value need to change without a deployment?

• If yes → Layer 3.
• If it’s truly process-lifetime and infrastructure environment-specific → Layer 2.

When this decision is wrong, how fast do you need to undo it?

• It doesn’t matter or hours are acceptable → Layer 1.
• Minutes are acceptable → Layer 2.
• Seconds → Layer 3.

We call the latest question the reversal test. Practically, in the organizational environment where reversibility should be a first-class property, that is the question that matters the most. The layer you choose determines the cost of the reversal operation.

Where most teams are – and the gap

If you’re reading this as an infrastructure or platform engineer, you probably have sophisticated practices for Layers 1 and 2. Your CI/CD pipelines are optimized. Your GitOps workflow is mature. Your Helm charts are templated, your Terraform modules are reusable, your ConfigMaps are version-controlled.

But, Layer 3? That’s usually the ad-hoc layer. Maybe there’s a homegrown configuration service someone built years ago. Maybe there are some if-statements checking environment variables with FEATURE_FLAG_ prefix but aren’t really flexible. Maybe there’s a database table that gets manually updated during incidents.

Most teams have a well-paved Layer 1, a well-paved Layer 2, and a dirt road for Layer 3.

This series is about paving that road. Not by replacing what you’ve built for Layers 1 and 2 as those are your foundations, and they are essential. But by developing the same maturity for runtime decisions that you already have for build-time and deploy-time decisions.

That paved path is a discipline that is called FeatureOps. A mature platform provides reversibility as a service – standardized runtime control that every team can rely on without building their own escape routes. And it starts with knowing where your decisions belong.

This is the 1st post of “The Runtime Control Layer” – a series on FeatureOps for infrastructure engineers.

Next: Everything you can’t do with environment variables (and what you can).