Green Across the Board

Three pod IPs sat in Endpoints/checkout-api when payments escalated. Every targetRef name ended in v417. Git on main already carried revision 418. Argo CD had been Synced for eleven minutes. The Rollout controller reported Healthy with canary weight at twenty percent. Those facts did not contradict each other until you asked which revision a paying checkout request would hit. The answer was still v417, because the Service selector had not moved.

I had reacted to the green release thread like everyone else. The fee-table fix existed in the image digest deploy-bot posted. It was not on the path Ingress used.

Same afternoon: green checks vs fee decline rate

The cluster names here are a composite. The failure mode is not. We had fourteen automated gates confirming that release machinery worked. None of them compared the Service front door to the labels on the pods the Rollout had created. Sync status is honest about Git. It is silent about traffic membership.

The checklist everyone trusted

Our release thread was not a celebration. It was an audit trail we had learned to treat as sufficient. Each line mapped to an automated gate that had been green for months. Argo CD Synced meant the cluster matched Git commit a9c4e11. The Rollout controller reported Healthy with canary weight at twenty percent. Synthetic smoke against /healthz and a handful of read-only API calls returned forty-seven greens. The global error budget for the namespace showed negligible burn. Helm revision 418 matched the manifest in the repo. The container image digest on new pods matched the digest in the build pipeline.

None of those statements was false. Together they implied a conclusion that was false: checkout traffic was exercising revision 418.

That implication failed because every gate inspected a different object than the one Ingress used to route paying requests. GitOps answered whether desired state matched applied manifests. Rollouts answered whether canary pods existed and whether a PromQL template over their metrics cleared a threshold. kubelet probes answered whether processes responded on port 8080. Smoke tests hit URLs that bypassed the same Service selector production used. The error budget aggregated HTTP 5xx across the whole namespace, not checkout-specific business failures on the route we had changed.

We had fourteen independent confirmations that the release machinery worked. We had zero confirmations that the Service's Endpoints object included an IP address from the canary ReplicaSet.

What the release checklist saw vs where requests went

The diagram is the whole incident in one frame. Green on the left is control-plane truth. Orange on the right is data-plane truth. They diverged at a single field we stopped reading years ago when checkout was a Deployment and the Service selector was app: checkout-api only.

Synced measures Git, not customers

Argo CD's Synced condition is doing honest work. It compares rendered manifests in Git to resources in the cluster and reports drift. When it says Synced, you can trust that the Service object in etcd carries whatever selector your Helm chart last wrote. It does not tell you that those selectors match the labels on pods the Rollout creates this week.

Our chart had evolved in two directions at once. The Rollout template gained a version label tied to the chart's appVersion. The Service manifest did not. It still selected app: checkout-api and version: v417 because that second line had been copied from an environment-specific values file in February and never removed when v418 promotions became routine. Argo synced that stale selector faithfully. Git was the source of truth. Git was wrong for traffic, and sync had no opinion about traffic.

I want to separate three verbs we had conflated:

We had automated the first two and called the release done. Adopted is the only verb customers experience.

Published, Applied, Adopted: only the last one reaches customers

The canary analysis made the gap easier to miss, not harder. Our AnalysisTemplate scraped request success rate from the canary pods' own /metrics endpoint. Canary pods handled synthetic traffic the Rollout injected internally and traffic from a cluster-local test runner. Both paths labeled requests correctly at the pod. Neither path went through Service/checkout-api with the stale selector. PromQL showed 99.4% success on canary. Production checkout, still pinned to v417, continued to apply the old fee table. Declines were a business metric. Our analysis metric was HTTP 200.

That is not a subtle bug in PromQL. It is a category error: we measured canary health on a graph that did not include the Service front door.

The AnalysisTemplate itself was not misconfigured for its own assumptions. It was misaligned with our routing architecture:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: checkout-success-rate
spec:
  metrics:
    - name: success-rate
      interval: 30s
      count: 10
      successCondition: result[0] >= 0.99
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{pod=~"checkout-api-.*-v418.*",status!~"5.."}[1m]))
            /
            sum(rate(http_requests_total{pod=~"checkout-api-.*-v418.*"}[1m]))

The query only sees pods whose names match the v418 ReplicaSet pattern. That is correct for "are v418 pods healthy among the requests they receive?" It is useless for "are customers hitting v418?" Customers never hit those pods on the Ingress path. A template that clears this query while Endpoints omit v418 is doing exactly what we asked. We asked the wrong question.

We considered switching to Ingress access-log metrics immediately. The join is slower to build but matches reality:

query: |
  sum(rate(checkout_ingress_requests_total{service_version="v418",status!~"5.."}[2m]))
  /
  sum(rate(checkout_ingress_requests_total{service_version="v418"}[2m]))

That series did not exist before the incident. We added service_version at the Ingress log exporter by reading the upstream Endpoints target metadata. Building the metric took a week. Running the wrong metric for six months took one afternoon of fee declines to notice.

Where checkout traffic actually went

Ingress for checkout terminates TLS and forwards to Service checkout-api in namespace payments. In Kubernetes, the Service does not load-balance to Deployments or Rollouts. It load-balances to Endpoints (or EndpointSlices): the set of pod IPs whose labels satisfy spec.selector at this moment.

When I pulled the objects after the incident, the Rollout showed v418 canary pods running with labels app=checkout-api, version=v418. The Service selector required version=v417. The Endpoints controller, correctly, listed only v417 IPs. Ingress sent 100% of external checkout traffic to those addresses. The Rollout's twenty percent canary weight applied to traffic the Rollout owned internally, not to the Service-backed path our customers used.

# Service (synced faithfully from Git)
apiVersion: v1
kind: Service
metadata:
  name: checkout-api
  namespace: payments
spec:
  selector:
    app: checkout-api
    version: v417          # pinned in env values; not updated by Rollout
  ports:
    - port: 80
      targetPort: http

# Rollout pod template (revision 418)
template:
  metadata:
    labels:
      app: checkout-api
      version: v418

No amount of sync status bridges that mismatch. The Endpoints controller is level-triggered and literal. It is not going to infer that canary promotion intent should override a selector field.

We confirmed with a command that should be boring and therefore was not in any gate:

kubectl get endpoints checkout-api -n payments -o jsonpath='{range .subsets[*].addresses[*]}{.ip}{"\t"}{.targetRef.name}{"\n"}{end}'

Output listed three pod names, all from ReplicaSet checkout-api-7d4f9c-v417. Zero rows from v418. While #platform-releases celebrated, this command would have taken four seconds.

Two controllers, one Service, zero overlap

Two controllers and one Service

Progressive delivery added a second controller with opinions about labels and weights. We never updated our definition of done to require those opinions to agree with the Service.

The Rollout spec used a stable Service and a canary Service in the conventional dual-Service pattern in documentation, but our chart still pointed Ingress at the stable Service name while the stable Service selector had frozen on v417. The canary Service existed for analysis traffic and was never wired to Ingress. Documentation assumed operators would advance weights on the stable Service's Endpoints membership. We advanced Rollout weights on a parallel track.

This is the kind of detail that reads as misconfiguration in hindsight and as "the platform team's problem" in the moment. The platform team had shipped Rollouts, analysis templates, and GitOps. Product teams shipped chart bumps. The gap lived in the contract between them: who owns the selector when version becomes a moving label?

We answered that question only after the incident, in a Conftest policy that runs after Helm render and before Argo sync:

package release.checkout
 
deny[msg] {
  input.kind == "Service"
  input.metadata.name == "checkout-api"
  stable := input.spec.selector.version
  rollout := input.rollout.canary.labels.version
  stable != rollout
  msg := sprintf(
    "Service selector version %v must match Rollout canary label %v before sync",
    [stable, rollout],
  )
}

The policy needs the rendered Rollout injected into the Conftest input. That is extra plumbing. It is also the first check we wrote that compares two resources against each other instead of checking each resource in isolation.

Argo Rollouts' own docs describe analysis metrics and Service routing as separate concerns. We treated them as one green checkbox because both appeared in the same Slack message.

Probes that approve a broken graph

Pod probes were the third layer that reassured us without touching Endpoints. Every v418 pod passed liveness and readiness on /healthz. Readiness only proved the process started and the handler returned 200. It did not prove the pod was a member of the Service customers used. A pod can be Ready and unroutable. That state even has a name in incident vocabulary: "ready but not receiving traffic."

Our readiness handler checked an in-memory flag set after database ping succeeded. Database ping succeeded on v418. The fee fix was loaded. No customer request hit the handler on the production path because Endpoints never listed the pod.

Smoke tests compounded the blind spot. They ran from a cluster Job that called http://checkout-api.payments.svc.cluster.local/healthz and three GET endpoints. Cluster DNS resolved the Service name to ClusterIP, kube-proxy (or dataplane equivalent) forwarded to Endpoints, and Endpoints still listed v417. Smoke tested v417 and reported green. The test name said checkout-api. The revision under test was not the revision in Git.

Three green probe paths that never touched the Ingress route

We changed two probe-related rules after the incident. Readiness for externally routed apps must fail if the pod's IP is not present on the Service Endpoints the Ingress references. That requires a post-start hook or a small sidecar check against the Endpoints API, which adds latency and RBAC. We accepted the cost for tier-one routes only. Second, smoke tests must assert response headers or body fields that include the running revision, and they must fail if the header does not match the Git tag being promoted.

readinessProbe:
  httpGet:
    path: /ready
    port: http
  periodSeconds: 5

The /ready handler now includes a cheap Endpoints membership check for tier-one services: compare pod IP against the cached Endpoints slice for checkout-api. If missing, return 503 and keep the pod out of any selector that might accidentally broaden. This is not elegant. It is explicit.

Global error budget burn made the declines harder to see early. Namespace-level aggregation swallowed a rise in checkout-specific business errors because unrelated services had quiet hours. We added a burn-rate alert on a single SLI: successful fee application on checkout POST /v1/intent, sliced by service.version label on the handler that actually executed. That label existed on v417 responses. The alert fired twelve minutes before payments escalated. It would have fired at minute one if anyone had been watching it. Nobody was. It was a dashboard, not a gate.

What we added to release done

We stopped asking "is it synced?" and started asking "is it adopted?" Release done is now a matrix, not a boolean. Rows are cross-resource checks. Columns are who owns them. A deploy cannot close until required cells are green.

Release done: before and after the incident

The new rows fall into three buckets.

Membership checks compare Service selectors to pod labels on the promoted revision, list Endpoints IPs, and verify at least one canary IP appears before analysis promotion is allowed to continue. These run in a post-sync Job, not in Argo's sync hook alone, because we want the Job to fail the release thread rather than silently retry sync.

Path checks run synthetics through the same Ingress hostname customers use, with TLS and the same Host header, and assert revision in the response. We deliberately moved one synthetic from in-cluster Service DNS to external Ingress to catch selector drift.

Analysis checks require PromQL queries to include traffic that entered through the Service front door. For checkout, that meant recording service_version on HTTP metrics at the Ingress access log and joining analysis queries to that series, not only to pod-local metrics.

We also demoted several signals from "done" to "informational." Rollout Healthy alone is informational. Synced alone is informational. Image digest match is informational. They still post to Slack. They no longer close the release.

Closing a release now writes a small JSON artifact to object storage: Git SHA, Helm revision, Endpoints snapshot hash, Ingress synthetic result, and the Conftest policy version. Auditors and future on-call engineers get a bundle that proves adoption, not just application.

The post-sync Job is the enforcement point. Argo runs it after sync with PreSync/PostSync hooks disabled for this path because we want a hard failure visible in the release thread, not a silent hook retry loop:

apiVersion: batch/v1
kind: Job
metadata:
  name: checkout-adoption-verify-418
  namespace: payments
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: verify
          image: registry/release-verify:1.4
          env:
            - name: EXPECTED_VERSION
              value: "v418"
            - name: SERVICE
              value: checkout-api
            - name: INGRESS_URL
              value: https://checkout.example.com/v1/intent
          command:
            - /bin/verify-adoption
            - --endpoints-required
            - --ingress-header-version

The verifier binary does three things in order: fail if Endpoints contain zero IPs with version=$EXPECTED_VERSION; fail if Ingress synthetic does not return the same version header; write the JSON artifact either way. The Job failure blocks the release bot from posting the final release-complete line. Slack shows blocked: adoption verify with a link to Job logs. That is intentionally unpleasant.

Route-level SLIs completed the picture. Namespace error budget remains for infrastructure weather. Checkout fee-application success is now a separate SLI with a 30-day target and a burn-rate alert wired to paging, not a dashboard tile. The SLI definition is narrow on purpose:

slos:
  - name: checkout-fee-application
    objective: 99.95
    sli:
      ratio:
        good:
          query: checkout_fee_apply_success_total{route="intent",version=~"v.*"}
        total:
          query: checkout_fee_apply_attempts_total{route="intent",version=~"v.*"}
    alert:
      burn_rate:
        short_window: 5m
        long_window: 1h
        factor: 14

During the incident this SLI would have burned inside two minutes because every attempt still ran v417 logic. We had the SLI spec in draft for a quarter and had not promoted it to paging because namespace budget was quiet. Quiet budget on the wrong aggregate is how progressive delivery accidents hide.

Running the deploy backwards

The runbook we wish we had used on Friday reads in reverse chronological order. Start from customer pain, walk upstream until you find the first object that disagrees with Git intent.

Reverse runbook: symptom first, Synced last

When checkout declines rise after a deploy, pull the revision customers actually hit. Ingress access logs with a service_version field are fastest if they exist. If not, read Endpoints before reading Rollout. Rollout tells you what should exist. Endpoints tells you what receives traffic. In our incident, Endpoints were the entire story.

If Endpoints list only old IPs, diff Service selector against pod labels on the new ReplicaSet. Do not assume Helm values tracks Rollout labels automatically. Check environment-specific values files that pin version: independently from chart appVersion.

If selectors match and Endpoints are still stale, then investigate kube-proxy or dataplane lag, EndpointSlice delay, or network policies. Those problems exist. They were not our problem that day. Skipping the selector diff because Rollout looked healthy sent three engineers on a thirty-minute side quest into CNI metrics that did not apply.

If analysis passed but customers see old behaviour, compare the PromQL query's label matchers to the path synthetic uses. Analysis traffic that never traverses the Service is a false friend. Fix the query or the route, not the canary weights.

Only after adoption is confirmed do we treat sync status and error budget as meaningful. Error budget without route-level SLI is a namespace weather report.

We rehearse this backwards runbook monthly on a game day. One engineer injects a selector drift into a staging chart. Another must identify adoption failure without looking at Git. The goal is muscle memory: Endpoints before Rollout, Ingress before in-cluster smoke.

Staging drill script (reverse order)

The drill fits on one page and deliberately avoids Argo UI until step six.

Step 1 — Symptom: Synthetic checkout POST returns fee line item schema=2024-1 when Git on main shows schema=2025-3 for revision 418.

Step 2 — Ingress proof: Search access logs for host=checkout.example.com and group by upstream_pod. If all upstream pods match v417 names, stop. Do not open the Rollout dashboard yet.

Step 3 — Endpoints proof:

kubectl get endpoints checkout-api -n payments \
  -o jsonpath='{range .subsets[*].addresses[*]}{.targetRef.name}{"\n"}{end}' \
  | sort -u

If the set lacks any pod from the canary ReplicaSet, the incident is membership, not application logic.

Step 4 — Selector diff: Render Helm locally and diff Service selector against Rollout pod labels. The two commands below should print matching version values:

helm template checkout-api charts/checkout -f values/prod-eu-1.yaml \
  | yq 'select(.kind=="Service" and .metadata.name=="checkout-api") | .spec.selector'
 
helm template checkout-api charts/checkout -f values/prod-eu-1.yaml \
  | yq 'select(.kind=="Rollout") | .spec.template.metadata.labels'

Mismatched version keys are the most common output in the drill. Less common but worse: stable Service name in Ingress does not match the Service the Rollout updates.

Step 5 — Rollout state: Only now read kubectl argo rollouts get rollout checkout-api -n payments. Healthy here means the controller did its job, not that customers migrated.

Step 6 — Git / Argo: Confirm Synced last. If you start at Synced, you can waste twenty minutes proving Git matches a Service that should never have routed traffic.

Drills surfaced a second failure mode we had not hit in production: dual Service pattern with Ingress still pointed at stable while stable selector froze. We added an Ingress annotation lint that warns when canaryService and stableService in the Rollout spec disagree with the Ingress backend Service name during chart review.

What this cost

Honest release done is slower and noisier. Post-sync Jobs add four to nine minutes on tier-one services. Conftest policies break when chart structure changes and someone must update the Rego input wiring. Tier-one readiness that checks Endpoints membership caused two false negatives during a legitimate EndpointSlice propagation delay; we added a short grace window with a hard cap.

Slack threads look worse before they look better. Releases that would have closed green now sit in "blocked: Endpoints missing v418" for six minutes while the Job reruns. That is the point. The previous green thread was cosmetic.

The alternative cost is what we paid Friday: a fee fix shipped in Git, celebrated in chat, and invisible to customers until declines showed up in a downstream report. Control-plane green is cheap. Data-plane wrong is expensive. We traded a few minutes per release for a category of false confidence we cannot afford on checkout.

Where the values file hid

The stale version: v417 selector didn't come from a bad merge in the Rollout template. It lived in values/prod-eu-1.yaml, a file platform owns but application teams read only when adding environment variables. When checkout moved from Deployment to Rollout in March, the Rollout template started stamping version: {{ .Chart.AppVersion }} on pods. Nobody removed the Service selector pin because the Service predated the Rollout change and still routed traffic correctly as long as v417 pods stayed up.

That is a common platform migration footgun: you introduce a new controller with new labels, but the routing object keeps a selector that made sense under the old controller's label scheme. GitOps sync propagates both faithfully. The bug is cross-object consistency, not drift from Git.

We now generate Service selectors from the same template helper as Rollout pod labels:

# templates/_helpers.tpl
{{- define "checkout.versionLabel" -}}
version: {{ .Values.image.tag | quote }}
{{- end }}

Both Service and Rollout reference the helper. Conftest still runs as belt-and-suspenders because template refactors break silently when someone copies YAML instead of including the helper.

Ownership mattered as much as tooling. Before the incident, "release done" was owned by the team merging the chart bump. After, tier-one adoption checks are owned by platform SRE and cannot be waived by the merging team. That separation feels bureaucratic until you watch a product team interpret Rollout Healthy as customer proof.

Why sync culture made this invisible

GitOps teams learn to trust Synced the way monolith teams learned to trust "build went green." The trust is earned for configuration drift. It is not earned for traffic membership. We had five years of Argo without a selector mismatch because Deployments and Services evolved together in one chart bump. Rollouts split the lifecycle: controller updates pod labels on one cadence, environment values pin Service selectors on another, Ingress annotations change rarely. Synced treats each object as an independent truth. Traffic is a cross-object truth nobody owned.

Platform reviews focused on chart best practices, resource limits, and network policy. Application reviews focused on fee logic unit tests. The gap between them was a label field in two YAML files that only meet in Endpoints, and Endpoints were not on either review checklist. That is not a tooling gap. It is an ownership gap that tooling surfaced once we looked.

I do not think the fix is "more Argo." The fix is treating adoption as an integration test between controllers, the same way we treat integration tests between microservices. Conftest, post-sync Jobs, and route SLIs are just how that integration test runs in a GitOps world.

Field: Cloud, DevOps & Reliability

References

• Argo Rollouts: Analysis and progressive delivery
• Kubernetes Services: virtual IPs and Endpoints
• Argo CD: Sync vs Health
• Google SRE Workbook: Canarying releases
• Open Policy Agent: Testing Kubernetes manifests with Conftest