Fixing a Stuck vSphere Supervisor Upgrade (Stage 16): Why cpvmpod_validator Blocked Control-Plane Pods

Added , By with No Comments

Upgrading a vSphere Supervisor cluster in an air-gapped environment can occasionally stall at Stage 16 (Configured Core Supervisor Services), even when core control-plane nodes and networking are completely healthy.

In this post, we walk through a real-world failure where ://vmware.com failed to reconcile during a Supervisor upgrade to Kubernetes v1.32.9, how we traced it from a generic ProgressDeadlineExceeded error down to a new App Platform AdmissionPolicyConfig security rule, and the exact fix that unblocked the upgrade.

Executive Summary

  • Symptom: The vSphere Supervisor upgrade hung at Stage 16 with Service: ://vmware.com. Reason: ReconcileFailed, citing Deployment is not progressing: ProgressDeadlineExceeded on machine-agent-server in svc-tkg-domain-c10.
  • Root Cause: During the upgrade to Kubernetes v1.32.9, vSphere App Platform introduced AdmissionPolicyConfig, which restricts which namespaces can schedule pods onto Supervisor Control Plane VMs. Because svc-://vmware.com was pinned to an external custom package version (3.6.2+v1.35) rather than the built-in embedded version (3.6.3-embedded+v1.35), App Platform excluded svc-tkg-domain-c10 from its firstParty trusted allowlist. Consequently, the ://vmware.com webhook blocked all control-plane pod creation in that namespace.
  • Resolution: Patching the svc-://vmware.com PackageInstall to target 3.6.3-embedded+v1.35 restored first-party trust in AdmissionPolicyConfig, allowing all TKG/Cluster API pods to schedule and Stage 16 to complete automatically.

The Symptom: Stage 16 Stuck on ://vmware.com

During an air-gapped Supervisor upgrade to v1.32.9, the upgrade workflow stalled at Stage 16 with the following error:

Configured Core Supervisor Services
Service: ://vmware.com. Reason: ReconcileFailed. Message: kapp: Error: waiting on reconcile packageinstall/runtime-extension (packaging.carvel.dev/v1alpha1) namespace: svc-tkg-domain-c10: Finished waiting unsuccessfully: Reconcile failed: message: kapp: Error: waiting on reconcile deployment/machine-agent-server (apps/v1) namespace: svc-tkg-domain-c10: Finished waiting unsuccessfully: Deployment is not progressing: ProgressDeadlineExceeded, message: ReplicaSet "machine-agent-server-57bcd9cff6" has timed out progressing..
Service: ://vmware.com. Status: Running

Checking kubectl get pkgi -A confirmed that while velero and core system services were healthy, svc-://vmware.com and almost all of its child Carvel packages inside svc-tkg-domain-c10 (runtime-extension, tanzu-cluster-api, tanzu-cluster-api-provider-vsphere, tkg-controller, etc.) were in a Reconcile failed state.

Step-by-Step Investigation

1. Checking for Failing Pods (And Finding None)

Normally, ProgressDeadlineExceeded points to an ImagePullBackOff or CrashLoopBackOff—especially in air-gapped environments. However, listing the pods in svc-tkg-domain-c10 revealed something unexpected:

kubectl get pods -n svc-tkg-domain-c10 | grep machine-agent-server
# (empty output)

Not only was machine-agent-server missing, but capv-controller-manager and capi-controller-manager were also completely absent. Kubernetes was not failing to start the containers—it was blocking the creation of the pod objects altogether.

2. Inspecting Namespace Events

Checking the ReplicaSet and namespace events immediately revealed the blocker:

kubectl get events -n svc-tkg-domain-c10 --sort-by='.lastTimestamp' | tail -n 10
Warning   FailedCreate   replicaset/capv-controller-manager-867d5b49c9   Error creating: admission webhook "://vmware.com" denied the request: running pod svc-tkg-domain-c10/capv-controller-manager-867d5b49c9-bkg4 in an untrusted namespace on the control plane VM is not allowed

Every pod spec targeting node-role.kubernetes.io/control-plane= inside svc-tkg-domain-c10 was being rejected by the App Platform validating webhook (://vmware.com) because the namespace was considered untrusted.

3. Tracing Trust Rules to AdmissionPolicyConfig

Inspecting the App Platform CRDs revealed two new resources created during the upgrade: ://vmware.com and ://vmware.com.

Looking inside admission-policy-config showed the exact firstParty allowlist evaluated by cpvmpod_validator:

spec:
  firstParty:
    namespaces:
    - svc-velero-domain-c10
    - svc-cci-ns-domain-c10
    serviceAccounts:
    - system:serviceaccount:vmware-system-supervisor-services:kube-state-metrics-pkg-sa
    - system:serviceaccount:vmware-system-supervisor-services:sa-velero-0m611
    - system:serviceaccount:vmware-system-supervisor-services:sa-cci-ns-5zrfq

svc-tkg-domain-c10 and its service account (sa-tkg-vgw6i) were completely missing from firstParty.

4. Finding Why App Platform Excluded svc-tkg-domain-c10

Comparing the embedded-service-versions ConfigMap against the active PackageInstall (svc-://vmware.com) exposed the root cause:

  • embedded-service-versions ConfigMap: Expected ://vmware.com: 3.6.3-embedded+v1.35.
  • svc-://vmware.com PackageInstall: Was pinned to 3.6.2+v1.35, an out-of-band package previously imported from an external private Harbor registry.

Because ://vmware.com was running a non-embedded external package version rather than the signed embedded version bundled with the Supervisor upgrade, App Platform revoked its first-party trust status during the upgrade’s security policy enforcement.

The Resolution

First, we verified that the embedded 3.6.3-embedded+v1.35 package was already present in the Supervisor’s local package repository and configured to pull images from the internal registry (docker-registry.kube-system.svc:5000), ensuring full air-gap compatibility:

kubectl get pkg ://vmware.com.3.6.3-embedded+v1.35 -n vmware-system-supervisor-services -o yaml | grep -i image
# image: docker-registry.kube-system.svc:5000/tkg-svs/package/tkg-service:3.6.3

After backing up the existing PackageInstall and AdmissionPolicyConfig manifests, we patched svc-://vmware.com to select 3.6.3-embedded+v1.35:

kubectl patch pkgi svc-://vmware.com \
  -n vmware-system-supervisor-services \
  --type merge \
  -p '{"spec":{"packageRef":{"versionSelection":{"constraints":"3.6.3-embedded+v1.35"}}}}'

Immediate Impact

  1. Trust Restored Automatically: Within seconds, App Platform detected that ://vmware.com was reconciled against an official embedded version and automatically added svc-tkg-domain-c10 and sa-tkg-vgw6i back into admission-policy-config under spec.firstParty.
  2. Pods Scheduled: ://vmware.com stopped rejecting pod creation, and machine-agent-server, capv-controller-manager, and capi-controller-manager spun up cleanly.
  3. Upgrade Completed: Once all child PackageInstall objects reached Reconcile succeeded, Stage 16 detected Service: ://vmware.com. Status: Running and finished the Supervisor upgrade without disrupting any running workload clusters.

Key Takeaways for Future Air-Gapped Supervisor Upgrades

  • Watch for Out-of-Band Service Overrides: If you previously uploaded a custom or hotfix TKG/VKS Supervisor Service YAML from a private registry, ensure you either switch it back to the -embedded version before a Supervisor upgrade or stage the matching updated version in your registry beforehand.
  • Check AdmissionPolicyConfig When Control-Plane Pods Vanish: In vSphere Supervisor v1.32+, if a Deployment reports ProgressDeadlineExceeded with zero pods created, check namespace events for ://vmware.com denials and verify spec.firstParty in admission-policy-config.