How to Uninstall the Spherelet from an ESXi Host

If you need to uninstall the spherelet from an ESXi host, you might encounter some issues. This blog post will guide you through the process based on the following CLI outputs.

Initial Steps

First, check if the Spherelet is installed on the ESXi host:

[root@vcf-mgmt-esx1:~] esxcli software vib list | grep spherelet
spherelet                      2.3.2-24203661                        VMware  VMwareCertified   2024-12-27    host

If the Spherelet is listed, proceed with the uninstallation process.

Attempt Uninstallation

Try to remove the Spherelet VIB:

[root@vcf-mgmt-esx1:~] esxcli software vib remove -n spherelet

You might encounter the following error.

 [LiveInstallationError]
 VMware_bootbank_spherelet_2.3.2-24203661: Failed to unmount tardisk spherele.v00 of VIB VMware_bootbank_spherelet_2.3.2-24203661: Error in running [/bin/rm /tardisks/spherele.v00]:
 Return code: 1
 Output: rm: can't remove '/tardisks/spherele.v00': Device or resource busy

To resolve this, follow these steps:

Check Spherelet Service Status

Check the status of the Spherelet service:

[root@vcf-mgmt-esx1:~] /etc/init.d/spherelet status
2024-12-28 23:17:55,772 init.d/spherelet spherelet is running

Stop the Spherelet Service

Stop the Spherelet service to free up the resources:

[root@vcf-mgmt-esx1:~] /etc/init.d/spherelet stop
2024-12-28 23:18:02,329 init.d/spherelet spherelet stopped

Verify that the service has stopped:

[root@vcf-mgmt-esx1:~] /etc/init.d/spherelet status
2024-12-28 23:18:06,950 init.d/spherelet spherelet is not running

Remove the Spherelet VIB

After stopping the service, retry the VIB removal:

[root@vcf-mgmt-esx1:~] esxcli software vib remove -n spherelet
Removal Result
   Message: Operation finished successfully.
   VIBs Installed:
   VIBs Removed: VMware_bootbank_spherelet_2.3.2-24203661

Verify that the Spherelet VIB is removed:

[root@vcf-mgmt-esx1:~] esxcli software vib list | grep spherelet
[root@vcf-mgmt-esx1:~]

Troubleshooting: Supervisor Deployment Stuck at “Host Config Status – Configuring”

When deploying a Supervisor in VMware vSphere with Tanzu, it’s not uncommon to encounter an issue where the “Host Config Status” becomes stuck at “Configuring.” This blog post outlines the observations, steps to resolve the issue, and crucial details to consider in environments without internet access.

Observations

  • Stuck at “Configuring Host Nodes”:
    Each ESXi host shows the status “Configuring Host Nodes” stuck at “Installed and Started Kubernetes Node Agent on the ESXi Host.” See the screenshot below for reference.
  • Image Compliance Issues:
    Under Cluster > Updates > Image, you might see the message: “The cluster needs remediation to finish enabling these solutions: VMware vSphere with Kubernetes – Supervisor Cluster 8.0.3.” This suggests that certain components required for the Supervisor Cluster are not compliant with the desired image.
  • Verify Spherelet Installation:
    To verify if the Spherelet is installed on the ESXi hosts, SSH into the ESXi hosts and use the esxcli command:
   ssh root@<esxi-host>
   esxcli software vib list | grep spherelet
  • Check Spherelet Logs:
    Check the Spherelet logs to identify any issues. The logs are located at:
   /var/log/spherelet.log

Steps to Fix

Please note that in this environment, there’s no internet access. If the internet were available, the vLCM could remediate the cluster with the Spherelet VIBs. Here are the steps to manually resolve the issue:

  1. Copy the Spherelet VIB:
    Copy the Spherelet VIB from vCenter to each ESXi host. The Spherelet VIB is part of the following ZIP file in vCenter:
   /storage/lifecycle/vmware-wcp/spherelet/vsphere-wcp-depot.zip
  1. Extract and Upload the VIB:
    Extract the latest VIB from the ZIP file and upload it to the /tmp directory of each ESXi host.
  2. Install the VIB:
    Install the Spherelet VIB on each ESXi host using the esxcli command:
   esxcli software vib install -v /tmp/VMware_bootbank_spherelet_2.3.2-24203661.vib
  1. Confirm the VIB Installation:
    Verify that the VIB has been installed successfully by running:
   esxcli software vib get -n spherelet
  1. Login to VAMI of vCenter:
    Access the vCenter Appliance Management Interface (VAMI).
  2. Restart Workload Control Plane:
    Stop and start the ‘workload control plane’ service.
  3. Verify Configuration Completion:
    After performing these steps, you should see that the host configuration is completed.

Simplifying Host Commissioning in VMware Cloud Foundation with Bulk Commission

Managing large-scale VMware Cloud Foundation (VCF) environments can be challenging, especially when it comes to adding multiple hosts. The bulk commission feature, which uses a JSON template, simplifies this process significantly, making it more efficient and error-free. In this blog, we’ll walk through how to commission hosts in VCF using the bulk commission method with JSON, along with screenshots for each step.

Why Use Bulk Commission?

  1. Efficiency: Quickly add multiple hosts without repetitive manual steps.
  2. Consistency: Ensure all hosts are configured according to predefined standards.
  3. Scalability: Ideal for large environments, reducing administrative overhead.

Step-by-Step Guide to Bulk Commission Hosts Using JSON

Step 1: Prepare the JSON Template

First, create a JSON file that includes the details of the hosts you want to commission. Here’s an example template:

{
    "hostsSpec": [
        {
            "hostfqdn": "wd1-cls1-esx1.test.com",
            "username": "root",
            "storageType": "VSAN",
            "password": "password",
            "networkPoolName": "wd1-cls1-networkpool-001"
        },
        {
            "hostfqdn": "wd1-cls1-esx2.test.com",
            "username": "root",
            "storageType": "VSAN",
            "password": "password",
            "networkPoolName": "wd1-cls1-networkpool-001"
        },
        {
            "hostfqdn": "wd1-cls1-esx3.test.com",
            "username": "root",
            "storageType": "VSAN",
            "password": "password",
            "networkPoolName": "wd1-cls1-networkpool-001"
        }
    ]
}

Ensure that each host meets the necessary criteria for VCF, such as compliance with the VMware Hardware Compatibility Guide.

Step 2: Upload the JSON Template to SDDC Manager
  1. Log in to the SDDC Manager.
  2. Navigate to the ‘Hosts’ section.
  3. Click on ‘Commission Hosts’.
  4. ‘Select All’ in the checklist and click on Proceed
  5. Click on ‘Import’ and Upload the JSON file containing your host details.
Step 3: Validate and Commission the Hosts
  1. SDDC Manager will validate the JSON template and the hosts listed.
  2. Review the validation results. If any issues are found, correct them in the JSON file and re-upload.
  3. Confirm the commissioning to proceed with adding the hosts to the SDDC Manager.
Step 4: Monitor the Commissioning Process
  1. Monitor the progress in the SDDC Manager dashboard.
  2. Check for any errors or warnings during the process and resolve them as needed.

Conclusion

Using the bulk commission feature in VMware Cloud Foundation with a JSON template streamlines the process of adding multiple hosts, making it faster and more reliable. This method not only enhances efficiency but also ensures consistency across your infrastructure. By following these steps, you can easily scale up your VCF environment with minimal effort.

Updating the Harbor Certificate on Your TKG Cluster (vCD 10.5.1, CSE 4.2)

This blog post guides you through replacing the certificate for your Harbor registry deployed on a Tanzu Kubernetes Grid (TKG) cluster using Helm charts. We’ll assume you’re using VCD version 10.5.1 and Container Service Extension (CSE) version 4.2.

Understanding the Need for Certificate Replacement

Harbor certificates, like any security certificate, may need to be replaced due to expiration, security upgrades, or changes in your PKI infrastructure. This process ensures secure communication within your container registry.

Prerequisites

  • Access to your TKG cluster and kubectl CLI.
  • New certificate and key files (harbor-v2.crt and harbor-v2.key).

Steps:

  1. Create a New Secret:
  • We’ll store the new certificate and key in a Kubernetes secret for secure management. Use the kubectl create secret tls command to create a secret named harbor-secret-v2:
   kubectl create secret tls harbor-secret-v2 --cert=harbor-v2.crt --key=harbor-v2.key --namespace=harbor-system

Replace harbor-v2.crt and harbor-v2.key with the actual filenames of your certificate and key files.

  1. Update the values.yaml file:
  • The values.yaml file defines various configurations for your Harbor deployment. Locate this file from your initial Harbor deployment process.
  • Edit the values.yaml file to point to the newly created secret. Look for the certSource section and update it to use secrets:
   certSource: secret
   secret:
     secretName: "harbor-secret-v2"

This configures Harbor to use the certificate and key stored in the harbor-secret-v2 secret.

  1. Upgrade Harbor with the New Configuration:
  • Deploy the updated configuration with the new certificate using the helm upgrade command:
   helm upgrade harbor harbor -n harbor-system -f ./values.yaml

This command upgrades the harbor deployment in the harbor-system namespace using the configuration specified in the updated values.yaml file.

Conclusion

By following these steps, you’ve successfully replaced the certificate for your Harbor registry deployed on your TKG cluster. Remember to update your Harbor clients or local configurations to reflect the new certificate details for continued secure communication.

Simplifying VM Deployments with vGPU Policies in VMware Cloud Director 10.5.1

Virtual Graphics Processing Units (vGPUs) are a game-changer for cloud deployments, enabling high-performance graphics processing for workloads like 3D design, video editing, and AI applications within virtual machines (VMs). VMware Cloud Director (VCD) streamlines vGPU management through vGPU policies, allowing you to define the allocation of these powerful resources to your VMs.

This blog post will guide you through creating a vGPU policy in VCD, ensuring your VMs have the graphics horsepower they need:

Prerequisites:

  • Access to the VCD Provider Portal with administrative privileges.
  • Pre-configured vGPU profiles in VCD. These profiles represent the different types of vGPUs available in your environment, typically created from the capabilities of your underlying vSphere cluster with NVIDIA GPUs.

Creating a vGPU Policy:

  • Log in to the VCD Provider Portal with your administrative credentials.
  • Verify vGPU Profile Visibility: Navigate to Infrastructure Resources > vGPU Profiles. Ensure the vGPU profiles corresponding to your available GPUs are listed here. If not, you’ll need to create them beforehand (refer to your VCD documentation for specific steps).
  • Create the vGPU Policy:
    • Go to Cloud Resources > vGPU Policies.
    • Click New.
    • On the “What is a vGPU Policy?” screen, click Next.
  • Define Policy Details:
    • Name: Enter a descriptive name for your vGPU policy. Ideally, match it to the vGPU profile it references for clarity (e.g., “High Performance vGPU”).
  • vGPU Profile: Select the vGPU profile that defines the type and capabilities of the vGPU to be assigned.
  • Provider VDC Scope : Choose the PVDC has access to the poloicy.
  • Placement: Choose No for placement flexibility. You can assign this policy to VMs and let VCD determine optimal placement based on available resources.
  • Sizing: Select No for sizing flexibility. You can configure VM CPU, memory, and storage independently during VM deployment.
  • Finalize the Policy:
    • Select the Organization VDC where you want this policy to be available.
    • Review the policy details on the “Ready to Complete” screen and click Finish to create the vGPU policy.

Congratulations! You’ve successfully created a vGPU policy in VCD. Now, when deploying VMs in the chosen Organization VDC, you can assign this policy to provide the necessary vGPU power for your graphics-intensive workloads.

Additional Considerations:

  • You can create multiple vGPU policies with different vGPU profiles to cater to varying VM requirements.
  • For more granular control, explore the options for placement and sizing policies within VCD, allowing you to define specific placement rules and resource allocation for vGPU-enabled VMs.

By leveraging vGPU policies, you can efficiently manage and allocate vGPU resources within your VCD environment, empowering your tenants with the graphics processing capabilities they need for their demanding workloads.

Enable ssh login in CSE Ubuntu K8s nodes

  • Login to VMware Cloud Director tenant/provider portal.
  • Access console of the k8s master node.
  • Login as root. You can find the root user password in Guest customization section of the VM in VMware Cloud Director.
  • Open /etc/sshd/sshd_config
  • In the file, find the PasswordAuthentication line and make sure it ends with yes.
  • Save the file and exit.
  • Restart the SSH service by typing the following command:
sudo systemctl restart sshd
  • Its good to create a user for ssh login rather than enabling root login over ssh.
#useradd kube
#passwd kube
  • You can ssh to the master node with kube user and do ‘su -‘.

How to validate TKGm Cluster?

Please find the steps to validate a TKGm cluster deployed through VMware Container Service Extension.

Step 1 : Download kubeconfig file

  • Download the Kubeconfig file to a windows machine which has access to the Native Kubernetes cluster.
  • Create folder .kube under $HOME.

$HOME\.kube

  • Copy the configfile dowloaded to .kube folder.
  • Rename the file to ‘config’ without any extensions.

Step 2 : Download kubectl

  • Download Kubectl for Windows from
https://dl.k8s.io/release/v1.22.0/bin/windows/amd64/kubectl.exe
  • Create folder $HOME\kubectl and copy kubectl.ext to the folder. Add the folder to the ‘Path’ User variable in Environment Variables.

Run kubectl

Step 3: Run a ‘hello world’ application in the cluster.

Follow the steps from following article to deploy a Hello World applicaiton in the K8S cluster created.

Exposing an External IP Address to Access an Application in a Cluster | Kubernetes

Note: In the following command use NodePort instead of LoadBalancer

kubectl expose deployment hello-world --type=LoadBalancer --name=my-service

How to create NSX-T Routed network in VCD for Tanzu Kubernetes Grid (TKG) clusters?

Please find the steps for configuring the Network in VCD for deploying TKG clusters.

Add the public IP to the Static IP Pool of T0 GW

  • Login to VCD Provider portal.
  • Navigate to Resources > Cloud Resources > Tier-0 Gateways.
  • Select the T0 Gateway.
  • Select ‘Network Specification’
  • Edit
  • Add the Public IP(s) to the ‘Static IP Pools’

Create Edge Gateway (T1 Router)

  • Login to VCD Provider portal.
  • Navigate to Resources > Cloud Resources. > Edge Gateways
  • Select New
  • Select the Org VDC and click Next
  • Provide a name for the Edge.
  • Select the appropriate T0 Gateway
  • Choose the appropriate Edge Cluster option for your environment.
  • Assign the Public IP for SNAT as Primary IP
  • Cleck Next review the settings and click Finish.

Create Organization Network

  • From provider portal select the Test organization.
  • Navigate to Networking > Networks.
  • Click New
  • Select Org VDC
  • Select Network Type ‘Routed
  • Select the Edge Gateway (T1)
  • Provide the Name and Gateway CIDR
  • Provide the DNS server accessible from the Org Network created. The DNS server should be able to resolve the FQDNS in the public domain/internet.
  • Click Next, review the settings and click on Finish.

Create SNAT

  • From provider portal select the Test organization.
  • Navigate to Networking > Edge Gateways
  • Select the Edge Gateway (T1)
  • Navigate to Services > NAT
  • Click New
  • Provide the details as mentioned in the screenshot.

Modify default Firewall rule

  • From provider portal select the Test organization.
  • Navigate to Networking > Edge Gateways
  • Select the Edge Gateway (T1)
  • Navigate to Services > Firewall
  • Select ‘Edit Rules’
  • Select the ‘default_rule’
  • Edit
  • Select Allow as Action.

How to run VMware Container Service Extension (CSE) as Linux Service?

After installing CSE please follow the steps below to run it as a service.

Create cse.sh file

Create cse.service file. You can copy the following code or create new one based on following link.
container-service-extension/cse.sh at master · vmware/container-service-extension (github.com)

# vi /opt/vmware/cse/cse.sh
#!/usr/bin/env bash
export CSE_CONFIG=/opt/vmware/cse/encrypted-config.yaml
export CSE_CONFIG_PASSWORD=<passwd>
cse run

Copy encrypted-config.yaml to /opt/vmware/cse directory.

Change the file permission

chmod +x /opt/vmware/cse/cse.sh

Create cse.service file

Create cse.service file. You can copy the following code or create new one based on following link.
container-service-extension/cse.service at master · vmware/container-service-extension (github.com)

vi /etc/systemd/system/cse.service
[Unit]
Description=Container Service Extension for VMware Cloud Director

[Service]
ExecStart=/opt/vmware/cse/cse.sh
User=root
WorkingDirectory=/opt/vmware/cse
Type=simple
Restart=always

[Install]
WantedBy=default.target

Enable and start the service

# systemctl enable cse.service
# systemctl start cse.service

Check the service status

# systemctl status cse.service
  cse.service - Container Service Extension for VMware Cloud Director
   Loaded: loaded (/etc/systemd/system/cse.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-11-24 14:43:56 +01; 1min 9s ago
 Main PID: 770 (bash)
   CGroup: /system.slice/cse.service
           ├─770 bash /opt/vmware/cse/cse.sh
           └─775 /usr/local/bin/python3.7 /usr/local/bin/cse run

Nov 24 14:44:06 cse01.lab.com cse.sh[770]: Validating CSE installation according to config file
Nov 24 14:44:06  cse.sh[770]: MQTT extension and API filters found
Nov 24 14:44:06 cse01.lab.com cse.sh[770]: Found catalog 'cse-site1-k8s'
Nov 24 14:44:06 cse01.lab.com  cse.sh[770]: CSE installation is valid
Nov 24 14:44:06 cse01.lab.com cse.sh[770]: Started thread 'MessageConsumer' (140229531580160)
Nov 24 14:44:06 cse01.lab.com l cse.sh[770]: Started thread 'ConsumerWatchdog' (140229523187456)
Nov 24 14:44:06 cse01.lab.com  cse.sh[770]: Container Service Extension for vCloud Director
Nov 24 14:44:06 cse01.lab.com  cse.sh[770]: Server running using config file: /opt/vmware/cse/encrypted-config.yaml
Nov 24 14:44:06 cse01.lab.com  cse.sh[770]: Log files: /root/.cse-logs/cse-server-info.log, /root/.cse-logs/cse-server-debug.log
Nov 24 14:44:06 cse01.lab.com  cse.sh[770]: waiting for requests (ctrl+c to close)

Upgrade vRealize Operations Management Pack for vCloud Director from 5.5 to 8.6

I’ve recently upgraded vRealize operations Manager from 8.4 to 8.6. The installed version of vROPs Management Pack for VCD was 5.5, which is incompatible with vROps 8.6 and VMware Cloud Director 10.3.1. To make it compatible I had to upgrade vROps Management Pack for VCD to 8.6.


Please find the steps below to upgrade the Management Pack.

  1. Download the following vRealize Operations Management Pack for vCloud Director 8.6 from VMware Marketplace.
    • vmware-vcd-mp-8-1634219770748.pak
  2. Once downloaded, login to vRealize Operations Manager 8.6 UI – https://<vROps FQDN/IP>/ui
  3. Navigate to Data Sources > Integrations > Repository.
  4. From ‘Installed Integrations‘ locate ‘Management Pack for VMware Cloud Director.

5. Click on More Options menu and select Upgrade.

5. Select the Install the PAK file even if it is already installed check box.

This selection reloads the PAK file (Management Pack) but retains the custom preferences of the user. Also, this selection does not overwrite or update the solution alerts, symptoms, recommendations, and policies.

6. Select the Reset Default Content, overwriting to a newer version provided by this update check box.

This selection reloads the PAK file and overwrites the existing solution alerts, symptoms, recommendations, and policies with newer versions provided with the current PAK file.

WARNING: User modifications to DEFAULT Alert Definitions, Symptoms, Recommendations, Policy Definitions, Views, Dashboards, Widgets and Reports supplied by the current version of Management Pack will be overwritten. To save your modifications to default content, clone or backup the content before you proceed.

7. Click on Upload.

8. Click Next.
9. Read and accept the EULA and click Next. The install might take several minutes to complete.
10. Click Finish once the installation is completed.

11. Confirm the Upgrade is completed by checking the version of Management Pack. ‘More Options‘ > About.

12. Check and confirm the ‘Cloud Director Adapter‘ is collecting the data from VCD. The status of Cloud Director Adapter should be OK.

  • Navigate to Data Sources > Integrations > Accounts > Cloud Director Adapter.