Deploying a Tanzu Kubernetes cluster using tanzu CLI in vSphere with Tanzu

Regular readers will have seen a number of articles on this site which use the tanzu command line to create and delete TKGm clusters. TKGm is the nomenclature that I am using to describe multi-cloud TKG clusters (also known as standalone TKG clusters) that can be deployed onto numerous different IaaS, including vSphere. In this post, I want to show you how to use the same tanzu CLI tools to deploy a Tanzu Kubernetes cluster via the TKG service (TKGS) on vSphere with Tanzu. I have always shown that to deploy TKG clusters on vSphere with Tanzu, you login to the Supervisor cluster, select the appropriate namespace, populate a YAML manifest with the cluster details, and then deploy the cluster using kubectl. Here will explore an alternative way of doing it with the tanzu CLI.

A number of assumptions around pre-requisites are made in this post.

  1. Tanzu CLI is already installed. See official documentation for details.
  2. vSphere with Tanzu is installed. This post described a deployment on vSphere 7.0U3c.
  3. A namespace has been created. In my case, it is called workload. This can be created via the Namespace Service through the CLI or via the UI

With these requirements in place we can proceed.

Step 1: Add a user to Namespace Permissions

Ensure that there is a valid account added to the permissions tab of the Namespace, and that you use those account privileges when accessing the cluster context. I am using the administrator SSO account for simplicity, but you may wish to use something more appropriate.

Step 2: Add a Storage Class / Storage Policy to Namespace for TKG PVs

When we build the workload cluster, it will require at least one Storage Class. The Storage Class is used for the root file system disks on both the control plane nodes and the worker nodes when the cluster is created. A Storage Class is also necessary if you wish to create applications in the workload cluster that have a persistent volume requirement. In this example, I am adding two Storage Policies (both related to vSAN) to the Namespace. These Storage Policies appear as Storage Classes when queried via kubectl.

% kubectl-vsphere login --server=https://xx.xx.62.18 --vsphere-username administrator@vsphere.local --insecure-skip-tls-verify

Logged in successfully.

You have access to the following contexts:
  xx.xx.62.18
  workload

If the context you wish to use is not in this list, you may need to try
logging in again later, or contact your cluster administrator.


% kubectl config use-context workload
Switched to context "workload".


% kubectl get sc
NAME                          PROVISIONER              RECLAIMPOLICY  VOLUMEBINDINGMODE  ALLOWVOLUMEEXPANSION  AGE
raid1                        csi.vsphere.vmware.com  Delete          Immediate          true                  19h
vsan-default-storage-policy  csi.vsphere.vmware.com  Delete          Immediate          true                  19h
%

Step 3: Add a VM Class to Namespace for TKG Nodes

We now need to add at least one virtual machine class to the Namespace. There are already a number of example VM classes provided that can be selected as needed, or you can also build your own VM Class. Querying VM classes via kubectl is a little strange. The command kubectl get vmclass will display any VM classes that have been assigned to any namespace on the Supervisor cluster. However, it will not show any classes that have not been assigned, so it will be difficult to see the available VM classes from the CLI in most clusters. Thus, you will definitely need to visit to vSphere client UI to see what is available, especially if this is the first namespace. In this example, I am assigning a single VM class guaranteed-small to the namespace. Once the VM class is associated with the  namespace, you should be able to verify the association via the kubectl get vmclassbinding command.

% kubectl get vmclass
NAME              CPU  MEMORY  AGE
guaranteed-small  2    4Gi      19h


% kubectl get vmclassbinding
NAME              VIRTUALMACHINECLASS  AGE
guaranteed-small  guaranteed-small      19h
%

I have also added a Content Library for the VM Classes in this namespace. This step is not a requirement for this exercise.

Step 4: Select a Tanzu Kubernetes release (TKr)

The next step is to decide which release of Kubernetes to include in our cluster. The command kubectl get tkr (short for tanzukubernetesreleases) will display the list of available releases, and whether or not they are compatible. These can then be added to the cluster configuration file, along with the storage class and VM class.

% kubectl get tkr
NAME                                VERSION                          READY  COMPATIBLE  CREATED  UPDATES AVAILABLE
v1.16.12---vmware.1-tkg.1.da7afe7   1.16.12+vmware.1-tkg.1.da7afe7   True    True        3d23h    [1.17.17+vmware.1-tkg.1.d44d45a 1.16.14+vmware.1-tkg.1.ada4837]
v1.16.14---vmware.1-tkg.1.ada4837   1.16.14+vmware.1-tkg.1.ada4837   True    True        3d23h    [1.17.17+vmware.1-tkg.1.d44d45a]
v1.16.8---vmware.1-tkg.3.60d2ffd    1.16.8+vmware.1-tkg.3.60d2ffd    False   False       3d23h    [1.17.17+vmware.1-tkg.1.d44d45a 1.16.14+vmware.1-tkg.1.ada4837]
v1.17.11---vmware.1-tkg.1.15f1e18   1.17.11+vmware.1-tkg.1.15f1e18   True    True        3d23h    [1.18.15+vmware.1-tkg.2.ebf6117 1.17.17+vmware.1-tkg.1.d44d45a]
v1.17.11---vmware.1-tkg.2.ad3d374   1.17.11+vmware.1-tkg.2.ad3d374   True    True        3d23h    [1.18.15+vmware.1-tkg.2.ebf6117 1.17.17+vmware.1-tkg.1.d44d45a]
v1.17.13---vmware.1-tkg.2.2c133ed   1.17.13+vmware.1-tkg.2.2c133ed   True    True        3d23h    [1.18.15+vmware.1-tkg.2.ebf6117 1.17.17+vmware.1-tkg.1.d44d45a]
v1.17.17---vmware.1-tkg.1.d44d45a   1.17.17+vmware.1-tkg.1.d44d45a   True    True        3d23h    [1.18.15+vmware.1-tkg.2.ebf6117]
v1.17.7---vmware.1-tkg.1.154236c    1.17.7+vmware.1-tkg.1.154236c    True    True        3d23h    [1.18.15+vmware.1-tkg.2.ebf6117 1.17.17+vmware.1-tkg.1.d44d45a]
v1.17.8---vmware.1-tkg.1.5417466    1.17.8+vmware.1-tkg.1.5417466    True    True        3d23h    [1.18.15+vmware.1-tkg.2.ebf6117 1.17.17+vmware.1-tkg.1.d44d45a]
v1.18.10---vmware.1-tkg.1.3a6cd48   1.18.10+vmware.1-tkg.1.3a6cd48   True    True        3d23h    [1.19.7+vmware.1-tkg.2.f52f85a 1.18.15+vmware.1-tkg.2.ebf6117]
v1.18.15---vmware.1-tkg.1.600e412   1.18.15+vmware.1-tkg.1.600e412   True    True        3d23h    [1.19.7+vmware.1-tkg.2.f52f85a 1.18.15+vmware.1-tkg.2.ebf6117]
v1.18.15---vmware.1-tkg.2.ebf6117   1.18.15+vmware.1-tkg.2.ebf6117   True    True        3d23h    [1.19.7+vmware.1-tkg.2.f52f85a]
v1.18.5---vmware.1-tkg.1.c40d30d    1.18.5+vmware.1-tkg.1.c40d30d    True    True        3d23h    [1.19.7+vmware.1-tkg.2.f52f85a 1.18.15+vmware.1-tkg.2.ebf6117]
v1.19.7---vmware.1-tkg.1.fc82c41    1.19.7+vmware.1-tkg.1.fc82c41    True    True        3d23h    [1.20.7+vmware.1-tkg.1.7fb9067 1.19.7+vmware.1-tkg.2.f52f85a]
v1.19.7---vmware.1-tkg.2.f52f85a    1.19.7+vmware.1-tkg.2.f52f85a    True    True        3d23h    [1.20.7+vmware.1-tkg.1.7fb9067]
v1.20.2---vmware.1-tkg.1.1d4f79a    1.20.2+vmware.1-tkg.1.1d4f79a    True    True        3d23h    [1.20.7+vmware.1-tkg.1.7fb9067]
v1.20.2---vmware.1-tkg.2.3e10706    1.20.2+vmware.1-tkg.2.3e10706    True    True        3d23h    [1.20.7+vmware.1-tkg.1.7fb9067]
v1.20.7---vmware.1-tkg.1.7fb9067    1.20.7+vmware.1-tkg.1.7fb9067    True    True        3d23h

In this example, I am going to use the latest release, which is version 1.20.7.

Step 5: Create a TKG cluster configuration file

With all of the previous information gathered, I can now proceed with the creation of the configuration file for my Tanzu Kubernetes cluster. An example of a configuration file, with various comments, can be found in the official documentation here. Here is a simplified example that I am going to use to deploy my cluster. I am going to deploy a simple “dev” plan cluster made up of a single control plane node and a single worker node. I am not enabling MachineChecks or AutoScaler. I am using the same VM class and the same Storage Class for both the control plane and the worker node. I have also give the cluster a name, and set the namespace context appropriately.

% cat my-vsphere-tkc.yaml
INFRASTRUCTURE_PROVIDER: tkg-service-vsphere
CLUSTER_PLAN: dev
CLUSTER_NAME: workload1
NAMESPACE: workload
CNI: antrea
CONTROL_PLANE_STORAGE_CLASS: raid1
CONTROL_PLANE_VM_CLASS: guaranteed-small
SERVICE_DOMAIN: rainpole.com
STORAGE_CLASSES: vsan-default-storage-policy
WORKER_STORAGE_CLASS: raid1
WORKER_VM_CLASS: guaranteed-small
SERVICE_CIDR: 100.64.0.0/13
CLUSTER_CIDR: 100.96.0.0/11
ENABLE_MHC: false
ENABLE_AUTOSCALER: false

Step 6: Connect Supervisor context via tanzu CLI, and deploy a TKG

The next step is to use the tanzu CLI to connect to the Supervisor. My Supervisor cluster context is still set, from my login in step 1, so I just need to use the correct tanzu login syntax. Once set, I can now use the tanzu CLI to deploy the cluster, specifying the configuration file and the desired TKr.

% tanzu login --context xx.xx.62.18
? Give the server a name xx.xx.62.18
✔  successfully logged in to management cluster using the kubeconfig xx.xx.62.18


% tanzu cluster create --file ./my-vsphere-tkc.yaml --tkr=v1.20.7---vmware.1-tkg.1.7fb9067
You are trying to create a cluster with kubernetes version '1.20.7+vmware.1-tkg.1.7fb9067' on vSphere with Tanzu, 
Please make sure virtual machine image for the same is available in the cluster content library.
Do you want to continue? [y/N]: y
Validating configuration...
Error: failed to apply the cluster configuration: kubectl apply failed, output: 
tanzukubernetescluster.run.tanzu.vmware.com/workload1 unchanged
secret/workload1-capabilities configured
secret/workload1-config-values configured
Error from server (Forbidden): error when retrieving current configuration of:
Resource: "addons.cluster.x-k8s.io/v1alpha3, Resource=clusterresourcesets", 
GroupVersionKind: "addons.cluster.x-k8s.io/v1alpha3, Kind=ClusterResourceSet"
Name: "workload1-capabilities", Namespace: "workload"
from server for: "/var/folders/lv/b1078qn55y52sq__y1533nlc0000gp/T/kubeapply-3429424587": 
clusterresourcesets.addons.cluster.x-k8s.io "workload1-capabilities" is forbidden: 
User "sso:Administrator@vsphere.local" cannot get resource "clusterresourcesets" 
in API group "addons.cluster.x-k8s.io" in the namespace "workload"
: exit status 1

✖  exit status 1

Note that the error thrown during the validation is expected. It does not prevent the cluster from being deployed. It is also documented as something which can be ignored in the TKG v1.4 Release Notes Known Issues.

Step 7: Query the status of the workload cluster

You can query the status of the cluster using various commands.

% kubectl get cluster -n workload
NAME        PHASE
workload1  Provisioned


% kubectl get tanzukubernetescluster -n workload
NAME        CONTROL PLANE  WORKER  TKR NAME                          AGE  READY  TKR COMPATIBLE  UPDATES AVAILABLE
workload1  1              1        v1.20.7---vmware.1-tkg.1.7fb9067  16m  True   True


% tanzu cluster list
  NAME      NAMESPACE  STATUS  CONTROLPLANE  WORKERS  KUBERNETES                    ROLES  PLAN
  workload1  workload  running  1/            1/      1.20.7+vmware.1-tkg.1.7fb9067  <none>


% tanzu cluster get workload1 -n workload
  NAME      NAMESPACE  STATUS  CONTROLPLANE  WORKERS  KUBERNETES                    ROLES
  workload1  workload  running  1/            1/      1.20.7+vmware.1-tkg.1.7fb9067  <none>
ℹ

Details:

NAME                                                          READY  SEVERITY  REASON  SINCE  MESSAGE
/workload1                                                    True                     13m
├─ClusterInfrastructure - WCPCluster/workload1                True                     17m
├─ControlPlane - KubeadmControlPlane/workload1-control-plane  True                     13m
│ └─Machine/workload1-control-plane-jc5gh                     True                     14m
└─Workers
  └─MachineDeployment/workload1-workers-6nh9p
    └─Machine/workload1-workers-6nh9p-6ff9659d78-l62cw        True                     11m


% kubectl get virtualmachines
NAME                                      POWERSTATE  AGE
workload1-control-plane-jc5gh             poweredOn    25m
workload1-workers-6nh9p-6ff9659d78-l62cw  poweredOn    21m

Step 8: Access the cluster context

The cluster context can now be accessed using either the kubectl-vsphere command, or using the tanzu CLI. I have provided examples of how to do both below.

8.1 Using the tanzu CLI

In the first example, we use the tanzu CLI. I have repeated the login steps and connecting the tanzu CLI with the context, but these may not be necessary if you are still in the same context which we setup earlier.

% kubectl-vsphere login --server xx.xx.62.18 --vsphere-username administrator@vsphere.local 
--insecure-skip-tls-verify

Logged in successfully.

You have access to the following contexts:
  xx.xx.62.18
  workload

If the context you wish to use is not in this list, you may need to try
logging in again later, or contact your cluster administrator.

To change context, use `kubectl config use-context <workload name>`


% tanzu login --context xx.xx.62.18
? Select a server xx.xx.62.18 ()
✔  successfully logged in to management cluster using the kubeconfig xx.xx.62.18


% tanzu cluster list
  NAME      NAMESPACE  STATUS  CONTROLPLANE  WORKERS  KUBERNETES                    ROLES  PLAN
  workload1  workload  running  1/            1/      1.20.7+vmware.1-tkg.1.7fb9067  <none>


% tanzu cluster kubeconfig get workload1 -n workload --admin
Credentials of cluster 'workload1' have been saved
You can now access the cluster by running 'kubectl config use-context workload1-admin@workload1'


% kubectl config use-context workload1-admin@workload1
Switched to context "workload1-admin@workload1".


% kubectl get nodes
NAME                                      STATUS    ROLES                  AGE    VERSION
workload1-control-plane-pz6p7              Ready    control-plane,master   4h8m   v1.20.7+vmware.1
workload1-workers-nqkvn-5c996c47f6-t4qlm   Ready    <none>                 4h5m   v1.20.7+vmware.1

8.2 Using the kubectl-vsphere command

Alternatively, you could use the original way of accessing the cluster using the kubectl-vsphere command, and selecting the namespace and cluster.

% kubectl-vsphere login --server xx.xx.62.18 --vsphere-username administrator@vsphere.local 
--insecure-skip-tls-verify --tanzu-kubernetes-cluster-namespace workload 
--tanzu-kubernetes-cluster-name workload1

Logged in successfully.

You have access to the following contexts:
  xx.xx.62.18
  workload1

If the context you wish to use is not in this list, you may need to try
logging in again later, or contact your cluster administrator.

To change context, use kubectl config use-context <workload name>


% kubectl config get-contexts
CURRENT  NAME                            CLUSTER          AUTHINFO                                      NAMESPACE
          xx.xx.62.18                    xx.xx.62.18      wcp:xx.xx.62.18:administrator@vsphere.local
          workload                       xx.xx.62.16      wcp:xx.xx.62.16:administrator@vsphere.local   workload
*         workload1                      xx.xx.62.20      wcp:xx.xx.62.20:administrator@vsphere.local


% kubectl get nodes
NAME                                      STATUS    ROLES                  AGE    VERSION
workload1-control-plane-pz6p7              Ready    control-plane,master   4h3m   v1.20.7+vmware.1
workload1-workers-nqkvn-5c996c47f6-t4qlm   Ready    <none>                 4h     v1.20.7+vmware.1

In conclusion, I think this is an interesting exercise to use the tanzu CLI to deploy workload clusters on vSphere with Tanzu. I guess tanzu CLI becomes more interesting if your are deploying Tanzu Kubernetes clusters to multiple IaaS, one of which is vSphere with Tanzu.

Note that Tanzu Services is now available in VMware Cloud. VMware Cloud allows vSphere administrators to deploy Tanzu Kubernetes clusters without having to manage the underlying SDDC infrastructure. Read more about Tanzu Services here.

The post Deploying a Tanzu Kubernetes cluster using tanzu CLI in vSphere with Tanzu appeared first on CormacHogan.com.