Container Services Extension – Kubernetes as a Service for vCloud Director. By Nicolas Bardier, Cloud admin at Pyxis

Container Services Extension (CSE), is a vCloud Director extension developed by VMware to deploy Kubernetes clusters by vCloud Director (Kubernetes as a Service) organization administrators.

Summary

– Introduction
– Use cases
– Roles and Functions in CSE
– Deployment and Usage Notes
– To conclude
– Acknowledgements

Introducción

With the emergence of container-based services and DevOps, a multitude of solutions for automatic infrastructure deployment have appeared. CSE is one of them.

CSE leverages existing vCloud Director infrastructure to rapidly generate Kubernetes clusters, to make them available to teams consuming that service, such as QA and development.

Use Case

Based on our testing, the most suitable use case for CSE is deploying ‘throwaway’ Kubernetes clusters, for development or QA teams, by IaaS administrators with no experience deploying Kubernetes.

Another very similar use case is the deployment of Kubernetes clusters for container-based application demos.

Roles and functions in CSE

For the purposes of CSE, there are four defined roles:

-Cloud administrator.
Configures, deploys and manages the CSE service and its connection to vCloud Director and vCenter.
Is the administrator of the cloud platform based on vCloud Director.

-Organization administrator
Manages the vCloud Director organization (also called tenant).
Creates, reconfigures and destroys Kubernete clusters defined in vCloud Director.

-K8s Administrator
Manages the K8s cluster, with kubectl, via SSH directly to the cluster nodes.

-K8s user
Manages pods and services of a given Kubernetes cluster, for example using kubectl.

Note that the official CSE documentation does not distinguish between K8s administrator and user.

Notes on deployment and usage

There is a good example how to deploy a Kubernetes cluster and a group of pods with CSE on the CSE website, there are a few points to consider when deploying Kubernetes with CSE

When creating the cluster:

-Define the SSH key to use.
Once the cluster is deployed, it does not allow to change the SSH key from vcd-cli.

-Define an NFS node.
From vcd-cli you cannot add an NFS node to an existing cluster.

-There is no explicit control of the IPs to use in each VM of the vAPP.
The cluster creation process takes available IP addresses from the local network pool.

This may involve redoing existing DNATs and firewall rules in case of rebuilding the cluster.

-Thevcd cse cluster create command may take a few minutes.

It delays depending on the speed of the Internet link and the performance of the infrastructure on which it runs.

For example, in a lab with nested virtualization on local disks and a 10Mbit link, deploying a cluster of 3 VMs (master, node and nfs), takes about 30 minutes.

The K8s and OS versions are conditioned to the available templates. This condition is transferred in a transitive way to the final customer. These templates reside officially in a remote repository, maintained and managed by the CSE project collaborators.

-Consider using CSE version 2.5.
According to the official documentation, CSE 2.5 version offers several improvements, such as being able to use custom templates.

-Consider vcd-cli and container-service-extension version compatibility.
It may cause problems not to use a version identical to the server-side components.

-The possible architecture of k8s is always 1 master + n workers.
This implies that it is not advisable to use this solution for productive loads requiring high availability.

Operating the cluster:

-Adjust the configuration file settings for kubectl accordingly.
Take into account the ‘server’, ‘certificate-authority-data’ and ‘insecure-skip-tls-verify’ parameters, depending on whether you are using a test environment, and/or accessing kubectl from outside the vCloud Director organization.

For example, for cluster kaas01 you can generate a .conf file for kubectl like this:

$ vcd cse cluster config kaas01 > ~/kaas01.conf

-Crear, modificar y destruir clusters CSE sólo es posible a través de la API o de vcd-cli.
No hay integración con las interfaces web (HTML5 ni Flash) a la versión 9.7 de vCloud Director.

Para escalar el cluster, ejecutar

$ vcd cse cluster resize <cluster-name> -N <count>

-El storage persistente que habilita la solución es solo NFS.
No fué posible habilitar storageOS.

To conclude

-CSE allows you to deploy Kubernetes clusters quickly.
-A Kubernetes user who wants to focus on testing their application can quickly have a cluster to work on.
-An organization administrator can create and destroy a Kubernetes cluster with relative ease through the API or vcd-cli.

I further quote the conclusions of the DevOps team:

“The ease of deployment stands out, which makes it extremely simple to set up a cluster, without flaunting previous knowledge. On the other hand, zero-day administration is very limited, since the solution does not have basic components, such as ingress controller or dashboard. This requires a more advanced K8s user to make it available.

This makes the delivery of the solution as a product, requiring technical knowledge of k8s for its consumption.

technical knowledge of K8s for its consumption.

On the other hand, the inability to have Multi-Master deployments would limit the use to development, demo or QA environments only.

In general terms, I find the solution with several limitations from the K8s administrator’s point of view, limitations that seem to be typical of the maturity of the project. The initial GA release was in mid-2018.”

Thanks

To the Research and DevOps teams at Pyxis for coordinating and executing the testing of the solution, and giving me feedback on it.

As always, to the Communication team for helping us make the blog post look as good as possible 🙂

Links of interest

• Web de CSE

Kubernetes as a Service on vCloud Director por Steve Dockar