vSphere with Kubernetes

The first of the vSphere 7 features is vSphere with Kubernetes (formerly Project Pacific). This is a big topic and we have plenty of content planned to dive deeper into how vSphere has been transformed in order to support both VMs and containers. As Krish mentioned, Tanzu Kubernetes Grid Service is how customers can run fully compliant and conformant Kubernetes with vSphere. However, when complete conformance with the open source project isn’t required, the vSphere Pod Service can provide optimized performance and improved security through VM-like isolation. Both of these options are available through VMware Cloud Foundation 4.

The important takeaway is that Kubernetes is now built into vSphere which allows developers to continue using the same industry-standard tools and interfaces they’ve been using to create modern applications. vSphere Admins also benefit because they can help manage the Kubernetes infrastructure using the same tools and skills they have developed around vSphere. To help bridge these two worlds we’ve introduced a new vSphere construct called Namespaces, allowing vSphere Admins to create a logical set of resources, permissions, and policies that enable an application-centric approach.If Kubernetes isn’t on your radar, we still have plenty of new and improved features in this release. In fact, we’ve made large steps forward for two of our most mature technologies: DRS and vMotion. In addition to Namespaces, we have quite a few brand new features to discuss.

Improved Distributed Resource Scheduler (DRS)

vSphere DRS has been reimagined to better serve both containers and VMs. DRS used to focus on the cluster state and the algorithm would recommend a vMotion when it would benefit the balance of the cluster as a whole. This meant that DRS used to achieve cluster balance by using a cluster-wide standard deviation model.

Now, let us look at some of the key capabilities in vSphere 7.

Simplified Lifecycle Management

As our customers start providing infrastructure services with a cloud consumption model, it is critical to have a solution that allows you to fully automate and simplify the lifecycle management of the infrastructure software and hardware firmware. With vSphere 7, we are introducing the next generation of the vSphere Lifecycle Manager and Update Planner. It allows you to seamlessly manage the lifecycle of the infrastructure using a desired state paradigm.  In addition, we have added vCenter Server profiles to provided desired state configuration management for vCenter Server instances.

Intrinsic Security and Control

Security is always important, regardless of whether you are running in your datacenter or in a cloud. With vSphere 7, you now can better secure infrastructure, data, and access with a simple, comprehensive, and policy-driven model. vSphere 7 introduces remote attestation for sensitive workloads using the new vSphere Trust Authority. Moreover, it can provide secure vCenter Server authentication using external Identity Federation. vSphere 7 also supports Intel Software Guard Extensions to allow SGX extensions to user applications.

Performance and Resiliency for Application Acceleration

vSphere has always been terrific at running large, business critical workloads – perhaps better than some cloud deployments, and with vSphere 7, we continue to innovate in this area. You can now host large workloads with an improved Distributed Resource Scheduler (DRS) that takes a workload-centric approach by using the VM DRS score for hosts as the metric to decide placements and keep VMs happy, instead of only focusing on cluster-level balancing. vSphere 7 also delivers enhanced application performance on supported hardware with vSphere Persistent Memory, providing applications access to ultra-fast storage at a lower cost. Moreover, we have also updated vMotion to improve performance, and as a result now support live vMotion for your databases and mission critical workloads. With vSphere 7, you can also benefit from cost efficient AI/ML hardware pools of resources, using supported hardware such as NVIDIA GPUs. It also delivers predictable quality of service for time-critical applications with the Precision Time Protocol (PTP).

 

The following breakthrough capabilities are available for customers using containers and Kubernetes. Note that the Kubernetes capabilities of vSphere 7 are available only as part of VMware Cloud Foundation 4 with Tanzu.

Common Platform for Running both Kubernetes/Containerized Workloads and VMs

You no longer need to have multiple separate systems for your virtual machines and containers. We believe this release will have a transformative impact because we are removing a key barrier to the adoption of Kubernetes in enterprises: multiple, silo technology stacks. In the past, we have seen customers set up separate environments for Kubernetes with specialized teams and processes. By building Kubernetes into vSphere natively, customers can now consolidate their modern and traditional application environments into a single stack and immediately leverage existing technology and processes. This also addresses another key constraint of Kubernetes success: the skills gap. With minimal additional training, your vSphere administrators are now able to support Kubernetes and the modern applications run in containers.

Streamlined Development of Kubernetes Applications

vSphere 7 enables the DevOps model with infrastructure access for developers through Kubernetes APIs. It includes the Tanzu Kubernetes Grid Service, which is VMware’s compliant and conformant Kubernetes implementation for building modern containerized applications. In addition, customers who are not looking for full Kubernetes compliance for their container instances are able to use the new vSphere Pod Service, which provides improved performance and security and is built into the hypervisor with VM-like isolation.

Agile Operations for Kubernetes Applications

Finally, we are introducing a lot of value in vSphere with Kubernetes for the VI admin. We deliver a new way to manage infrastructure, called ‘application-focused management’ for containerized applications. This enables admins to apply policies to an entire group of objects and organize multiple objects into a logical group and then apply policies to the entire group. For example, an administrator can apply security policies and storage limits to a group of containers and Kubernetes clusters that represent an application, rather than to each of the objects individually. This helps improve productivity and reduce errors that can be costly to identify and correct.

 

vSphere with Kubernetes is available through VMware Cloud Foundation 4 with Tanzu. One key innovation available only in VMware Cloud Foundation is a set of developer-facing services and a Kubernetes API surface that IT can provision, called VMware Cloud Foundation Services.

VMware Cloud Foundation Services

Powered by innovations in vSphere 7 with Kubernetes, VMware Cloud Foundation Services is a new, integrated Kubernetes and REST API surface that enables you to control the infrastructure through API access to all core services.

 

But, what about individual VMs? How would that vMotion impact the VM that was moved or it’s old or new neighbors? The new DRS logic takes a very different approach that addresses these questions. It computes a VM DRS score on the hosts and moves the VM to a host that provides the highest VM DRS score. The biggest difference from the old DRS version is that it no longer balances host load. This means DRS cares less about the ESXi host utilization and prioritizes the VM “happiness”. The VM DRS score is also calculated every minute and this results in a much more granular optimization of resources.

Assignable Hardware

In vSphere 7, there is a new framework called Assignable Hardware that was developed to extend support for vSphere features when customers utilize hardware accelerators. It introduces vSphere DRS (for initial placement of a VM in a cluster) and vSphere High Availability (HA) support for VM’s equipped with a passthrough PCIe device or a NVIDIA vGPU. Related to Assignable Hardware is the new Dynamic DirectPath I/O which is a new way of configuring passthrough to expose PCIe devices directly to a VM. The hardware address of a PCIe device is no longer directly mapped to the configuration (vmx) file of a virtual machine. Instead, it is now exposed as a PCIe device capability to the VM.

Together, Dynamic DirectPath I/O, NVIDIA vGPU, and Assignable Hardware are a powerful new combination unlocking some great new functionality. For example, let’s look at a VM that requires an NVIDIA V100 GPU. Assignable Hardware will now interact with DRS when that VM is powered on (initial placement) to find an ESXi host that has such a device available, claim that device, and register the VM to that host. If there is a host failure and vSphere HA kicks in, Assignable Hardware also allows for that VM to be restarted on a suitable host with the required hardware available.

vSphere Lifecycle Manager

vSphere Lifecycle Manager accounts for a number of the new vSphere 7 features, bringing a suite of capabilities to make lifecycle operations better. With vSphere Lifecycle Manager we have a paradigm shift in both vCenter Server and ESXi host configuration management. Using a desired state configuration model, vSphere Administrators can create configurations once, apply them, and continue to monitor that desired state through new tools called vCenter Server Profiles and Image Cluster Management. vCenter Server Profiles enable administrators to standardize on a configuration for all of their vCenter Servers and monitor to protect against configuration drift.

Cluster Image Management allows administrators to create images at the cluster level that dictate how hosts within the cluster will be configured. A cluster image can comprise the vSphere (ESXi) release, a vendor add-on (which would be the delta between the gold ESXi image and the OEM ISO in VUM terminology), and a firmware add-on which would allow vSphere Lifecycle Manager to communicate with a vendor provided firmware management tool (or Hardware Support Manager) such as Dell OMIVV. Our partners at this launch are Dell EMC and HPE with more to come.

 

Third, inside vSphere Lifecycle Manager we have vCenter Server Update Planner. vCenter Server Update Planner provides native tooling to help plan, discover, and upgrade customer environments successfully. Receive notifications when an upgrade is available directly in the vSphere Client. Then use Update Planner to easily monitor the VMware product interoperability matrix to ensure that the available upgrade is compatible with other VMware software in the environment. Run a suite of available prechecks to assist with version compatibility prior to beginning an upgrade. Everything is good? You’ll have a successful upgrade, with no surprises.

It is important to note that vCenter Server Update Planner only works with vSphere 7 and onwards. So, Update Planner cannot help plan your upgrade from vSphere 6.x to vSphere 7 but it will drastically simplify your upgrades once you are running vSphere 7.

Refactored vMotion

As with DRS, we needed to review the vMotion process and look closely at how we could improve vMotion to support today’s workloads. VMs with a large memory & CPU footprint, like SAP HANA and Oracle database backends, had challenges being live-migrated using vMotion. The performance impact during the vMotion process and the potentially long stun-time during the switchover phase meant that customers were not comfortable using vMotion for these large workloads. With vSphere 7, we are bringing back that capability as we have greatly improved the vMotion logic.

At a high level, vMotion is comprised of several processes. For most VMs these processes can execute very quickly, often fast enough to not be noticed. For VMs that have large CPU and memory allocations these processes can become noticeable, and even last long enough for the application running within the VM to think there is a problem. So, several of those processes have been improved to mitigate vMotion issues for those larger VMs. One such process uses page tracers where vMotion keeps track of memory paging activity during a migration. Prior to vSphere 7, page tracing occurred on all vCPUs within a VM, which could cause the VM and its workload to be resource constrained by the migration itself. With vSphere 7, a dedicated vCPU is used for page tracing which means that the VM and its applications can keep working while the vMotion processes are occurring.

Another process that was improved was the memory copy. Prior to vSphere 7, memory was transferred between the hosts in 4k pages. vSphere 7 now uses 1 GB pages, along with a few other optimizations, to make this data transfer much more efficient. To make sure the stun time stays within the 1 second target (the time when the switch over between hosts occurs), the VM state and the bitmap of the memory pages are transferred. This stun time is important and with very large VMs, it becomes difficult to transfer that bitmap in less than the desired 1 second. So, instead of transferring the entire bitmap – which could be hundreds of megabytes in size for large VMs – only the pages required are transferred. Most of the pages are actually already on the destination host from the original transfer so we can reduce the transfer time from seconds to milliseconds.

As with all topics in this post, more details will be available – as upcoming posts here – on this new process. The key end result is that vMotion can now be used for even the largest of VMs.

Intrinsic Security

One of the biggest ways that our customers can improve their security is through good password policies, and one of the easiest ways to do that is to implement multifactor authentication (MFA). The problem, then, is that there are so many ways to implement MFA, and it’s nearly impossible to extend vCenter Server with all of them. Furthermore, even if VMware implements some of them, we’re duplicating what many customers already have in their corporate identity management systems, and that doesn’t mesh with our desire to make life better for our users, the vSphere Admins.

 

The solution is federation using open authentication & authorization standards like OAUTH2 and OIDC. With vSphere 7 and Identity Federation, vCenter Server can talk to an enterprise identity provider and get the vSphere Admins and vCenter Server out of the process. This simplifies the vSphere Admin’s job and reduces helps reduce compliance audit scope. It also opens the door to lots of different MFA methods because they already know how to plug into things like Active Directory Federation Services (ADFS). With vSphere 7 we are supporting ADFS out of the box and will build support for more providers over time.

We’re also introducing vSphere Trust Authority (vTA), helping to make it easier to establish trust throughout the entire stack – from bare metal all the way through the workloads. vSphere Trust Authority creates a hardware root of trust using a small, separately-managed cluster of ESXi hosts which takes over the task of attestation. Host attestation is where the UEFI Secure Boot process, a server’s Trusted Platform Module (TPM), and an external service work together using cryptographic to verify that the host is running authentic software, in a good configuration.

In vSphere 7, vTA gives attestation the ability to enforce the rules by having the trusted hosts take over the communications with the key management systems (KMSes). This simplifies the connections to the KMSes, which simplifies risk auditing, as well as ensuring that a host that fails attestation doesn’t get access to secrets. Without those secrets the host can’t run an encrypted VM, which is good. We don’t want a secured VM on an untrusted server.

Certificate management also continues to be improved by reducing the amount of certificates that are required to be managed as well as the introduction of a new certificate import wizard. Solution User certificates no longer need to be managed and ESXi has also been simplified so that its services use a common certificate. Last, there is a REST API for operations such as renewing a certificate from the VMware Certificate Authority (VMCA), making the process easier to automate.

 

Other Improvements

This blog post is not meant to be exhaustive but there are a few other vSphere 7 features that I’d like to mention. First, we’ve continued to simplify the vCenter Server architecture. With vSphere 7, there is no longer the ability to deploy external Platform Services Controllers (PSCs) or vCenter Server for Windows. If you have either of these types of deployments, the vCenter Server 7 installer will automatically migrate that vCenter Server instance to a vCenter Server appliance with an embedded PSC. There is no multi-step process that involves multiple tools. It is an integrated, seamless experience.

Support has also been added for multiple NICs for the vCenter Server appliance, new CLI Tools, and an improved Developer Center in the vSphere Client. There is a new VM Hardware version, 17, that brings more new features like a precision clock for PTP support, vSGX, and a virtual watchdog to help monitor clustered applications. Over the course of the next few weeks we’ll be releasing detailed blogs on all these vSphere 7 features and more. Please stay up to date through the links and information posted in the footer below.

Conclusion

As you may have gathered by now, vSphere 7 really is a substantial and game-changing release. There has been a big focus on making our customers’ lives better through the lifecycle and security improvements. We also continue to keep pushing the boundaries of what is possible thanks to our great partnerships and customers. And, with the addition of Kubernetes, we’re not slowing down any time soon. vSphere 7 is technology for the hybrid cloud.