There are a few questions that come up quite often regarding vCenter Server upgrades and mixed-versions that we would like to address. In this blog post we will discuss and attempt to clarify the guidance in the vSphere Documentation for Upgrade or Migration Order and Mixed-Version Transitional Behavior for Multiple vCenter Server Instance Deployments. This doc breaks down what happens during the vCenter Server upgrade process and describes the impacts of having components – vCenter Server and Platform Services Controller (PSC), running at different versions during the upgrade window. For example, once you get some vCenter Server instances upgraded, say to 6.5 Update 1, you won’t be able to manage those upgraded instances from any 5.5 instances. While most of the functionality limitations manifest themselves when upgrading from 5.5 to 6.x, there could also be some quirks in environments running a mix of 6.0 and 6.5. There are a couple of additional questions that seem to arise from this doc so let’s see if we can address them.
The Upgrade Process
I’m not going to go through the entire process here, but it is important to understand the basics of how a vCenter Server upgrade works. Remember that there are two components to vCenter Server – the Platform Services Controller (PSC) which runs the vSphere (SSO) Domain and vCenter Server itself. For a vCenter Server upgrade, the vSphere Domain and all PSCs within it, must be upgraded first. Once that is complete, then the vCenter Servers can be upgraded. Obviously, if you have a standalone vCenter Server with an embedded PSC, this is a much simpler proposition. But, for those requiring external PSCs because of other requirements such as Enhanced Linked Mode, just remember the PSCs need to be upgraded first.
The other important point to make here is that upgrading by site is not supported. Looking at the above example, you can see there are two sites each with an external PSC and a vCenter Server. It is a common that a customer would like to upgrade an entire site, test, and then move onto the next site. Unfortunately, this is not supported and all PSCs within the vSphere Domain across all sites must be upgraded first.
Now, on to the questions mentioned earlier. The first question is, “Can I run vCenter Servers and Platform Services Controllers (PSCs) of different versions in my vSphere Domain?” The answer here is yes, but only during the time of an upgrade. VMware does not support running different versions of these components under normal operations within a vSphere Domain. The exact verbiage from the article is, “Mixed-version environments are not supported for production. Use these environments only during the period when an environment is in transition between vCenter Server versions.” So, do not plan on running different versions of vCenter Server and PSC in production on an ongoing basis.
The second question is then, “How long can I run in this mixed-version mode?” This question is a bit tougher to answer. There is no magic date or time bomb when things will just stop working. This is really more of a question of understanding the risks and knowing how problems may affect the environment should something go wrong while in this mixed-version state.
An example of one such risk would be if you were upgrading to vSphere 6.5 from 5.5. Let’s say you had your vSphere Domain (i.e. PSCs) and one vCenter Server already upgraded leaving you with 1 or more vCenter Server 5.5 instances. Imagine that something happens leaving a vCenter Server 5.5 completely wiped out. You could restore that vCenter Server 5.5 instance and be back in production as long as you have a good, current backup. If the backup you need to restore from was taken prior to the start of the vSphere Domain upgrade, you would not be able to use it to restore. The reason for this is that the vCenter Server instance that you would be restoring is expecting a 5.5 vSphere Domain and the communication between that restored vCenter Server instance and the 6.5 PSC would not work. An alternative to this would be to rollback the entire vSphere Domain and any other vCenter Servers that were upgraded.
Another risk would be if we are unable to restore that instance because the backups were bad (it does happen) or you couldn’t accept the outcome of losing the data since that backup was taken. The result here is that you would be forced to rebuild that vCenter Server instance and re-attach all the hosts. This may not be desirable because this new vCenter Server instance would have a new UUID and all of the hosts, VMs, and other objects would also have new moref IDs. This means that any backup tools or monitoring software would see these as all net new objects and you would lose continuity of backups or monitoring. You also would have to rebuild the vCenter Server instance as 6.5 which also may not be desirable because you may have an application or other constraint that requires a specific version of vCenter Server. If you rebuild the instance as 6.5 you may break that application.
Finally, let’s consider the possibility of having a PSC failure instead of losing a vCenter Server. What happens? Normally, you could easily repoint a vCenter Server instance to another external PSC within the same SSO Site. However, this would not be possible if the vCenter Server is not running the same version as the PSC you are attempting to repoint to. For example, if you had a vCenter Server 5.5 or 6.0 and they were pointing to a 6.5 PSC (because it has already been upgraded), if that PSC failed you would not be able to repoint that vCenter Server to another PSC. Remember that all PSCs must be upgraded first so all PSCs should be running 6.5 already. The only way to recover from this scenario is to restore or redeploy the failed PSC which may take longer than repointing.
So, give the above scenarios, what do we tell a customer who asks, “My upgrade plan spans multiple sites over multiple months. How should I plan my upgrade?” Here are our recommendations:
- 1. Minimize the upgrade window
2. Follow the upgrade documentation
3. Take full backups before, during, and after the upgrade
4. Check the interop matrices and test the upgrade first
The first recommendation is to minimize the upgrade window as much as possible. We understand that there’s only so much you can do here, but it is important to reduce the amount of time you’ll be running different versions of vCenter Server (and PSC) in the same vSphere Domain. The second recommendation is to, no matter how tempting to do otherwise, upgrade the entire vSphere Domain (SSO Instances and PSCs) first as is called out in the vSphere Documentation. It is not supported to upgrade everything in one site and then move onto the next. You must upgrade all SSO Instances and PSCs in the vSphere Domain, across ALL sites and locations, first. Third, make sure you have good backups every step of the way. While snapshots can be a path to a quick rollback, when dealing with SSO, PSCs, and vCenter Server they don’t always work. Taking a full backup ensures the ability to restore to a known clean state. Last, and certainly not least, do your interoperability testing and test the upgrade in a lab environment that represents your production environment as much as possible.
Emad has a great 3-part series on upgrades (Part 1, Part 2, Part 3) so be sure to check it out prior to testing and beginning your upgrade. Also know and understand the risks and impacts of problems during the upgrade process. Finally, know how the upgrade process is going to affect all of the yet-to-be-upgraded parts of your environment and have good rollback and mitigation plans if any issues come up.
About the Author
Adam Eckerle manages the vSphere Technical Marketing team in the Cloud Platform Business Unit at VMware. This team is responsible for vSphere launch, enablement, and ongoing content generation for the VMware field, Partners, and Customers. In addition, Adam’s team is also focused on preparing Customers and Partners for vSphere upgrades through workshops, VMUGs, and other events.
vCenter Server migrations have typically taken massive planning, a lot of effort, and time. The new Migration Tool included in the vCenter Server Appliance (VCSA) 6.5 is a game changer. No longer requiring scripts and many long nights of moving hosts one cluster at a time. The Migration Tool does all the heavy lifting. Copying the configuration and inventory of source vCenter Server by default. The migration workflow includes upgrading from either a Windows vCenter Server 5.5 or 6.0 to VCSA 6.5. A new guided migration walkthrough is available on the VMware Feature Walkthroughl site. This click-by-click guide covers an embedded migration from a Windows vCenter 6.0 to a VCSA 6.5.
The first step of the migration workflow requires running the Migration Assistant (MA). The Migration Assistant serves two purposes. The first is running pre-checks on the source Windows vCenter Server. The Migration Assistant displays warnings of installed extensions and provides a resolution for each. It will also show the source and the destination deployment types. Keep in mind changing a deployment type is not allowed during the migration workflow. More information on deployment type considerations prior to a migration can be found here. The MA also displays some information about the source Windows vCenter Server. These included: FQDN, SSO User, SSL Thumbprint, Port, and MA log folder. At the bottom of the MA is the Migration Steps, which will be available until the source Windows vCenter Server is shutdown. This is a helpful guide of the migration steps that need to be completed. The second purpose of the MA is copying the source Windows vCenter Server data. By default, the configuration and inventory data of the Windows vCenter Server is migrated. The option to copy historical and performance data is also available. During the migration workflow, no changes are made to the source Windows vCenter Server. This allows for an easy rollback plan. Do not close the Migration Assistant at any point during the migration workflow. Closing the MA will result in starting the entire migration process over. If everything is successful there will be a prompt at the bottom of the Migration Assistant to start the migration.
Step two of the migration workflow is starting the wizard driven Migration Tool. This requires the vCenter Server Appliance 6.5 Installer. Since the identity of the source Windows vCenter Server is preserved, the migration tool needs to run on a separate Windows Server from the source. Like the VCSA 6.5 Deployment, Migration is also a two stage deployment. The Migration Tool will first deploy a new vCenter Server Appliance. The new VCSA will have a temporary IP address while the source Windows vCenter data is copied. The second stage configures the VCSA 6.5 and imports the source Windows vCenter Server data. This includes the identity of the source Windows vCenter server. The vCenter Server identity includes FQDN, IP address, UUID, Certificates, MoRef IDs, etc. As far as other solutions that communicate with vCenter Server nothing has changed. During the migration workflow, no changes are made to the source Windows vCenter Server. This allows for an easy rollback plan. Other solutions may require an upgrade, consult the VMware and any third party interoperability matrixes. Once the migration workflow is completed, login to the vSphere Client and validate your environment.
vCenter Server 6.0 Embedded Migration to Appliance Walkthrough is available here. This guide will show how to migrate a Windows vCenter Server and the Platform Services Controller 6.0 components on a single virtual machine to a vCenter Server Appliance 6.5. Another feature walkthrough for external migration including vSphere Update Manager (VUM) will be available soon. In the mean time go through the embedded migration and provide any feedback in the comments section below. Also feel free to reach out to me on Twitter @emad_younis.
About the Author
Emad Younis is a Staff Technical Marketing Architect and VCIX 6.5-DCV working in the Cloud Platform Business Unit, part of the R&D organization at VMware. He currently focuses on the vCenter Server Appliance, vCenter Server Migrations, and VMware Cloud on AWS. His responsibilities include generating content, evangelism, collecting product feedback, and presenting at events. Emad can be found blogging on emadyounis.com or on Twitter via @emad_younis.
Posted on October 18, 2016 by Ramesh B S
This video demonstrates how to backup and restore an embedded vCenter Server 6.0 vPostgres database. Backing up your database protects the data stored in your database. Of course, restoring a backup is an essential part of that function.
This follows up on our recent blog & video: How to backup and restore the embedded vCenter Server Appliance 6.0 vPostgres database.
Note: This video is only supported for backup and restore of the vPostgres database to the same vCenter Server. Use of image-based backup and restore is the only solution supported for performing a full, secondary appliance restore.
VMware vCenter Server™ 6.0 substantially improves performance over previous vCenter Server versions. This paper demonstrates the improved performance in vCenter Server 6.0 compared to vCenter Server 5.5, and shows that vCenter Server with the embedded vPostgres database now performs as well as vCenter Server with an external database, even at vCenter Server’s scale limits. This paper also discusses factors that affect vCenter Server performance and provides best practices for vCenter Server performance.
What’s New in vCenter Server 6.0
vCenter Server 6.0 brings extensive improvements in performance and scalability over vCenter Server 5.5:
- Operational throughput is over 100% higher, and certain operations are over 80% faster.
- VMware vCenter Server™ Appliance™ now has the same scale limits as vCenter Server on Windows with an external database: 1,000 ESXi hosts, 10,000 powered-on virtual machines, and 15,000 registered virtual machines.
- VMware vSphere® Web Client performance has improved, with certain pages over 90% faster.
In addition, vCenter Server 6.0 provides new deployment options:
- Both vCenter Server on Windows and VMware vCenter Server Appliance provide an embedded vPostgres database as an alternative to an external database. (vPostgres replaces the SQL Server Express option that was available in previous vCenter versions.)
- The embedded vPostgres database supports vCenter’s full scale limits when used with the vCenter Server Appliance.
Performance Comparison with vCenter Server 5.5
In order to demonstrate and quantify performance improvements in vCenter Server 6.0, this section compares 6.0 and 5.5 performance at several inventory and workload sizes. In addition, this section compares vCenter Server 6.0 on Windows to the vCenter Server Appliance at different inventory sizes, to highlight the larger scale limits in the Appliance in vCenter 6.0. Finally, this section illustrates the performance gained by provisioning vCenter with additional resources.
The workload for this comparison uses vSphere Web Services API clients to simulate a self-service cloud environment with a large amount of virtual machine “churn” (that is, frequently creating, deleting, and reconfiguring virtual machines). Each client repeatedly issues a series of inventory management and provisioning operations to vCenter Server. Table 1 lists the operations performed in this workload. The operations listed here were chosen from a sampling of representative customer data. Also, the inventories in this experiment used vCenter features including DRS, High Availability, and vSphere Distributed Switch. (See Appendix A for precise details on inventory configuration.)
Figure 3 shows vCenter Server operation throughput (in operations per minute) for the heaviest workload for each inventory size. Performance has improved considerably at all sizes. For example, for the large inventory setup (Figure 3, right), operational throughput has increased from just over 600 operations per minute in vCenter Server 5.5 to over 1,200 operations per minute in vCenter Server 6.0 for Windows: an improvement of over 100%.
The other inventory sizes show similar gains in operational throughput.
Figure 4 shows median latency across all operations in the heaviest workload for each inventory size. Just as with operational throughput in Figure 3, latency has improved at all inventory sizes. For example, for the large inventory setup (Figure 4, right), median operational latency has decreased from 19.4 seconds in vCenter Server 5.5 to 4.0 seconds in vCenter Server Appliance 6.0: a decrease of about 80%. The other inventory sizes also show large decreases in operational latency.
Download a full VMware vCenter Server 6.0 Performance and Best Practices Technical White Paper
The VMware vCenter Server™ 6.0 release introduces new, simplified deployment models. The components that make up a vCenter Server installation have been grouped into two types: embedded and external. Embedded refers to a deployment in which all components—this can but does not necessarily include the database—are installed on the same virtual machine. External refers to a deployment in which vCenter Server is installed on one virtual machine and the Platform Services Controller (PSC) is installed on another. The Platform Services Controller is new to vCenter Server 6.0 and comprises VMware vCenter™ Single Sign-On™, licensing, and the VMware Certificate Authority (VMCA).
Embedded installations are recommended for standalone environments in which there is only one vCenter Server system and replication to another Platform Services Controller is not required. If there is a need to replicate with other Platform Services Controllers or there is more than one vCenter Single Sign-On enabled solution, deploying the Platform Services Controller(s) on separate virtual machine(s)—via external deployment—from vCenter Server is required.
This paper defines the services installed as part of each deployment model, recommended deployment models (reference architectures), installation and upgrade instructions for each reference architecture, postdeployment steps, and certificate management in VMware vSphere 6.0.
VMware vCenter Server 6.0 Services
A few requirements are common to both installing vCenter Server on Microsoft Windows and deploying VMware vCenter Server Appliance™. Ensure that all of these prerequisites are in place before proceeding with a new installation or an upgrade.
- DNS – Ensure that resolution is working for all system names via fully qualified domain name (FQDN), short name (host name), and IP address (reverse lookup).
- Time – Ensure that time is synchronized across the environment.
- Passwords – vCenter Single Sign-On passwords must contain only ASCII characters; non-ASCII and extended (or high) ASCII characters are not supported.
Installing vCenter Server 6.0 on a Windows Server requires a Windows 2008 SP2 or higher 64-bit operating system (OS). Two options are presented: Use the local system account or use a Windows domain account. With a Windows domain account, ensure that it is a member of the local computer’s administrator group and that it has been delegated the “Log on as a service” right and the “Act as part of the operating system” right. This option is not available when installing an external Platform Services Controller.
Windows installations can use either a supported external database or a local PostgreSQL database that is installed with vCenter Server and is limited to 20 hosts and 200 virtual machines. Supported external databases include Microsoft SQL Server 2008 R2, SQL Server 2012, SQL Server 2014, Oracle Database 11g, and Oracle Database 12c. When upgrading to vCenter Server 6.0, if SQL Server Express was used in the previous installation, it will be replaced with PostgreSQL. External databases require a 64-bit DSN. DSN aliases are not supported.
When upgrading vCenter Server to vCenter Server 6.0, only versions 5.0 and later are supported. If the vCenter Server system being upgraded is not version 5.0 or later, such an upgrade is required first.
Table 2 outlines minimum hardware requirements per deployment environment type and size when using an external database. If VMware vSphere Update Manager™ is installed on the same server, add 125GB of disk space and 4GB of RAM.
Download a full VMware vCenter Server™ 6.0 Deployment Guide