VMware Cloud Foundation: Manual Product Upgrades

Why You Shouldn’t Upgrade VMware NSX Manually

The Issue?

It is important to understand the impacts of manually upgrading products in a VMware Cloud Foundation environment. In this article, VMware NSX was previously upgraded to a ‘non-VCF’ version, and caused lifecycle issues for the entire domain.

Unless you are upgrading in a supported manner, i.e. the async patch tool, then the official guidance is simple, do not upgrade any products outside of SDDC Manager. The screenshots below are what you will see if the versions do not match what is in the BOM and database on SDDC Manager. This is a NSX Federated environment, however, the error message will be the same for VMware NSX local managers.

Diagnosing The Issue

The release version in SDDC Manager shows each VCF release and the expected BOM for said release, in this case it is a 4.5 environment, and the relevant boxes have been highlighted. As there were some out of band updates, it seems to have caused an issue with SDDC Manager being able to recognize the domain release version, hence, a 0 instead of a 1.

Below you can see VMware NSX was upgraded to version 4.1, and the expected version was 3.2.1.2.

To further investigate, ssh onto SDDC Manager and su to root. Run the following line.

curl localhost/inventory/nsxt | json_pp

The highlighted text shows VMware NSX’s current version and instance ID in the database.

Fixing The Version Mismatch

Log back onto SDDC manager and su to root if you are not already logged on and run the following line of text.

curl localhost/inventory/entities/cb1de489-de2e-4e8e-8972-d87c22ebd250 -X PATCH -d '{"type":"NSXT_CLUSTER","status":"ACTIVE","version":"4.1.0.0.0.21332672"}' -H 'Content-Type:application/json'

Ensure you change the ID after localhost/inventory/entities/<vmwareNSXID> to your ID, and the version to match what VMware NSX was manually upgraded to, which you saw in a previous screenshot.

You can see the version in the output above now reflects VMware NSX’s current version.

SDDC Manager upgrade prechecks are now successful.

Summary

It is important to understand the issues that arise from manual product upgrades, unless performed in a supported way or VMware support guidance. Whilst this article focussed on VMware NSX, it will likely be possible to perform similar operations for other solutions in the VMware Cloud Foundation (VCF) stack.

My VCF Troubleshooting guide has a few other tips for administrators.

Check out these links for all my other VCF articles and my recently published VCF course