DavidRing.ie

VxVerify – Pre-upgrade Health Check Tool

VxVerify is an incredibly useful tool to have at your disposal while working with VxRail. Complimentary to the native VxRail manager checks VxVerify performs a health check analysis on a VxRail cluster and it is highly recommended to run VxVerify in advance of upgrades, expansions or for general maintenance operations. VxVerify when run on VxRail Manager deploys what is referred to as ‘Minions’ (small python programs) which are sent to each of the VxRail nodes in the cluster, these minions in turn perform host checks on each of the nodes. In addition to ESXi host specific tests VxVerify also performs checks on VxRail Manager, VMs, vCSA and at a cluster level. Here are just a sample of what gets tested:

• Maintenance mode status
• Check hostd service
• IPMI tool for VxRail hardware
• NTP status and time delta
• Secure Boot status
• vSAN Health
• Service Datastore available capacity
• ESXi version consistency
• vSAN Health
• Free space
• Password for unsupported characters
• Check free space in root and /tmp
• Host Lockdown Mode
• Reboot required for ESXi host
• Check for RecoverPoint
• Check for external vCenter
• vMotion compatibility
• Mounted ISO on VM
• VM-to-host affinity rules
• Docker Servers Running
• VCF installation type, if present
• Check time difference between VxRM & VC

It is highly recommended that you run the VxVerify tool before proceeding with a VxRail upgrade; more details on how to download and run VxVerify can be found here:

https://support.emc.com/kb/536801

There is a VxVerify download available for each of the following VxRail versions:

• VxVerify 1.xx.xxx is for VxRail 4.0 only
• VxVerify 2.xx.xxx is for VxRail 4.5, 4.7 & 7.0.000
• VxVerify 3.xx.xxx is for VxRail 7.0.010+

Note: Always download the very latest edition as new checks are being added regularly. In addition each version of VxVerify is timelimited to prevent outdated versions.

Running VxVerify

In this example VxRail is running code version 7.0.100 so we begin by downloading the VxVerify 3.x edition:

Once download completes extract the .zip and review the readme.txt to review all the recent updates:

Before uploading VxVerify to VxRail Manager we need to create a directory with the required permission set. Login to VxRail manager via SSH elevating to root user and execute the following set of commands to create a ‘vxv’ dir in /tmp and set the permissions:

#mkdir /tmp/vxv
#chmod 777 /tmp/vxv
#cd /tmp/vxv

Upload the ‘.pyc’ file to the newly created directory (in this example I am leveraging ‘winscp’ to perform the upload to VxRail manager):

Once uploaded navigate to /tmp/vxv/ and use --help to expose all the VxVerify associated command line arguments:

Simply running the .pyc script as root will kick off VxVerify:

As can be seen from the results table below; all hosts and cluster tests completed successfully. You can also see the status 3 critical alert notifying that ‘VC login failed’ the VC management credentials get decrypted automatically from the VxRail Manager database and runtime (although they can also be specified with -u & -p), so if those are failing, you’ll need to check the vxv.log

Inserting the vCenter root credentials performs the respective vCSA tests such as free space tests for the VCSA partitions:

Results table below now inclusive of vCenter health checks:

VxVerify creates a number of files in ‘/tmp/vxv/’ these include .log files which can be used for troubleshooting purposes etc. also the summary report seen on the previous screen can be recalled by viewing the  ‘/tmp/vxv/vxverify.txt’ file:

Another really useful output here is the ‘vxtii.txt’ file which includes detailed information on each ESXi host in the VxRail cluster providing a nice overview of the hardware:

Creating a dummy failure by attaching an .iso to the vCenter VM using local storage results in the following failure scenario. You will note for each warning or failure there is an associated support.dell.com KB referenced in the results table which will assist with resolving any issue highlighted by VxVerify (do not attempt an upgrade until these yellow or red event codes are investigated):

Special mention to our Escalation Engineering team for maintaining such a valuable tool! Thank you!