VxVerify – Pre-upgrade Health Check Tool

VxVerify is an incredibly useful tool to have at your disposal while working with VxRail. Complimentary to the native VxRail manager checks VxVerify performs a health check analysis on a VxRail cluster and it is highly recommended to run VxVerify in advance of upgrades, expansions or for general maintenance operations. VxVerify when run on VxRail Manager deploys what is referred to as ‘Minions’ (small python programs) which are sent to each of the VxRail nodes in the cluster, these minions in turn perform host checks on each of the nodes. In addition to ESXi host specific tests VxVerify also performs checks on VxRail Manager, VMs, vCSA and at a cluster level. Here are just a sample of what gets tested:

  • Maintenance mode status
  • Check hostd service
  • IPMI tool for VxRail hardware
  • NTP status and time delta
  • Secure Boot status
  • vSAN Health
  • Service Datastore available capacity
  • ESXi version consistency
  • vSAN Health
  • Free space
  • Password for unsupported characters
  • Check free space in root and /tmp
  • Host Lockdown Mode
  • Reboot required for ESXi host
  • Check for RecoverPoint
  • Check for external vCenter
  • vMotion compatibility
  • Mounted ISO on VM
  • VM-to-host affinity rules
  • Docker Servers Running
  • VCF installation type, if present
  • Check time difference between VxRM & VC

It is highly recommended that you run the VxVerify tool before proceeding with a VxRail upgrade; more details on how to download and run VxVerify can be found here:

https://support.emc.com/kb/536801

There is a VxVerify download available for each of the following VxRail versions:

  • VxVerify 1.xx.xxx is for VxRail 4.0 only
  • VxVerify 2.xx.xxx is for VxRail 4.5, 4.7 & 7.0.000
  • VxVerify 3.xx.xxx is for VxRail 7.0.010+

Note: Always download the very latest edition as new checks are being added regularly. In addition each version of VxVerify is timelimited to prevent outdated versions.

Running VxVerify

In this example VxRail is running code version 7.0.100 so we begin by downloading the VxVerify 3.x edition:

Once download completes extract the .zip and review the readme.txt to review all the recent updates:

Before uploading VxVerify to VxRail Manager we need to create a directory with the required permission set. Login to VxRail manager via SSH elevating to root user and execute the following set of commands to create a ‘vxv’ dir in /tmp and set the permissions:

#mkdir /tmp/vxv
#chmod 777 /tmp/vxv
#cd /tmp/vxv

Upload the ‘.pyc’ file to the newly created directory (in this example I am leveraging ‘winscp’ to perform the upload to VxRail manager):

Once uploaded navigate to /tmp/vxv/ and use --help to expose all the VxVerify associated command line arguments:

Simply running the .pyc script as root will kick off VxVerify:

As can be seen from the results table below; all hosts and cluster tests completed successfully. You can also see the status 3 critical alert notifying that ‘VC login failed’ the VC management credentials get decrypted automatically from the VxRail Manager database and runtime (although they can also be specified with -u & -p), so if those are failing, you’ll need to check the vxv.log

Inserting the vCenter root credentials performs the respective vCSA tests such as free space tests for the VCSA partitions:

Results table below now inclusive of vCenter health checks:

VxVerify creates a number of files in ‘/tmp/vxv/’ these include .log files which can be used for troubleshooting purposes etc. also the summary report seen on the previous screen can be recalled by viewing the  ‘/tmp/vxv/vxverify.txt’ file:

Another really useful output here is the ‘vxtii.txt’ file which includes detailed information on each ESXi host in the VxRail cluster providing a nice overview of the hardware:

Creating a dummy failure by attaching an .iso to the vCenter VM using local storage results in the following failure scenario. You will note for each warning or failure there is an associated support.dell.com KB referenced in the results table which will assist with resolving any issue highlighted by VxVerify (do not attempt an upgrade until these yellow or red event codes are investigated):

Special mention to our Escalation Engineering team for maintaining such a valuable tool! Thank you!

20 Comments »

  1. Hey David. That’s a really good article and thanks for that.
    I would add though that the login error, listed under VxRM would be for the VC management credentials, whereas the VCSA root credentials are used for the tests lsited under VCSA (such as free space tests for the VCSA partitions).
    The VC management credentials should decrypted automatically from the VxRM DB and runtime (although they can also be specified with -u & -p), so if those are failing, you’ll need to check the vxv.log.

  2. Hi David, nice post.
    do the output files in /tmp/vxv have consist names? purely to grab those files for analysis at a later date ? looks a great tool, and hope to run soon to see the results.

    • Hi Kevin, thanks for the feedback. Great Q. I’ll update post to address this; file names are consistent and you can chose to save the VxVerify output if you leverage the -l or –log option along with a path that the logs should be saved to, for example:
      python vxverify.pyc -l /tmp/vxv1logs

      Does that help?

  3. Thanks for the excellent post David. I am unable to run VxVerify when i do not have enabled ssh on the nodes manually before. I am just getting a timeout when the script tries to push the minions to the nodes, Results for VxRail-Manager and vCenter are green. VxVerify is 3.10.212 and VxRail is 7.0.131

  4. Wonderful post! I am thankful for the effort you made to put all in your writing, can you help me resolve issue i get when doing precheck, I am getting this warning “VxRail: VxRM health-check fails for test ‘vc_pw_char’ “

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s