EMC NAS Plug-In For vSphere VAAI (VNXe Example)

The ‘EMC NAS Plug-in’ is required in order to enable VAAI (vSphere APIs for Array Integration) operations on ‘NFS Datastores’ on an ESXi 5.x host. If you are not familiar with VAAI; the purpose of enabling the VAAI API is to offload certain storage related I/O tasks to the storage array. As a result this will reduce the I/O requirement on the ESXi hosts and their associated networks. Instead of the ESXi host using resources to send I/O across the network for such tasks as Storage vMotion or cloning a VM, the hypervisor will now just send the NFS related commands that are required for the storage array to perform the necessary data movement. For block based storage arrays the VAAI primitives are available as default on the ESXi host and no plug-in is required.

Installation Of The NAS Plug-In On ESXi 5.x
1. Upload the .zip install package (EMCNasPlugin-1.0-11.zip) to the ESXi datastore.
2. Open an SSH Session to the ESXi host and change directory to the location of the install package:
# cd /vmfs/volumes/
If you need to list the name of your datastore:
/vmfs/volumes # ls -l
/vmfs/volumes # cd /vmfs/volumes/DatastoreName/
ls again to confirm the .zip package is present.
3. Ensure the NAS Plug-In is VMwareAccepted:
/vmfs/volumes/DatastoreName # esxcli software sources vib list -d file:///vmfs/volumes/DatastoreName/EMCNasPlugin-1.0-11.zip
Acceptance Level: VMwareAccepted
4. Run the installation:
/vmfs/volumes/DatastoreName # esxcli software vib install -n EMCNasPlugin -d file:///vmfs/volumes/DatastoreName/EMCNasPlugin-1.0-11.zip
Installation Result: completed successfully
Reboot Required: true
VIBs Installed: EMC_bootbank_EMCNasPlugin_1.0-11

5. Reboot the ESXi host and confirm the EMCEMCNasPlugin vib is loaded:
~ # esxcli software vib list | grep EMCNasPlugin

VAAI Example: ‘Full File Clone’ Primitive Operation With VNXe
‘Full File Clone’ is one of the VAAI NAS primitives which is used to copy or migrate data within the same physical array (Block equivalent is known as XCOPY). In this example we are using a ‘VNXe 3150’ with two NFS Datastores presented to one ESXi 5.5 host with the NAS Plug-In installed (VAAI enabled) and another ESXi 5.5 host without the NAS Plug-In installed (VAAI disabled).

NAS_VAAI0

Running a Storage vMotion from the NFS01 datastore to NFS02 on the ESXi host with VAAI enabled generates zero network traffic:

NAS_VAAI1

Running a Storage vMotion from the NFS01 datastore to NFS02 on the ESXi host without VAAI enabled maxes out the 1Gig ethernet link on the Host:

NAS_VAAI2

This is a rather simple example but it displays how the primitive operates by offloading the I/O tasks to the VNXe array.

Note: If you are accessing the NFS datastore directly via the datastore browser for Copy/Paste functionality then you will not see any benefit from VAAI. This is because the datastore browser has its own API and does not use the internal VMkernel Data Mover or VAAI.

VNXe CPU performance stats during the first SVMotion with VAAI enabled displays approximately 20% Storage Processor utilization and without VAAI enabled you can see CPU % at approx 70% util:

NAS_VAAI3

VNXe Network performance stats display no network traffic with VAAI enabled and without VAAI both read and write for SPA use approx 70MB of bandwidth each:
NAS_VAAI4

Note: For the ‘Full File Clone’ primitive to perform the offload during an SVMotion the VM needs to be powered off for the duration of the SVMotion.

See also Cormac Hogan’s blog post: VAAI Comparison – Block versus NAS

VMware VAAI XCOPY Primitive with VNX OE Release 32

UPDATE: The issue outlined below with code level R32 has now been fixed. This issue is addressed in VNX OE for Block 05.32.000.5.209 (Inyo MR1 SP3). This was covered in EMC Technical Advisory (ETA) 172796 and KB article 172796.

Original Post:
There is a known latency issue with the VMware XCOPY primitive when used with EMC VNX Operating Environment Release 32. The problem is detailed in EMC KB90433 please read this before proceeding with any changes.

“EMC are developing a fix for this issue for the next release of OE R32 currently scheduled for Q4 2013”

The following operations may trigger the bug:
Deploying Templates with VMware
Storage VMotions
Any ESX operation which is used to copy or migrate data within the same physical array

Until the fix is released it would be adivisable to disable the XCOPY primitive. The other VAAI Primitives are functioning correctly.

Please refer to VMware KB http://kb.vmware.com/kb/1033665 on details of how to disable the VAAI Primitives.

On ESX 5.x hosts, to determine if VAAI XCOPY is enabled, run the following command and check if Int Value is set to 1 (enabled):

# esxcli system settings advanced list -o /DataMover/HardwareAcceleratedMove

Expected output for XCOPY ENABLED:

XCOPY1

Command to change the XCOPY to DISABLED mode on an ESX 5.X host:
# esxcli system settings advanced set –int-value 0 –option /DataMover/HardwareAcceleratedMove
XCOPY0

Disabling VAAI using the vSphere Client:

1.Click the Configuration tab.

2.Under Software, click Advanced Settings.

3.Click DataMover.

4.Change the DataMover.HardwareAcceleratedMove setting to 0.

XCOPY2

VAAI USAGE:
HardwareAcceleratedLocking Atomic Test & Set (ATS), which is used during creation of files on the VMFS volume
HardwareAcceleratedMove Clone Blocks/Full Copy/XCOPY, which is used to copy data
HardwareAcceleratedInit Zero Blocks/Write Same, which is used to zero-out disk regions

esxcli system settings advanced list –option=/VMFS3/HardwareAcceleratedLocking
esxcli system settings advanced list –option=/DataMover/HardwareAcceleratedMove
esxcli system settings advanced list –option=/DataMover/HardwareAcceleratedInit