DavidRing.ie

Known ViPR&VPLEX Storage Provisioning Issue:
The following error may be encountered while provision a shared VPLEX distributed volume to an ESXi Cluster using ViPR v2.x – 2.3:

The reason why this issue occurs during a ViPR storage provisioning task with VPLEX is due to the fact that ViPR incorrectly attempts to apply two simultaneous updates to the Cisco MDS IVR database, correctly the MDS database is locked by the first task and the second task times out resulting in a failed ViPR provisioning process. The tasks should be executed in a sequential fashion allowing each task to complete and then commit changes to the IVR database thus removing the lock it held once the commit is successful. Once the database lock is removed then the subsequent task may execute on the database.

Workaround:
Executing an exclusive storage provisioning order from ViPR catalog for a single ESXi host works perfectly, including automatically creating the required Cross-Connect Zoning, this is due to the fact the single workflow performs MDS IVR database updates sequentially. During the single ESXi host exclusive storage provisioning task ViPR creates the necessary initiators, storage views and IVR Zones (both local and cross-connect zoning) for a single host. BUT performing a shared storage provisioning task to an ESXi Cluster fails in a single catalog order, it will also fail if two exclusive storage provision orders are executed at the same time. In summary the workaround is to execute an exclusive storage provisioning order for each host in the cluster individually one at a time. Once this is complete and each host has a volume presented and VPLEX has the correct initiators and storage views created by ViPR, you may then create a new distributed LUN for the whole ESXi cluster. ViPR simply adds the new distributed volumes to existing storage views in VPLEX (there is no zoning going on when you run the ddev creation, thus no locking). Once you have a working distributed volume for all of the hosts, you may then remove the exclusive volumes and everything should function accordingly. Ensure to verify that all the required zoning (including IVR Zones) is configured correctly on all switches and the ESXi hosts can see all associated paths.

NOTE: ViPR engineering plan to enhance the Zoning workflow with an additional step to obtain/monitor any IVR database locks before proceeding with the IVR zoning operations. This will be targeted for the next ViPR release. I will provide updates to this post in due course.

Solution Example:
The below diagram depicts the connectivity requirements in order to implement a ViPR storage provisioning solution with a VPLEX Metro configuration using Cross-Connect Zoning:

From the above digram you can see that an ISL is in place for Site-to-Site connectivity, in this example configuration the ISL carries VPLEX-FC-WAN-Replication traffic over VSAN30(Fabric-A) and VSAN31(Fabric-B) -(VPEX FC WAN COM). VSAN30 is stretched between Fabric-A switches on both sites and VSAN31 is stretched between both switches on Fabric-B for Site1&2. VSAN30&31 can be used as transit VSANs for this example IVR configuration.

In order for ViPR v2.x to successfully execute the task of automatically creating the required cross-connect zoning the following configuration needs to be in place (as per example diagram above):

Site1:
Fabric-A, VSAN10: associated interfaces|PC (even ESX hba of site1, VPLEX FE&BE and PC30) added as members to vsan10.
Fabric-B, VSAN11: associated interfaces|PC (odd ESX hba of site1, VPLEX FE&BE and PC30) added as members to vsan11.
Site 2:
Fabric-A, VSAN20: associated interfaces|PC (even ESX hba of site2, VPLEX FE&BE and PC31) added as members to vsan20.
Fabric-B, VSAN21: – associated interfaces|PC (odd ESX hba of site2, VPLEX FE&BE and PC31) added as members to vsan21.

Site1 – Site2:
Fabric-A: VSAN30 used as a transit vsan over Port-channel 30.
Fabric-B: VSAN31 used as a transit vsan over Port-channel 31.

A prereq is required in order for ViPR to successfully create the cross-connect zoning automatically as part of the provisioning workflow, the prereq is to manually create an IVR zone on fabric A, connecting vsan 10 and vsan 20 and an IVR zone on Fabric B connecting vsan11 and vsan 21 (example IVR Zones provided below).

In the case of ViPR v2.2 an additional prereq task is required and that is to stretch the VSANs between sites, as per this example VSAN20 gets added to switch-A on Site 1 and vice-versa VSAN10 added to switch-A on Site2, repeat same for Fabric-B switches but no local interfaces are assigned to these dummy VSANs, essentially a VSAN20 is created without any member on Switch-A Site1 etc. This is done for all respective VSANs as can be seen in the example configuration provided below. As part of the VSAN stretch ensure to add the allowed VSANs to the respective port-channels:

Port-Channel 30 Allowed VSAN 10,20,30
Port-Channel 31 Allowed VSAN 11,21,31

Once the VSAN is stretched across the sites as per the prereq for ViPR v2.2, ViPR will then automatically create the required IVR zones as part of the provisioning workflow.

Note: The vArray should be set for Automatic Zoning for all this to occur.

Example MDS Configuration
These are example configuration steps to be completed on both sites MDS switches in order to enable Cisco Inter-VSAN Routing (IVR is the standard for cross-connect zoning with VPLEX Metro) and to enable automatic cross-connect zoning with ViPR:

FABRIC ‘A’ Switches

feature ivr
ivr nat
ivr distribute
ivr commit

interface port-channel 30
channel mode active
switchport mode E
switchport trunk allowed vsan 10
switchport trunk allowed vsan add 20
switchport trunk allowed vsan add 30
switchport description CROSS-SITE-LINK
switchport speed 8000
switchport rate-mode dedicated

Configuring FABRIC A switches Fcdoamin priorities:

Site1:
fcdomain priority 2 vsan 10
fcdomain domain 10 static vsan 10
fcdomain priority 100 vsan 20
fcdomain domain 22 static vsan 20
fcdomain priority 2 vsan 30
fcdomain domain 30 static vsan 30

Site2:
fcdomain priority 100 vsan 10
fcdomain domain 12 static vsan 10
fcdomain priority 2 vsan 20
fcdomain domain 20 static vsan 20
fcdomain priority 100 vsan 30
fcdomain domain 32 static vsan 30

Example: configuring Inter-VSAN routing (IVR) Zones connecting an ESXi host HBA0 over VSANs 10 and 20 from site1->site2 and vice versa site2->site1 utilising the transit VSAN30:

device-alias database
device-alias name VPLEXSITE1-E1_A0_FC02 pwwn 50:00:14:42:A0:xx:xx:02
device-alias name VPLEXSITE1-E1_B0_FC02 pwwn 50:00:14:42:B0:xx:xx:02
device-alias name VPLEXSITE2-E1_A0_FC02 pwwn 50:00:14:42:A0:xx:xx:02
device-alias name VPLEXSITE2-E1_B0_FC02 pwwn 50:00:14:42:B0:xx:xx:02
device-alias name ESXi1SITE1-VHBA0 pwwn xx:xx:xx:xx:xx:xx:xx:xx
device-alias name ESXi1SITE2-VHBA0 pwwn xx:xx:xx:xx:xx:xx:xx:xx
device-alias commit
device-alias distribute

ivr zoneset activate name IVR_vplex_hosts_XC_A
ivr commit

FABRIC ‘B’ Switches

feature ivr
ivr nat
ivr distribute
ivr commit

interface port-channel 31
channel mode active
switchport mode E
switchport trunk allowed vsan 11
switchport trunk allowed vsan add 21
switchport trunk allowed vsan add 31
switchport description CROSS-SITE-LINK
switchport speed 8000
switchport rate-mode dedicated

Configuring FABRIC B switches Fcdoamin priorities:

Site1:
fcdomain priority 2 vsan 11
fcdomain domain 11 static vsan 11
fcdomain priority 100 vsan 21
fcdomain domain 23 static vsan 21
fcdomain priority 2 vsan 31
fcdomain domain 31 static vsan 31

Site2:
fcdomain priority 100 vsan 11
fcdomain domain 13 static vsan 11
fcdomain priority 2 vsan 21
fcdomain domain 21 static vsan 21
fcdomain priority 100 vsan 31
fcdomain domain 33 static vsan 31

Example configuring Inter-VSAN routing (IVR) zones connecting an ESXi host HBA1 over VSANs 11 and 21 from site1->site2 and vice versa site2->site1 utilising the transit VSAN31:

device-alias database
device-alias name VPLEXSITE1-E1_A0_FC02 pwwn 50:00:14:42:A0:xx:xx:03
device-alias name VPLEXSITE1-E1_B0_FC02 pwwn 50:00:14:42:B0:xx:xx:03
device-alias name VPLEXSITE2-E1_A0_FC02 pwwn 50:00:14:42:A0:xx:xx:03
device-alias name VPLEXSITE2-E1_B0_FC02 pwwn 50:00:14:42:B0:xx:xx:03
device-alias name ESXi1SITE1-VHBA1 pwwn xx:xx:xx:xx:xx:xx:xx:xx
device-alias name ESXi1SITE2-VHBA1 pwwn xx:xx:xx:xx:xx:xx:xx:xx
device-alias commit
device-alias distribute

ivr zoneset activate name IVR_vplex_hosts_XC_B
ivr commit

Verification commands to check status of configuration:
show fcdomain domain-list
Verifies unique domain ID assignment. If a domain overlap exists, edit and verify the allowed-domains list or manually configure static, non-overlapping domains for each participating switch and VSAN.

show interface brief
Verifies if the ports are operational, VSAN membership, and other configuration settings covered previously.

show fcns database
Verifies the name server registration for all devices participating in the IVR.

show zoneset active
Displays zones in the active zone set. This should include configured IVR zones.
show zone active vsan X |grep -i ivr

show ivr fcdomain
Displays the IVR persistent fcdomain database.

show ivr internal
Shows the IVR internal troubleshooting information.

show ivr pending-diff
Shows the IVR pending configuration.

show ivr service-group
Shows the difference between the IVR pending and configured databases.

show ivr tech-support
shows information that is used by your customer support representative to troubleshoot IVR issues.

show ivr virtual-domains
Shows IVR virtual domains for all local VSANs.

show ivr virtual-fcdomain-add-status
Shows IVR virtual fcdomain status.

show ivr vsan-topology
Verifies the configured IVR topology.

show ivr zoneset
Verifies the IVR zone set configuration.

show ivr zone
Verifies the IVR zone configuration.

clear ivr zone database
Clears all configured IVR zone information.
Note: Clearing a zone set erases only the configured zone database, not the active zone database.

Useful CISCO Docs:
Cisco IVR Troubleshooting
IVR Zones and Zonesets

Inter-VSAN Routing (IVR) definition: An IVR zone is a set of end devices that are allowed to communicate across VSANs within their interconnected SAN fabric. An IVR path is a set of switches and Inter-Switch Links (ISLs) through which a frame from an end device in one VSAN can reach another end device in some other VSAN. Multiple paths can exist between two such end devices. A Transit VSAN is a VSAN that exists along an IVR path from the source edge VSAN of that path to the destination edge VSAN of that path, in the example solution diagram above you will see that VSAN 30 and VSAN 31 are transit VSANs. Distributing the IVR Configuration Using CFS: The IVR feature uses the Cisco Fabric Services (CFS) infrastructure to enable efficient configuration management and to provide a single point of configuration for the entire fabric in the VSAN.

Thanks to @HeagaSteve,Joni,Hans,@dclauvel & Sarav for providing valuable input.