DellEMC VxBlock – Cisco UCS Ethernet Adapter Policy

The following details the creation of a VxBlock Cisco UCS ‘Ethernet Adapter Policy’ for a compute B-Series blade running VMware ESXi. This policy is leveraged by a service-profile which gets associated with their respective compute blades.

Note: This guidance is based on vSphere 6.0

UCS Manager GUI

  1. From the UCS Manager GUI click on ‘Servers’ and navigate to Policies->root->Adapter Policies.
  2. Chose the option to ‘Add’ a new Ethernet Adapter Policy and provide a name & description (e.g VMQ-Default).ucs_eth_adpt_policy3
  3. Under Resources enter the following settings:

Continue reading

ViPR Controller – Ingest V(x)Block UCS Boot Volumes

The following provides guidance in a scenario where your ESXi host boots from SAN, for example this is the standard configuration for Cisco UCS blades included in a Vblock/VxBlock. These boot volumes will most likely have been created outside of ViPR Controller, thus it will be a requirement to perform an ingestion of these volumes prior to performing any host migration procedures, for example moving a UCS ESXi blade to another cluster in vCenter.

Note: by not performing the ingestion will result in the removal of the Boot volume from the ESXi host masking if you are initiating the migration process utilizing ViPR commands (more on this later).

Note: To avoid ViPR provisioning issues ensure the ESXi BOOT volume masking views have _NO_VIPR appended to their exclusive mask name, this will prevent ViPR from using the exclusive export mask when adding A NEW ESXi host to a cluster:

BootVolumeMaskingViewName_NO_VIPR

Continue reading

Cisco UCS – Determining ESXi FNIC&ENIC via PowerCLI

The following script allows the user to retrieve a listing of Network (ENIC) & Storage (FNIC) firmware drivers installed on Cisco UCS blades at a per vSphere cluster level. You may download the ‘Cisco_FNIC_ENIC.ps1‘ script here: Cisco_FNIC_ENIC.ps1 (Remove the .doc extension).

The script will begin by prompting you to enter the vCenter IP Address, username and password. A list of all the available clusters residing in vCenter will be returned. Followed by a prompt to enter the vSphere cluster name, from the cluster defined the script will retrieve a per ESXi listing of ENIC&FNIC firmware levels. The script will firstly prompt the user to enable SSH on all the hosts in the cluster:

UCS_FNIC_ENIC1

UCS_FNIC_ENIC2

 

Once you have completed the tasks on the hosts that required SSH Access, you may then return to the running script and type option ‘y’ in order to again disable SSH on all the hosts in the specified cluster:

UCS_FNIC_ENIC3

PowerCLI Script:

#######################################
# Confirm CISCO FNIC & ENIC Drivers
# Date: 2016-07-01
# Created by: David Ring
#######################################

###### vCenter Connectivity Details ######

Write-Host “Please enter the vCenter Host IP Address:” -ForegroundColor Yellow -NoNewline

$VMHost = Read-Host

Write-Host “Please enter the vCenter Username:” -ForegroundColor Yellow -NoNewline

$User = Read-Host

Write-Host “Please enter the vCenter Password:” -ForegroundColor Yellow -NoNewline

$Pass = Read-Host

Connect-VIServer -Server $VMHost -User $User -Password $Pass

###### Please enter the Cluster to check CISCO Versions #######

Write-Host “Clusters Associated with this vCenter:” -ForegroundColor Green

$VMcluster = ‘*’

ForEach ($VMcluster in (Get-Cluster -name $VMcluster)| sort)

{
Write-Host $VMcluster
}

Write-Host “Please enter the Cluster to lookup CISCO FNIC & ENIC Drivers:” -ForegroundColor Yellow -NoNewline

$VMcluster = Read-Host

###### Enabling SSH ######

Write-Host “Do you need to Enable SSH on the Cluster ESXi Hosts? ” -ForegroundColor Yellow -NoNewline

Write-Host ” Y/N:” -ForegroundColor Red -NoNewline

$SSHEnable = Read-Host

if ($SSHEnable -eq “y”) {

Write-Host “Enabling SSH on all hosts in your specified cluster:” -ForegroundColor Green

Get-Cluster $VMcluster | Get-VMHost | ForEach {Start-VMHostService -HostService ($_ | Get-VMHostService | Where {$_.Key -eq “TSM-SSH”})}

}

###### Confirm Driver Versions ######

Write-Host “Confirm CISCO FNIC & ENIC Drivers” -ForegroundColor Green

$hosts = Get-Cluster $VMcluster | Get-VMHost

forEach ($vihost in $hosts)

{

Write-Host -ForegroundColor Magenta “Gathering Driver versions on” $vihost

$esxcli = get-vmhost $vihost | Get-EsxCli

$esxcli.software.vib.list() | Where { $_.Name -like “net-enic”} | Select @{N=”VMHost”;E={$ESXCLI.VMHost}}, Name, Version

$esxcli.software.vib.list() | Where { $_.Name -like “scsi-fnic”} | Select @{N=”VMHost”;E={$ESXCLI.VMHost}}, Name, Version

}

###### Disabling SSH ######

Write-Host “Ready to Disable SSH? ” -ForegroundColor Yellow -NoNewline

Write-Host ” Y/N:” -ForegroundColor Red -NoNewline

$SSHDisable = Read-Host

if ($SSHDisable -eq “y”) {

Write-Host “Disabling SSH” -ForegroundColor Green

Get-Cluster $VMcluster | Get-VMHost | ForEach {Stop-VMHostService -HostService ($_ | Get-VMHostService | Where {$_.Key -eq “TSM-SSH”}) -Confirm:$FALSE}

}

 

Useful References:

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1027206

http://www.cisco.com/c/en/us/support/docs/servers-unified-computing/ucs-b-series-blade-servers/115764-ucs-san-tshoot-00.html

CISCO UCS – Rebooting Fabric Interconnect(s)

Begin by connecting to the cluster IP over SSH and checking which FI is Primary/Subordinate:

FI-A# show cluster state
A: UP, PRIMARY
B: UP, SUBORDINATE

Note: show cluster extended-state will provide more detailed information.

Having confirmed the ‘B’ fabric switch is the subordinate connect to FI-B mgmt cli interface:

FI-A# connect local-mgmt B

From FI-B local-mgmt interface issue the reboot command:

FI-B(local-mgmt)# reboot
Before rebooting, please take a configuration backup.
Do you still want to reboot? (yes/no):yes

Run the ‘cluster state’ command again to check on the status of FI-B:

FI-A(local-mgmt)# show cluster state
A: UP, PRIMARY
B: DOWN, INAPPLICABLE
HA NOT READY
Peer Fabric Interconnect is down

Once the cluster enters a HA READY status, make FI-B the Primary switch in order to reboot FI-A:

FI-A(local-mgmt)# cluster lead b

Note: After initiating a fail over the SSH session will disconnect, re-connect to the cluster and confirm cluster state.

Connect to local mgmt ‘a’ and reboot FI-A:

FI-B# connect local-mgmt a
FI-A(local-mgmt)# reboot
Before rebooting, please take a configuration backup.
Do you still want to reboot? (yes/no):yes

Confirm HA READY status before setting FI-A back to PRIMARY:

FI-B# show cluster state
B: UP, PRIMARY
A: UP, SUBORDINATE
HA READY

Set FI-A as PRIMARY, this will need to be set from current PRIMARY FI-B mgmt interface:

FI-B# connect local-mgmt b
FI-B(local-mgmt)# cluster lead a

The Vblock vCake

Created by Cliff Cahill @CliffCahill

In conjunction with the announcement of the ‘Vblock 740’ @VCE’s latest addition to the Vblock range Cliff kindly put his creative and baking skills to good effect by creating the ‘Vblock 740 vCake’. The Vblock System 740 is VCE’s flagship converged infrastructure, an IT infrastructure built on industry-leading technology combining Network and Compute components from Cisco, Storage in the form of the EMC VMAX³ and virtualization from VMWare – all market leaders in their respective technology sectors.

The steps below outline the process that was followed in order to deliver the Vblock vCake:

1. Logical Configuration Summary(LCS) & Bill Of Materials(BOM)
2. Physical Build Commences
3. Configuring the Advanced Management Platform (AMP)
4. Configuring the Cisco Network Components (MDS, NEXUS) & Compute Components (Cisco UCS)
5. Configuring the EMC VMAX³ & VMWare Virtualization Infrastructure (ESXi, vCenter, VUM)
6. Logical Configuration QA
7. Vblock Sent to Distribution
8. Deployment and Implentation Begins
9. Deployment and Implentation Knowledge Transfer
10. Vblock Enters Production

1. Logical Configuration Summary and Bill of Materials
The vArchitect has taken the customers requirements and expertly sized the Vblock to deliver optimum performance for the customers mission critical applications. The bill of materials has been completed and the Technical Program Team has qualified and validated the solution design. The Logical Configuration Survey has been signed off with the customer and the Vblock has been added to the Production Schedule. All components are laid out in preparation for integration:

vCake1

2. Physical Build Commences
Next our manufacturing team place each part of the Vblock platform into its correct location within the cabinet:
vCake3

The Cisco UCS, fabric interconnects, network switches and EMC VMAX³ are now installed:
vCake4

Then the power outlet units (POU’s) and all network and power cabling get installed and connected to the appropriate ports based on a Master Port Map:
vCake7

3. Configuring the Advanced Management Platform (AMP)
Next the Vblock platform goes through the Logical integration phase where it is expertly configured by the LB Team to meet specific customer requirements, beginning with the AMP configuration which runs the software that manages the platform (Cisco C220 servers and EMC VNXe storage array):
vCake11

4. Configuring the Cisco Network Components (MDS, NEXUS) & Compute Components (Cisco UCS)
Code Upgrades are complete based on the fully tested and validated RCM. Initial and advanced scripting executed:
vCake10

5. Configuring the EMC VMAX³ & VMware Virtualization infrastructure (ESXi, vCenter, VUM)
The EMC VMAX³ Bin File has been loaded and the VMWare ESXi clusters are provisioned via UIM:
vCake12

6. Logical Configuration QA
Logical Configuration Complete, QA Done and VCE Vision verifies compliance with RCM. Hand Off Email Sent:
vCake13

7. Vblock Sent to Distribution
The Vblock is down in distribution and ready for dispatch. Customer agreed delivery date is achieved. Vblock is covered to ensure immaculate delivery to its proud new owner:
vCake14

8. Deployment and Implementation Begins
Vblock arrives at the customer site as a piece of truly converged infrastructure and the Deployment and Implementation phase commences:
photo 1
photo 2

9. Deployment and Implentation Knowledge Transfer
The Vblock has been integrated into the customer environment and Deployment & Implementation knowledge transfer begins:
photo 3

10. Vblock Enters Production
The Vblock is now in operation – End User is already after taking a few compute slices:
photo 4
photo 5

Vblock Infrastructure Platform is Transforming IT
vCake Final

BOM for Biscuit Cake
400g digestives
2 Crunchies
450g milk choc
90g butter
5tbsp double cream
1tbsp golden syrup

BOM for Ganache:
180ml double cream
28g butter
227g dark choc

Logical Build Instructions for vCake
Biscuit Cake
Melt milk choc with butter over pot of simmering water.
Break up biscuits and crush crunchies in a large bowl.
Once milk choc melted, cool slightly and then add cream and golden syrup.
Mix using a whisk until completely mixed through.
Add to digestives and make sure every biscuit is covered.
Put it in to a prelared tim lined with cling film and refrigerate.

Ganache
Break Dark chocolate into Single Squares and add to Glass Bowl
Slowly Bring cream and butter to the boil and poor over dark chocolate.
Allow to sit for a couple of minutes and then mix thoroughly until choc completely melted.
Allow to set until desired consistency has been reached.

Cisco UCS – Reinitialize FC Port Channel

Fabric Interconnect CLI:
By connecting directly to the fabric interconnect you can quickly issue the following commands to Reinitialize (Disable&Enable) the specified FC Port Channel. Often this is required when initally configuring an FC Port Channel with an MDS. The example below will show how to Reinitialize Port Channel 10 on ‘Fabric A’ and Port Channel 11 on ‘Fabric B’:

Note: for assistance on cmds – tail command with a ‘?’

Fabric A
Commands issued in sequence:
• scope fc-uplink
• scope fabric A
• scope port-channel 10
• show
• disable
• commit-buffer
• enable
• commit-buffer
• show
• show detail expand

FI-A# scope fc-uplink
FI-A /fc-uplink # scope fabric A
FI-A /fc-uplink/fabric # scope port-channel 10
FI-A /fc-uplink/fabric/port-channel # show

Port Channel:
Port Channel Id Name Oper State Oper Speed (Gbps)
--------------- ---------- ---------------- -----------------
10 FC-PC10-Fabric-A
Up 32

FI-A /fc-uplink/fabric/port-channel # disable
FI-A /fc-uplink/fabric/port-channel* # commit-buffer
FI-A /fc-uplink/fabric/port-channel # enable
FI-A /fc-uplink/fabric/port-channel* # commit-buffer
FI-A /fc-uplink/fabric/port-channel # show

Port Channel:
Port Channel Id Name Oper State Oper Speed (Gbps)
--------------- ---------- ---------------- -----------------
10 FC-PC10-Fabric-A
Up 32

FI-A /fc-uplink/fabric/port-channel # show detail expand
Port Channel:
Port Channel Id: 10
Name: FC-PC10-Fabric-A
Admin State: Enabled
Oper State: Up
Admin Speed: Auto
Oper Speed (Gbps): 32

Member Port:
Fabric ID: A
Slot Id: 1
Port Id: 29
Membership: Up
Admin State: Enabled
Current Task:

Fabric ID: A
Slot Id: 1
Port Id: 30
Membership: Up
Admin State: Enabled
Current Task:

Fabric ID: A
Slot Id: 1
Port Id: 31
Membership: Up
Admin State: Enabled
Current Task:

Fabric ID: A
Slot Id: 1
Port Id: 32
Membership: Up
Admin State: Enabled
Current Task:

Fabric B
• scope fc-uplink
• scope fabric B
• scope port-channel 11
• show
• disable
• commit-buffer
• enable
• commit-buffer
• show
• show detail expand

In order to troubleshoot the FC Uplink ports:
• scope fc-uplink
• scope fabric A – A|B
• show interface – this will display all FC Uplink Ports
• show interface detail
• scope interface 1 29
• show interface detail
• show fault

Connect Via Nexus OS:
connect nxos a | b
show san-port-channel database
show san-port-channel summary
show san-port-channel internal info all
show interface
show interface brief
show interface fc1/29
show interface transceiver

Check for FLOGI:
show npv flogi-table

UCS Manager:
Click on the SAN tab in UCS Manager, click to expand ‘SAN Cloud’ – ‘Fabric A’ – ‘FC Port Channels’. From here you have the option to Disable|Enable the highlighted FC Port Channel:

UCS Restart FCPC

Cisco UCS – How to Query UCS for WWN’s Associated with Blades

These are very useful trick’s in order to obtain a list of WWN’s associated with the blades in a UCS environment. Instead of trawling through the GUI you can easily connect to UCS via the Cisco UCS PowerTool or SSH to the Fabric Interconnect and quickly retrieve the WWN information for each/All Host/blade’s in UCS.

UCS PowerTool
You can download the latest version of Cisco UCS PowerTool from:
UCS PowerTool Download

Firstly launch the UCS PowerTool and Connect to the UCS system by issuing the cmd:
PS C:\> Connect-Ucs
Enter your fabric interconnect IP address hitting the return key and then inputting your creds.

Once you have connected enter the following cmd to bring up a list of all the blades (service profiles) and their associated vHBA WWN’s:
PS C:\> Get-UcsServiceProfile -type instance | Get-UcsVhba | Select Dn,Addr,NodeAddr

UCS_WWN1

If you want to reduce the list to only WWPN‘s, then use the following:
PS C:\> Get-UcsServiceProfile -type instance | Get-UcsVhba | Select Dn,Addr

UCS_WWN2

Filter by “vHBA-0”:
Get-UcsServiceProfile -type instance |Get-UcsVhba | select DN,Name,Addr| where {$_.Name -eq “vHBA-0”} | sort DN

Filter by “vHBA-1”:
Get-UcsServiceProfile -type instance |Get-UcsVhba | select DN,Name,Addr| where {$_.Name -eq “vHBA-1”} | sort DN

In order to display the associated VSAN details:
PS C:\> Get-UcsServiceProfile -type instance | Get-UcsVhba | Get-UcsVhbaInterface | select Dn,Initiator,Vnet

UCS_WWN3

Get all service profile instances in UCS:
PS C:\> Get-UcsServiceProfile –Type instance

List Boot Policy assigned to each blade:
C:\> Get-UcsServiceProfile -type instance | select Name,BootPolicyName

Fabric Interconnect CLI
Another option is to connect directly to the fabric interconnect. Using the cmd “fi01-A# show server adapter identity” will return a detailed list all of the UCS blades WWN details:
UCS_WWN4

For a brief list of all WWN’s assigned to each host use this simple cmd:
sh identity wwn

• To list an individual server WWPN details:
fiA-31-A# scope chassis 1
fiA-31-A /chassis # scope server 1
fiA-31-A /chassis/server # scope adapter 1
fiA-31-A /chassis/server/adapter # show host-fc-if

FC Interface:
Id Wwn Model Name Operability
———- ———————– ———- ———- ———–
1 20:00:00:25:B5:25:A0:6F UCSB-MLOM-40G-01
vHBA-0 Operable
2 20:00:00:25:B5:25:B1:6F UCSB-MLOM-40G-01
vHBA-1 Operable

• From the scope server is is also possible to learn the DN(Distinguished Name):
fiA-31-A /chassis/server # show server adapter vnics
FC Interface:

Adapter Interface Vnic Dn Dynamic WWPN Type
——- ——— ———- ———— —-
1 1 org-root/ls-xap-esx001/fc-vHBA-0 20:00:00:25:B5:25:A0:6F Fc
1 2 org-root/ls-xap-esx001/fc-vHBA-1 20:00:00:25:B5:25:B1:6F Fc

• Check which vHBA is assigned to fabric A/B:
fiA-31-A# scope service-profile server 1/1
fiA-31-A /org/service-profile # show vhba

vHBA:
Name Fabric ID Dynamic WWPN
———- ——— ————
vHBA-0 A 20:00:00:25:B5:25:A0:6F
vHBA-1 B 20:00:00:25:B5:25:B1:6F

• Get a full List of all WWPN’s based on Fabric A/B:
fiA-31-A# scope org
fiA-31-A /org # show wwn-pool

WWN Pool:
Name Purpose Size Assigned
——————– —————————- ———- ——–
Global-WWNN-Pool Node WWN Assignment 128 9
vHBA-0-Fabric-A Port WWN Assignment 128 9
vHBA-1-Fabric-B Port WWN Assignment 128 9

• List all Fabric-A WWPN and DN information:
fiA-31-A# scope org
fiA-31-A /org # scope wwn-pool vHBA-0-Fabric-A
fiA-31-A /org/wwn-pool # show initiator
WWN Initiator:
Id Name Assigned Assigned To Dn
———————– ———- ——– ————–
20:00:00:25:B5:25:A0:6F Yes org-root/ls-xap-esx001/fc-vHBA-0

• List all Fabric-B WWPN and DN information:
fiA-31-A# scope org
fiA-31-A /org # scope wwn-pool vHBA-1-Fabric-B
fiA-31-A /org/wwn-pool # show initiator

WWN Initiator:
Id Name Assigned Assigned To Dn
———————– ———- ——– ————–
20:00:00:25:B5:25:B1:6F Yes org-root/ls-xap-esx001/fc-vHBA-1

Thank you ‘Brendan Lucey’ for your assistance.