DavidRing.ie

An issue I encountered recently was in relation to the number of FLOGIs achievable on a single CISCO 9148 MDS PortChannel running on NX-OS Release 5.2(8e):

Cisco MDS NX-OS Release 5.2(x) Maximum Configuration Limit FLOGIs per port channel = 114

In this example we had a UCS environment with greater than 114 Blades (128 Blades to be exact) FLOGI was successfully completing for 126 out of the 128 UCS B-Series Blades (surprising as the stated limit was 114).

Example command outputs while running code version 5.2(8e) and exceeding the stated maximum limit:

MDS-9K-A# show logging logfile | include FLOGI
2015 Nov 9 00:57:05.182 MDS-9K-A %FLOGI-1-MSG_FLOGI_REJECT_FCID_ERROR: %$V
SAN 10%$ [VSAN 10, Interface port-channel 10: mode[F]] (pwwn: 20:00:00:25:b5:05:
XX:XX) FLOGI rejected – FCID allocation failed with error 0x401b0000.

MDS-9K-A# show flogi internal info | i key|Interface | i key p 1
FLOGI rejected – FCID allocation failed with error 0x401b0000

MDS-9K-A# show flogi database interface port-channel 10
Total number of flogi = 127

There was two options available to remedy this problem: one was to split the port-channel, this configuration would create two port-channels on each SAN Fabric switch (4 port-channels total) and split the blades HBA’s between the two PortChannels (64 FLOGIs per PortChannel) giving us the ability to cater for our required 128 FLOGIs per FABRIC/switch or 256 in total across both FABRICs. In fact this solution would theoretically allow for 228 FLOGIs per switch divided equally across both port-channels per switch or a theoretical maximum of 456 FLOGIs across both FABRICs SwitchA&B.

The second option and my preferred route is to perform a code upgrade of the switch:

Cisco MDS NX-OS Release 6.2(x) Maximum Configuration Limit FLOGIs per port channel = 256

Output after upgrading to 6.2:
MDS-9K-A# show flogi database interface port-channel 10
Total number of flogi = 129
Success- all 128 HBA’s are successfully logging into MDS-A switch via a single PortChannel.

MDS-9K-A# show flogi database interface port-channel 11
Total number of flogi = 129
Success- all 128 HBA’s are successfully logging into MDS-B switch via a single PortChannel.

Also worth noting:

NX-OS 5.2.x
PortChannels and member ports in PortChannels = 16 PortChannels with 16 members ports in all PortChannels. For example you could have 16 PortChannels each with 1 member, 2 PortChannels with 8 members or 1 PortChannel with 16 members.
NX-OS 5.2 Limits

NX-OS 6.2.x
PortChannels and member ports in PortChannels = 48 PortChannels with a maximum of 16 members in each PortChannel.
NX-OS 6.2 Limits