EMC VNXe 3200 – Configuration Steps Via UEMCLI (Part1)

October 22, 2015 David Ring VNXe 3200, EMC VNXE, mcx, Uemcli

There are some minor cli changes with VNXe MCX which I will document as part of this series, for VNXe GEN1 please refer to these earlier posts:
EMC VNXe Gen1 Configuration Using Unisphere CLI

The initial configuration steps outlined in Part1 :

Accept End User License Agreement
Change the Admin Password
Apply License File
Commit the IO Modules
Perform a Health Check
Code Upgrade
Create A New User
Change the Service Password
Enable SSH

Accept End User License Agreement
uemcli -d 192.168.1.50 -u Local/admin -p Password123# /sys/eula set -agree yes

Change the Admin Password
uemcli -d 192.168.1.50 -u Local/admin -p Password123# /user/account show
uemcli -d 192.168.1.50 -u Local/admin -p Password123# /user/account -id user_admin set -passwd NewPassword -oldpasswd Password123#
uemcli -d 192.168.1.50 -u Local/admin -p NewPassword /user/account show

Reference ‘Help’ for any assistance:
uemcli -d 192.168.1.50 -u Local/admin -p NewPassword / -help

Apply License File
Firstly gather the Serial Number of the VNXe:
uemcli -d 192.168.1.50 -u Local/admin -p Password123# /sys/general show -detail
Then browse to the EMC registration site, entering the VNXe S/N to retrieve the associated lic file:
https://support.emc.com/servicecenter/registerProduct/

Upload the acquired license file:
uemcli -d 192.168.1.50 -u Local/admin -p NewPassword -upload -f C:\Users\david\Downloads\FL100xxx00005_29-July-2015_exp.lic license
uemcli -d 192.168.1.50 -u Local/admin -p NewPassword /sys/lic show

Commit the IO Modules
The following commits all uncommitted IO modules:
uemcli -d 192.168.1.50 -u Local/admin -p NewPassword /env/iomodule commit

Display a list of system IO modules:
uemcli -d 192.168.1.50 -u Local/admin -p NewPassword /env/iomodule show

Perform a Health Check
It is good practice to perform a Health Check in advance of a code upgrade:
uemcli -d 192.168.1.50 -u Local/admin -p NewPassword /sys/general healthcheck

Code Upgrade
uemcli -d 192.168.1.50 -u Local/admin -p NewPassword /sys/soft/ver show
uemcli -d 192.168.1.50 -u Local/admin -p NewPassword -upload -f “C:\Users\david\Downloads\VNXe 2.4.3.21980\VNXe-MR4SP3.1-upgrade-2.4.3.21980-RETAIL.tgz.bin.gpg” upgrade
uemcli -d 192.168.1.50 -u Local/admin -p NewPassword /sys/soft/ver show
uemcli -d 192.168.1.50 -u Local/admin -p NewPassword /sys/soft/upgrade create -candId CAND_1
uemcli -d 192.168.1.50 -u Local/admin -p NewPassword /sys/soft/upgrade show
uemcli -d 192.168.1.50 -u Local/admin -p NewPassword /sys/general healthcheck

Note: Please see a more detailed overview of the upgrade process in a previous post:
https://davidring.ie/2015/03/02/emc-vnxe-code-upgrade/

Create A New User
uemcli -d 192.168.1.50 -u Local/admin -p NewPassword /user/account create -name david -type local -passwd DavidPassword -role administrator
uemcli -d 192.168.1.50 -u Local/admin -p NewPassword /user/account show

The role for the new account can be:
• administrator — Administrator
• storageadmin — Storage Administrator
• operator — Operator (view only)

Change the Service Password
The Service password is used for performing service actions on the VNXe.
uemcli -d 10.0.0.1 -u Local/admin -p Password123# /service/user set -passwd newPassword -oldpasswd Password123#

Enable SSH
uemcli -d 192.168.1.50 -u service -p NewPassword /service/ssh set -enabled yes

EMC VNX2 – Drive Layout (Guidelines & Considerations)

September 7, 2015September 7, 2015 David Ring VNX DRIVE, EMC, Layout, mcx, VNX

Applies only to VNX2 Systems.

CHOICES made in relation to the physical placement of Drives within a VNX can have an impact on how the VNX performs. The intention here is to shed some light on how to best optimize the VNX by placing Drives in their best physical locations within the VNX Array. The guidelines here deal with optimising the Back-End system resources. While these considerations and examples may help with choices around the physical location of Drives you should always work with a EMC certified resource in completing such an exercise.

Maximum Available Drive Slots
You cannot exceed the maximum slot count, doing so will result in drives becoming unavailable. Drive form factor and DAE type may be a consideration here to ensure you are not exceeding the stated maximum. Thus the max slot count dictates the maximum drives and the overall capacity a system can support.

BALANCE
BALANCE is the key when designing the VNX drive layout:

Where possible the best practice is to EVENLY BALANCE each drive type across all available back-end system BUSES.This will result in the best utilization of system resources and help to avoid potential system bottlenecks. VNX2 has no restrictions around using or spanning drives across Bus 0 Enclosure 0.

DRIVE PERFORMANCE
These are rule of thumb figures which can be used as a guideline for each type of drive used in a VNX2 system.
Throughput (IOPS) figures are based on small block random I/O workloads:

Bandwidth (MB/s) figures are based on large block sequential I/O workloads:

Recommended Order of Drive Population:

1. FAST Cache
2. FLASH VP
3. SAS 15K
4. SAS 10K
5. NL-SAS

Physical placement should always begin at Bus0 Enclosure0 (0_0) and the first drives to get placed are always the fastest drives as per the above order. Start at the first available slot on each BUS and evenly balance the available Flash drives across the first slots of the first enclosure of each bus beginning with the FAST Cache drives. This ensures that FLASH Drives endure the lowest latency possible on the system and the greatest RoI is achieved.

FAST CACHE
FAST Cache drives are configured as RAID-1 mirrors and again it is good practice to balance the drives across all available back-end buses. Amount of FAST Cache drives per B/E Bus differs for each system but ideally aim for no more than 8 drives per bus (Including SPARE), this is due to the fact that FAST Cache drives are extremely I/O Intensive and placing more than the recommended maximum per Bus may cause I/O saturation on the Bus.

Note: Do not mix different drive capacity sizes for FAST Cache, either use all 100GB or all 200GB drive types.

Also for VNX2 systems there are two types of SSD available:
• ‘FAST Cache SSDs’ are single-level cell (SLC) Flash drives that are targeted for use with FAST Cache. These drives are available in 100GB and 200GB capacities and can be used both as FAST Cache and as TIER-1 drives in a storage pool.
• ‘FAST VP SSDs’ are enterprise Multi-level cell (eMLC) drives that are targeted for use as TIER-1 drives in a storage pool (Not supported as ‘FAST Cache’ drives). They are available in three flavors 100GB, 200GB and 400GB.

More detailed post on FAST Cache: ‘EMC VNX – FAST Cache’

DRIVE FORM FACTOR
Drive form factor (2.5″ | 3.5“) is an important consideration. For example if you have a 6 BUS System with 6 DAE’s (one DAE per BUS) consisting of 2 x 2.5” Derringer DAEs and 4 x 3.5” Viper DAEs as follows:

MCx HOT SPARING CONSIDERATIONS
Best practice is to ensure 1 spare is available per 30 of each drive type. When there are different drives of the same type in a VNX, but different speeds, form factors or capacities, then these should ideally be placed on different buses.

Note: Vault drives 0_0_0 – 0_0_3 if 300GB in size then no spare is required, but if larger than 300G is used and user luns are present on the Vault then a spare is required in this case.

While all un-configured drives in the VNX2 Array will be available to be used as a Hot Spare, a specific set of rules are used to determine the most suitable drive to use as a replacement for a failed drive:

1. Drive Type: All suitable drive types are gathered.
2. Bus: Which of the suitable drives are contained within the same bus as the failing drive.
3. Size: Following on from the Bus query MCx will then select a drive of the same size or if none available then a larger drive will be chosen.
4. Enclosure: This is another new feature where MCx will analyse the results of the previous steps to check if the Enclosure that contains the actual Failing drive has a suitable replacement within the DAE itself.

See previous post for more info: ‘EMC VNX – MCx Hot Sparing’

Drive Layout EXAMPLE 1:

VNX 5600 (2 BUS)

FAST Cache:
1 X Spare, 8 x FAST Cache Avail.
8 / 2 BUSES = 4 FAST Cache Drives Per BUS
1 x 2.5” SPARE Placed on 0_0_24
1 X Spare, 20 x Flash VP Avail.
———————————
Fast VP:
20 / 2 BUSES = 10 Per BUS
10 x 3.5” Placed on BUS 0 Encl 1
10 x 2.5” Placed on BUS 1 Encl 0
1 X 2.5” SPARE Placed on 1_0_24
———————————

Drive Layout EXAMPLE 2:
VNX 5800 (6 BUS)

Drive Layout EXAMPLE 3:
VNX 8000 (16 BUS)

Useful Reference:
EMC VNX2 Unified Best Practices for Performance

EMC VNXe 3200 – MCx Drive Mobility

May 20, 2015 David Ring VNXe drive mobility, EMC, Layout, mcx, Sparing, VNXe

MCx code has brought many new features including the revolutionary ‘Multicore Raid’ which includes the ‘Drive Mobility’ feature. Drive Mobility (also referred to as Portable Drives) allows for the physical relocation of drives within the same VNXe, this provides the flexibility to relocate a drive to another slot within the same DAE or to another DAE either on the same BUS or to another BUS. This option allows us to modify the storage layout of a VNXe which may be very useful for example if additional DAEs are purchased and/or for performance reasons a re-balance of certain drives across DAEs or to a different BUS is required. Another reason as outlined in the related post highlighted above is when a drive failure occurs and you wish to move the spared drive to the failed drive slot location once the rebuild has completed.

The Drive relocation can be executed online without any impact, provided the Drive is relocated within the 5 minute window allowed before the VNXe flags the missing drive as a failure and invokes a spare drive. No other drive within the RAID N+1 configuration can be moved at the same time, moving another drive at the same time if exceeding the RAID N+1 configuration, for example moving more than 1 drive at a time in a RAID-5 configuration may result in a Data Unavailable(DU) situation and/or data corruption. Once the drive is physically removed from its slot then a 5 minute timer kicks in, if the drive is not successfully relocated to another slot within the system by the end of this 5 minute window then a spare drive is invoked to permanently replace the pulled drive. During the physical relocation process of a single drive within a pool the health status of the pool shall display ‘degraded’ until such time as the drive has been successfully relocated to another slot at which time the pool shall return to a healthy state with no permanent sparing or data loss occurring due to the fact that a single drive in a RAID N+1 configuration was moved within the 5 minute relocation window allocated. At this stage a second drive from within the same pool can be moved, continuing this process until you achieve the desired drive layout.

You may wonder how this Drive Mobility is possible: With MCx when a Raid Group is created the drives within the Raid Group get recorded using the drives serial numbers rather than using the drives physical B_E_D location which was the FLARE approach. This new MCx approach of using the drives serial number is known as VD (Virtual Drive) and allows the drive to be moved to any slot within the VNXe as the drive is not mapped to a specific physical location but instead is recorded based on the drives serial number.

Note: System drives DPE_0_0 – 0_3 are excluded from any Drive Mobility:

Example Drive Relocation
For this example the drive located in SLOT-5 of the DPE will be physically removed and placed in SLOT-4 on the same DPE.

Examine the health status of the Drive in SLOT-5 prior to the relocation:
uemcli -d VNXe_IP -u Local/admin -p Password /env/disk -id dpe_disk_5 -detail

After the relocation of the drive (to SLOT-4 in this example) UNISPHERE will temporarily display a warning:

At this stage it is good practice to perform some checks on the Drive(SLOT-4), Pool and System:
uemcli -d VNXe_IP -u Local/admin -p Password /stor/config/pool show -detail uemcli -d VNXe_IP -u Local/admin -p Password /sys/general healthcheck uemcli -d VNXe_IP -u Local/admin -p Password /env/disk -id dpe_disk_4 -detail

Returning to UNISPHERE after performing the checks and you will notice all warnings have disappeared:

At this stage it is safe to proceed with the next move!

EMC VNX – Pool LUN Ownership

October 24, 2014October 29, 2014 David Ring VNX Allocation, EMC VNX, mcx, Ownership, POOL LUN

For every Pool LUN regardless of whether the LUN is thick or thinly provisioned there are 3 owner types assigned to each LUN:

♦ Allocation Owner
♦ Default Owner
♦ Current Owner

There has been cases where inconsistencies in Current, Default & Allocation LUN ownership results in poor performance. Please review the EMC KB88169 written by Dave Reed for more detail.

Jon Klaus goes into good detail on the topic in his post ‘VNX Storage Pool LUN Ownership’

All three owner types need to align with the same ‘VNX Storage Processor’ (A/B) for best performance. If all 3 owners are aligned correctly then there is no need for the system to redirect I/O across the CMI to a peer Storage Processor. This is an example of a best practice LUN ownership configuration as viewable from the LUN properties in Unisphere:

With VNX2 when you create multiple LUNs from Unisphere the ownership is evenly balanced between SP A/B, this results in a balanced LUN configration. With VNX1 you need to specify the owner under the advanced tab while creating the LUN(s) in Unisphere to keep the LUNs evenly balanced across Storage Processors, or you can create the LUNs using cli and specify the SP owner as follows:

naviseccli lun -create -type nonThin -poolName ‘PoolName’ -sp a -capacity 20 -sq gb -name ‘LunName_SPA’ -l 1

naviseccli lun -create -type nonThin -poolName ‘PoolName’ -sp b -capacity 20 -sq gb -name ‘LunName_SPB’ -l 2

To generate a report of all POOL LUN properties including Ownership values from Unisphere chose the system tab and click on the ‘Reports’ option:

From the next window chose the reports window and select Pool LUNs:

You will then be presented with a HTML report with details for each LUN:

You can also use the following Navicli command to generate a listing of all your LUNs Ownership values:
naviseccli -h SP_IP lun -list -alowner -default -owner

Example Output:
LOGICAL UNIT NUMBER 1 Name: LunName_SPA Current Owner: SP A Default Owner: SP A Allocation Owner: SP A

LOGICAL UNIT NUMBER 2 Name: LunName_SPB Current Owner: SP B Default Owner: SP B Allocation Owner: SP B

To lookup the ownership values on an individual LUN basis use the navicli command below or from Unisphere view the properties of the LUN (as per above):
naviseccli -h SP_IP lun -list -l 1 -alowner -default -owner
naviseccli -h SP_IP lun -list -l 2 -alowner -default -owner

Changing Ownership
Now we are going to look at how we can change the Pool LUN ownership types beginning with the Allocation owner. Changing the Allocation Owner is only recommended if you have an imbalance of SP LUN ownership in your system. The only way to achieve this (changing the Allocation Owner) is to create another LUN of equal characteristics but on the other Storage Processor and then migrate the source LUN to the newly created target LUN on the opposite SP as the following example demonstrates:

1. For example create a New pool LUN (target) to migrate the source LUN to:
naviseccli lun -create -type nonThin -poolName ‘PoolName’ -sp b -capacity 20 -sq gb -name ‘LunName_SPA’ -l 3

2. Migrate ‘LUN ID 1’ to the new LUN which now has an allocation owner of SP B (More on LUN migration HERE):
naviseccli -h SP_IP migrate -start -source 1 -dest 3 -rate asap

After the migration you can see that the Allocation Owner is now ‘SP B’ but the Current and Default Ownership’s remain with ‘SP A’:
naviseccli -h SP_IP lun -list -l 1 -alowner -default -owner

LOGICAL UNIT NUMBER 1 Current Owner: SP A Default Owner: SP A Allocation Owner: SP B

3. Changing the Default Owner using the LUN -modify command or change through the LUN properties window in Unisphere:
naviseccli -h SP_IP lun -modify -l 1 -sp B

4. To change the Current Owner you will need to execute a trespass command on the LUN using navicli or by right clicking on the LUN in Unisphere and click the trespass option:
naviseccli -h SP_IP trespass lun 1
If changing on multiple LUNs then running the trespass mine command from the SP will trespass all the LUNs that the SP has DEFAULT ownership of. For example to trespass LUNs with Default Ownership of ‘SP B’ but which are currently owned by ‘SP A’:
naviseccli -h SPB_IP trespass mine

After completing these 4 steps the LUN ID 1 now has Current and Default ownership to match the Allocation owner:
naviseccli -h SP_IP lun -list -l 1 -alowner -default -owner

LOGICAL UNIT NUMBER 1 Current Owner: SP B Default Owner: SP B Allocation Owner: SP B

Note: RAID Group LUNs do not have an allocation owner. With MCX Raid Group LUNs are truly symmetric active/active – Future enhancement for POOL based LUNs, until then please be aware of your LUN ownership values.

EMC VNX – MCx Hot Sparing Considerations

May 5, 2014July 10, 2015 David Ring VNX drive mobility, flare, hot spare, matrix, mcx, permanent, VNX

MCx has brought changes to the way Hot Sparing works in a VNX Array. Permanent Sparing is now the method used when a drive fails. How Permanent Sparing works: when a drive fails in a Raid Group (Traditional RG or Pool internal private RG) the RG rebuilds to a suitable spare drive in the VNX, this used Spare drive now becomes a permanent member of the RG. When the failed drive gets replaced then it becomes a Spare for eligible drives within the entire VNX Array. This new method of sparing eliminates the previous method used by Flare where the Hot Spare would equalize back to the original drive location (B_E_D) once the drive had been replaced.
Note: The rebuild does not initiate until 5 minutes after the drive has been detected as failed.
If you still prefer to keep the original physical Raid Group drive layout intact after the failed drive has been replaced then you can do a manual CopyToDisk from the original spare (which is now a member of the Raid Group) to the replaced drive by issuing the following navi command:
naviseccli –h SPA -user user -password password -scope 0 copytodisk sourcedisk destinationdisk
♦Source and Destination Disk = B_E_D format
For example to complete a manual CopyToDisk from source drive 2_0_14 to 2_0_0 then the following command would need to be run:
naviseccli –h SPA -user user -password password -scope 0 copytodisk 2_0_14 2_0_0
In addition to the CopyToDisk approach to keeping your original physical Raid Group B_E_D structures in place there is also another new MCx feature called Drive Mobility that allows you to swap the failed drive with the Spare that has been used as its replacement. For example you may have a failed drive in 2_0_0 which is part of a Raid_5(4+1) 2_0_0 – 2_0_4

After the 5 min timer 2_0_0 gets automatically replaced by a suitable spare drive in slot 2_0_14 and the Raid Group is rebuilt:

Once the rebuild is complete you may physically remove the drive in 2_0_14 and place it in 2_0_0 in order to restore your original Raid Group B_E_D structure (Once the drive is pulled from 2_0_14 then it must be relocated within 5 Minutes to slot 2_0_0 else a spare will be engaged to replace 2_0_14) Navicli can be used to ensure that the rebuild has 100% completed:
navicli -h SPA getdisk 2_0_14 -rb

Spare location 2_0_14 is then replaced with a new drive:

You may wonder how this Drive Mobility is possible: With MCx when a Raid Group is created the drives within the Raid Group get recorded using the drives serial numbers rather than using the drives physical B_E_D location which was the FLARE approach. This new MCx approach of using the drives serial number is known as VD (Virtual Drive) and allows the drive to be moved to any slot within the VNX as the drive is not mapped to a specific physical B_E_D location but instead is recorded based on the drives serial number. Note: Vault drives 0_0_0 – 0_0_3 are excluded from any Drive Mobility and in fact if the Vault drives are 300 Gig in size then no spare is required, but if larger than 300G is used and user luns are present on the Vault then a spare is required in this case (To avoid alerts in Unisphere for not having the required spare drives available in relation to VAULT drives change the hot spare policy to custom and set the keep unused to 0).

While all un-configured drives in the VNX Array will be available to be used as a HS, a specific set of rules are used to determine the most suitable drive to use as a replacement for a failed drive:

1.Drive Type: All suitable drive types are gathered. (See matrix below)
2.Bus: Which of the suitable drives are contained within the same bus as the failing drive.
3.Size: Following on from the Bus query MCx will then select a drive of the same size or if none available then a larger drive will be chosen.
4.Enclosure: This is another new feature where MCx will analyse the results of the previous steps to check if the Enclosure that contains the actual Failing drive has a suitable replacement within the DAE itself.

Hot Spare Drive Matrix:

♦Here is a pdf download of the Matrix.

Best practice is to ensure 1 spare is available per 30 of each drive type. There is known bug with MCx Revision: 05.33.000.5.051 where recommended is displayed as 1/60 as you can see below, this is due to be fixed with the next release to reflect the 1/30 ratio. The three policy options are Recommended, Custom or NO Hot Spare. If the ‘No Hot Spare’ option is chosen this does not necessarily mean that no HS will be used, the system will spare to a drive in the system if a suitable match is found; this option just allows the user to configure all available drives of this type for use within Raid Groups. You can either use CLI or Unsphere to analyse the Policies defined on the array:

Also see Jon Klaus Post “VNX2 Hot Spare and Drive Mobility Hands-On”

EMC VNX – Second Generation (Rockies)

September 11, 2013July 10, 2014 David Ring VNX EMC, mcx, Next Gen VNX, rockies, VNX, VNX2

“32 Cores firing MCx Technology”

“Leverage cores and scale them – core scaling to achieve the million IOPS”

“Amdahl’s law 80% scaling”

“Up to 1.1 million IOPS (I/Os per second) performance, 30-GBps bandwidth, and 6 PB of capacity”

“Million IO’s a second with sub ms response time”

“7600 all flash offering up to 1/2 million IOPS”

“Dedupe lun by lun not the entire array”

“6x the number of VMs on a common system”

“VDM: Containerise your filesystem and move seamlessly”

“Improved FAST VP granularity to 256 MB”

“Second-generation VNX will be at the center of a new preconfigured Vblock converged infrastructure solution from VCE”

These were just some of the highlights from the Second Generation VNX Launch.

In this post we will take a look at some of the changes in hardware focusing on DPE, SPE and the new CS.

The full range of VNX Models have been rearchitected with new hardware and software. The VNX OE releases are codenamed after mountain ranges with this latest release codenamed “Rockies”, more on VNX code naming schemes can be found on Chads blog post: EMC VNX Inyo: “Hello World!”.

The main components of Rockies:
• New “Megatron” Storage Processor Enclosure (SPE) – (VNX 8000)
• New “Jetfire” Disk Processor Enclosure (DPE) – (VNX 5400-7600)
• New “Dobie” Control Station (CS)
• Existing “Argonaut” Data Mover Enclosures (DME)
• Existing 15, 25, 60 drive Disk Array Enclosures (DAE)

Meet the new Range of VNX Models which are available in unified block and file, block only, or file only configurations:

“Not a refresh a re-architecture”

New Disk and Storage Processor Enclosures:

There is a new 3U Disk Processor Enclosure (DPE) containing two SPs with 25×2.5″ drives. This DPE (Jetfire) will be used in the 5400 to 7600 systems. As can be seen above there are different processing capabilities dependent on the model. Two 1100W power supplies, one per CPU module. Four fan packs, two per CPU module – Torque Limiting Screw (TLS) for removal/insertion.

For the 8000 model there is a new 4U Storage Processor Enclosure (SPE) that holds the two SP’s. Each SP consisting of 2x8core 2.7GHz Intel Sandy Bridge CPUs (16Cores Total), 128GB of 1600MHz DDR3 memory, 11 I/O PCI-E Gen3 slots with 9 slots supporting FC, FCoE, and iSCSI and the other 2 slots reserved for back-end connectivity. An update to the SP OS allowed for these greater memory configurations. This gives a total of 32 cores, 22 I/O Slots and 256GB of memory across the 2 SPs if you compare this to a VNX7500 which has 12 cores, 10 PCI-E Gen2 I/O slots and 96GB of memory you have a serious increase in performance. Two 2U dual Li-Ion SPSs are also included with One SPS dedicated to the Megatron SPE and the other one used for vault DAE.

New Control Station (Dobie)
The new 1U Control Station codenamed “Dobie” is an upgrade from the Maynard CS providing enahanced Unisphere performance using a multi core processor: 4 Core 3.1GHz Xeon CPU and 8GB memory. The same Dobie CS will be used across all models. File side connectivity continues to be provided for by the Argonaut X-Blades.

POOL Configurations

Control Station Cabling
The new CS has the same level of connectivity as the Maynard CS with the only change being the location of the IPMI and CS A ports being swithced: