Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment This paper covers Dell recommended best practices for configuring a VMware vSphere environment connected to Dell Storage PS Series arrays. Dell Storage Engineering December 2015

A Dell Technical White Paper

Revisions Version

Date

Description

1.0

August 2013

Initial Release

1.1

August 2013

Added best practices around CHAP and ACL

1.2

December 2014

Added: ESXi v5.5 EQL FW v7.0 Updated: Delayed ACK / LRO

1.3

December 2015

Added: ESXi v6.0, PS Series FW v8.0, NOOP Timeout, VVOL Moved to appendix: ESX v4.1, ESX v5.0/5.1 Removed: LRO

Authored By: Don Williams, David Sevene, Chuck Armstrong

THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND. Copyright © 2013 - 2015 Dell Inc. All rights reserved. Dell and the Dell logo are trademarks of Dell Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.

3

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

Table of contents 1

Introduction ................................................................................................................................................................................ 6

2

Host based connectivity and high availability ....................................................................................................................... 7

3

Keep ESXi updated to current revision ........................................................................................................................ 7

2.2

iSCSI subnets and routing recommendations ........................................................................................................... 7

2.3

Jumbo frames .................................................................................................................................................................. 7

2.4

CHAP authentication for access control .................................................................................................................... 7

2.5

Selecting optimal MPIO mode...................................................................................................................................... 8

Host based performance ........................................................................................................................................................ 13 3.1

Delayed ACK .................................................................................................................................................................. 13

3.2

Storage I/O control ....................................................................................................................................................... 18

4

Datastore size and queue depth ........................................................................................................................................... 20

5

Virtual machine considerations ............................................................................................................................................. 21

6

Data Drives ................................................................................................................................................................................ 22 6.1

VMDK on VMFS .............................................................................................................................................................. 22

6.2

iSCSI in the guest VM ................................................................................................................................................... 25

6.3

Raw Device Mapped LUNs .......................................................................................................................................... 31

7

Conclusion ................................................................................................................................................................................ 31

A

ESX 4.1 and ESX 5.0/5.1 information .................................................................................................................................... 32

B

4

2.1

A.1

Host based connectivity .............................................................................................................................................. 32

A.2

Datastore size and queue depth ................................................................................................................................ 34

Technical support and customer service ............................................................................................................................ 35 B.1

Contacting Dell ............................................................................................................................................................. 35

B.2

Warranty information ................................................................................................................................................... 35

B.3

Related documentation ............................................................................................................................................... 35

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

Preface Dell™ Storage PS Series arrays optimize resources by automating performance and network load balancing. Additionally, they offer all-inclusive array management software, host software, and free firmware updates. The information in this guide is intended for VMware ® and PS Series storage administrators and installers. Table 1

Related software and firmware used for the preparation of this paper

Vendor

Model

Software Revision

Dell

PS Series storage firmware

8.x

VMware

VMware vSphere®/ESXi™

5.5, 6.0

Executive summary This technical report is aimed at VMware and PS Series storage administrators that are configuring a virtual environment. This document covers connectivity and high availability as well as some common performance tuning best practices. In addition, there is a discussion on various methods to implement virtual machine data drive connectivity.

5

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

1

Introduction As more organizations are virtualizing datacenters that use both VMware vSphere and PS Series storage, best practices and configuration guides such as this one, help to keep these environments running smoothly. These best practices and recommendations are a combination of various technical report guidelines as well as knowledge provided by the technical support staff at Dell. Various documents provide extensive detail on how to configure some of the recommended settings. In these instances, instead of adding the same information to this document, the others are referenced. This document notes differences between versions of VMware vSphere within each section. For historic reference, specifics relating to older versions of VMware vSphere/ESX are contained in Appendix A of this document.

6

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

2

Host based connectivity and high availability Several best practices should be implemented when connecting VMware vSphere/ESXi servers to PS Series storage to ensure optimum performance and high availability.

2.1

Keep ESXi updated to current revision It is important to ensure that the most current ESXi version, also known as the build number, is installed. Using the VMware vSphere Web Client, select an individual ESXi server under Hosts and Clusters, and expand the Configuration window to display the version and build number. When using the VMware vSphere Client GUI, select an individual ESXi sever to display the version and build number. VMware Update Manager, part of vSphere vCenter, provides this updating functionality.

2.2

iSCSI subnets and routing recommendations VMware vSphere does not support routing when using the iSCSI port binding (described in the VMware Knowledge Base article 2038869) that is required for Round Robin and EqualLogic Multipath Extension Module (MEM) MPIO with PS Series storage. The VMkernel ports used to communicate with the PS Series storage must be on the same subnet and able to communicate directly with the SAN and not through a gateway.

2.3

Jumbo frames Jumbo frames are not required but are recommended for an end-to-end switching environment. In order to support jumbo frames the vmnic, ESXi VMkernel ports, and vSwitch must be configured to use an MTU of 9000. The physical switches must also be capable of using jumbo frames and properly configured for them. When using iSCSI offload cards, verify the compatibility of the cards and support for jumbo frames before enabling it in the environment.

2.4

CHAP authentication for access control Manipulating MPIO modules and increasing volume sizes are both great ways to manage volume iSCSI connections. There are other best practices and recommendations that can also help control how connections are made to PS Series volumes. CHAP authentication for access control lists authentication as the preferred method of volume access authentication from an ease of administration point of view. Each PS Series volume ACL (access control list) has a maximum of 16 entries (this is not related to iSCSI connections) that can be CHAP accounts, IQN strings or IP addresses. In keeping with designing a scalable, flexible environment, CHAP allows rapid deployment and discovery of volumes with a single ACL entry. For example, Cluster A could have a single CHAP entry to talk to all of its volumes and Cluster B has a different single CHAP entry to talk to all of its volumes. Shared volumes such as templates or shared data would have both Cluster A CHAP and Cluster B CHAP assigned to it dramatically decreasing the administration of ACL lists.

7

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

Note: In a VMware environment, this allows rapid movement of Datastores between clusters. By unregistering the VMs, changing the ACL and rescanning, entire volumes or Datastores can be moved between clusters. Using CHAP helps with scalability, as well as ease of management. When a new host server is introduced to a cluster, the administrator only needs to set the CHAP configuration on the iSCSI initiator so that the new host can access the cluster storage resources. When a new volume is created, only one ACL entry and an initiator rescan are needed for all the cluster host servers to be able to access the additional storage capacity.

2.5

Selecting optimal MPIO mode

2.5.1

Multipath Extension Module MEM is a software package that can be installed in VMware environments to provide intelligent load balancing, multipathing, and failover. If you have a VMware vSphere Enterprise or Enterprise+ license, VMware allows you to install vendor specific MPIO software. These Path Selection Policy (PSP) modules, allow storage vendors to tune MPIO optimally for their hardware. In environments with the appropriate licensing, this is the preferred option for best performance. The MEM will optimize both Datastore I/O and Raw Device Mapped (RDM) volumes as well. In addition to providing intelligent multipathing, the MEM package also includes setup scripts that can ease the configuration the iSCSI configuration on ESXi hosts. While the steps may differ between versions, the MEM is supported for all currently VMware ESXi versions. For more information on the MEM, including installation and configuration instructions, refer to

Configuring and Installing the EqualLogic Multipathing Extension Module for VMware vSphere and PS Series SANs at http://en.community.dell.com/dell-groups/dtcmedia/m/mediagallery/19991633.

2.5.2

VMware Round Robin If MEM is not an option, then set the Path Selection Policy to use VMware Round Robin. This enables all of the physical NICs to participate in load balancing to the datastore. The VMware default path policy (Fixed) will not load balance I/O across all of the physical NICs but will instead use the additional paths only in the event of a failure. This could potentially prevent performance scaling past one NIC. In order for Round Robin to be configured as an option, the iSCSI connectivity must be properly configured to use more than one physical adapter. In addition, each iSCSI VMkernel port will be bound to only one of these physical adapters as discussed in the iSCSI connectivity technical reports. It is possible to change the path selection policy from Fixed to Round Robin in both the VMware vSphere Web Client and the VMware vSphere Client GUI by managing the path for each Datastore. This can become unwieldy because the existing and new Datastores have to be changed manually. In addition to changing the path selection policy to Round Robin, modify the number of I/Os that have to be sent before switching paths in order to make Round Robin function better. The default behavior is to wait until 1000 I/Os have been sent before switching to the next available path. This may not fully utilize

8

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

the entire available bandwidth to the SAN when multiple paths are available. Change this from 1000 I/Os per path to 3 I/Os per path. This script sets all EqualLogic volumes to Round Robin and set the IOPs value to 3. This script can be copied to the console or SSH session to make it easier. This script must be run on all ESXi nodes esxcli storage nmp satp set --default-psp=VMW_PSP_RR --satp=VMW_SATP_EQL ; for i in `esxcli storage nmp device list | grep EQLOGIC|awk '{print $7}'|sed 's/(//g'|sed 's/)//g'` ; do esxcli storage nmp device set -d $i --psp=VMW_PSP_RR ; esxcli storage nmp psp roundrobin deviceconfig set -d $i -I 3 -t iops ; done Use the next command to set the default so that new PS Series volumes inherit the proper setting. To be effective, this new rule requires a reboot of the ESX host. Until then, if you add more PS Series (EQL) volumes, re-run the script above on all ESX nodes. esxcli storage nmp satp rule add -s "VMW_SATP_EQL" -V "EQLOGIC" -M "100E-00" -P "VMW_PSP_RR" -O "iops=3" To verify that the correct settings are in effect, use: esxcli storage nmp device list The output should look similar to the following: naa.6090a098703e5059e3e2e483c401f002 Device Display Name: EQLOGIC iSCSI Disk (naa.6090a098703e5059e3e2e483c401f002) Storage Array Type: VMW_SATP_EQL Storage Array Type Device Config: SATP VMW_SATP_EQL does not support device configuration. Path Selection Policy: VMW_PSP_RR Path Selection Policy Device Config: {policy=iops,iops=3,bytes=10485760,useANO=0;lastPathIndex=3: NumIOsPending=0,numBytesPending=0} Note: the storage array type VMW_SATP_EQL is used to indicate the PS Series volume and that the Path Selection Policy is VMW_PSP_RR for Round Robin. In addition, the policy=iops,iops=3 verifies that the number of I/Os per path has been set to three.

2.5.3

Virtual Volume (VVOL)-related settings In general, following non-VVOL best practices is appropriate when introducing VVOLs. However, additional requirements must be adhered to when implementing VVOLs: 

The VMware vCenter and the Dell EqualLogic Virtual Storage Manager (VSM) virtual appliances: -

9

Must be installed on an HA-enabled vSphere cluster Must be installed on a non-VVOL shared storage datastore Must not be migrated (Storage vMotion) to a VVOL datastore

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

 

2.5.4

The vSphere cluster HA settings must be modified to set a higher starting priority to vCenter and VSM over other VMs. NOOPTimeout settings (as described in section 2.5.4.2)

Default iSCSI login and NOOP timeout values In addition to configuring Round Robin or MEM for multipathing, there is a recommendation for the host iSCSI login and NOOP timeout settings. These timeout values might be too low in large environments and prevent paths from reconnecting or potentially cause an all paths down (APD) event. By default, the MEM configuration script attempts to set Login timeout values to the recommended 60 seconds. However, at this time, NOOP timeout values must be modified manually.

Login timeout

2.5.4.1

The default timeout value is 5 seconds. Change this to 60 seconds on each host to allow more time for the iSCSI traffic to reach the host. Timeout is the maximum time that the host will wait to receive iSCSI traffic from the PS Series storage before giving up on the login process resulting in the login process staring over. vSphere Web Client: 1.

On the Home screen, click Hosts and Clusters and select a host by drilling down into the datacenter. 2. Click the Manage tab; in the Storage section, click Storage Adapters and select iSCSI Software Adapter from the list. 3. Under Adapter Details, select Advanced Options and click Edit. 4. Scroll down to LoginTimout, change the value to 60 and click OK. vSphere Client GUI: 1. On the Home screen of the vCenter GUI, click Hosts and Clusters and select a host. 2. Click the configuration tab, then Storage Adapters under Hardware, select the iSCSI Software Adapter and click Properties. 3. Click Advanced… and scroll down to the LoginTimout section. Change this value to 60 and click OK.

This can also be done in the CLI with the following command: esxcli iscsi adapter param set --adapter=vmhbax --key=LoginTimeout --value=60 (where x represents the appropriate vmhba number in the system)

2.5.4.2

NOOP timeout The default NOOP timeout value is 10 seconds. Change this to 30 seconds on each host to allow more time for iSCSI traffic to reach the host from the volume. While this is a best practice for every install, it is critically important when using VVOLs with multi-member volumes.

10

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

vSphere Web Client: 1.

On the Home screen, click Hosts and Clusters and select a host by drilling down into the datacenter. 2. Click the Manage tab; in the Storage section, click Storage Adapters and select iSCSI Software Adapter from the list. 3. Under Adapter Details, select Advanced Options and click Edit. 4. Scroll down to NOOPTimout, change the value to 30 and click OK. vSphere Client GUI: 1. On the Home screen of the vCenter GUI, click Hosts and Clusters and select a host. 2. Click the configuration tab, then Storage Adapters under Hardware, select the iSCSI Software Adapter and click Properties. 3. Click Advanced… and scroll down to the NoopTimout section. Change this value to 30 and click OK.

Figure 1

11

Timeout Values – vSphere Web Client

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

Figure 2

Timeout Values – vSphere Client GUI

This can also be done in the CLI with the following command: esxcli iscsi adapter param set --adapter=vmhbax --key=NoopOutTimeout --value=30 (where x represents the appropriate vmhba number in the system)

12

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

3

Host based performance

3.1

Delayed ACK Delayed ACK is a TCP/IP method of allowing segment acknowledgements to piggyback on each other or other data passed over a connection with the goal of reducing I/O overhead.

Figure 3

Example of Delayed ACK from Multivendor Post on using iSCSI with VMware

One side effect of delayed ACK is that if the pipeline is not filled, acknowledgement of the data is delayed. In SANHQ, this can be seen as higher latency during lower I/O periods. Latency is measured from the time the data is sent to when the acknowledgement is received. With disk I/O, any increase in latency can result in decreased performance. As more customers are moving to 10Gbe and using jumbo frames, this situation shows up more often when monitoring an environment with SANHQ. This VMware KB 1002598 has additional information, including instructions for disabling it on ESX 3.5 through ESXi 6.x. http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=10 02598

13

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

3.1.1

Disabling Delayed ACK For new installations, using the vSphere Web Client: 1.

From the Home screen, click Hosts and Clusters and then select a host by drilling down into the datacenter. 2. Click the Manage tab, Storage section, Storage Adapters, select iSCSI Software Adapter from the list. 3. This step differs depending on if you are setting the Delayed ACK property globally for the iSCSI initiator or for an individual discovery address. 

Setting the Delayed ACK property globally for the iSCSI initiator a. Under Adapter Details, select Advanced Options and click Edit. b. Scroll down to Delayed ACK. c. Deselect the checkbox and click OK



Setting the Delayed ACK property for an individual discovery address a. b. c. d. e.

14

Under Adapter Details, select Targets, Dynamic Discovery. Select the discovery address to modify and click Advanced. Scroll down to Delayed ACK. Deselect the checkbox for DelayedAck in the Inherited column. Deselect the checkbox for DelayedAck in the Value column and click OK.

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

4. Do this for each discovery address that needs to be modified.

For new installations, using the vSphere Client GUi: 1. From the Home screen, click Hosts and Clusters and then select a host. 2. Click the configuration tab, Storage Adapters, iSCSI Software Adapter and click Properties. 3. Setting the Delayed ACK property is different if it is done globally for the iSCSI initiator or for an individual discovery address. 

Setting the Delayed ACK property globally for the iSCSI initiator: a. Click Advanced… and scroll down to Delayed ACK. b. Deselect the checkbox and click OK

15

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3



Setting the Delayed ACK property for an individual discovery address: a. b. c. d.

Click the Dynamic Discovery tab, select the discovery address to modify and click Settings… Click Advanced… and scroll down to Delayed ACK. Deselect Inherit from parent. Deselect the checkbox for DelayedAck and click OK.

4. Do this for each discovery address that needs to be modified.

16

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

For existing connections, using the vSphere Web Client: Disabling Delayed ACK using the above procedure only disables Delayed ACK for new volumes. Any existing volumes must use the following procedure to correct the entries already stored in the iSCSI initiator database. 1. Put ESXi node into maintenance mode. 2. To verify Delayed ACK status at the ESXi shell, enter the following command Note: If you have Broadcom iSCSI HBAs installed, the Delayed ACK setting can be ignored if the HBAs are not being used. In this example, the first two entries represent the Broadcom HW iSCSI HBAs. # vmkiscsid --dump-db | grep Delayed iSCSI MASTER Database opened. (0xff889008) `node.conn[0].iscsi.DelayedAck`='1' `node.conn[0].iscsi.DelayedAck`='1' `node.conn[0].iscsi.DelayedAck`='0' `node.conn[0].iscsi.DelayedAck`='0' `node.conn[0].iscsi.DelayedAck`='0' `node.conn[0].iscsi.DelayedAck`='0' A value of “1” indicates that DelayedACK is enabled, “0” is disabled. 3. Using the vSphere Web Client (Alternate steps for using vSphere Client GUI are found below), click the Manage tab and in the Storage section click Storage Adapters; select iSCSI Software Adapter from the list. (If you are using a HW iSCSI adapter, you must repeat this procedure for each adapter in use.) 4. Under Adapter Details, select Targets. 5. In the Static Discovery listings, select the top entry and then hold down Shift while clicking the bottom entry. 6. Click Remove to delete all the selected entries. 7. In the Dynamic Discovery listing, remove any PS Series storage entries. 8. Reboot the node. 9. Return to Manage, Storage, Storage Adapters, iSCSI Software Adapter, Adapter Details, Targets and select Dynamic Discovery again. 10. Add the PS Series storage Discovery address, but do NOT rescan the adapter yet. 11. From the Dynamic Discovery listing, select the discovery address to modify and click Advanced. 12. Scroll down to Delayed ACK. 13. Deselect the checkbox for DelayedAck in the Inherited column. 14. Deselect the checkbox for DelayedAck in the Value column and click OK. Do this for each discovery address that needs to be modified. This is also a good time to verify that the Login Timeout is set to 60 and NOOP Timeout is set to 30. 15. Now you can rescan the Storage adapter(s) and verify that Delayed ACK is disabled, using the command from Step 2.

17

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

For existing connections, using the vSphere Client GUI: 1. Using the vSphere Client GUI, on the Configuration tab, in Storage Adapters, for iSCSI Software Adapter click Properties. (If you are using a HW iSCSI adapter, repeat this procedure for each adapter in use.) 2. In the Static Discovery tab, select the top entry and then while holding Shift click the bottom entry. 3. Click Remove to delete all the selected entries. 4. In the Dynamic Discovery tab, remove any EQL storage entries. 5. Reboot the node. 6. Return to Configuration > Storage Adapters > iSCSI adapter > Properties and select the Dynamic Discovery tab again. 7. Enter in the PS Series storage Discovery address, but do NOT rescan the adapter yet. 8. Click the Dynamic Discovery tab, select the discovery address to modify and click Settings… 9. Click Advanced… and scroll down to Delayed ACK. 10. Uncheck Inherit from parent. 11. Uncheck DelayedACK and click OK. Do this for each discovery address that needs to be modified. This is also a good time to verify that the Login Timeout is set to 60 and NOOP Timeout is set to 30. 12. Now you can rescan the Storage adapter(s) and verify that Delayed ACK is disabled, using the command from Step 2 on page 17.

3.2

Storage I/O control Storage I/O Control (SIOC) is a means of ensuring that the excessive storage I/O demands of a particular virtual machine disk (VMDK) do not negatively impact the storage I/O needs of other VMDKs residing on the same datastore. Because of the various load balancers in the PS Series storage, disable the SIOC feature on PS Series storage based Datastores. Otherwise, VMware vCenter might get false information and attempt to load balance at the same time causing unnecessary data movement. For more information about SIOC, refer to Dell PS Series Arrays: Advanced Storage Features in VMware vSphere at http://en.community.dell.com/dell-groups/dtcmedia/m/mediagallery/19861418.aspx.

18

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

19

Figure 4

Datastore details showing Hardware Acceleration supported, and SIOC disabled as shown in the vSphere Web Client

Figure 5

Datastore details showing Hardware Acceleration supported, and SIOC disabled as shown in the vSphere Client GUI

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

4

Datastore size and queue depth While administrators continually try to maintain an optimized data layout and performance, the size of the Datastore becomes a question. Because every environment is different, there is no hard-set number. However, you will find recommendations of 10-30 VMs per Datastore. A number of factors in this decision include speed of the disks, RAID type and intensity of the VMs. Having one VM per volume can pose an abundance of administrative overhead, while putting all VMs on a single datastore could cause a performance bottleneck. Determine the most beneficial compromise by monitoring the environment with SANHQ to find volumes that may be underperforming. In addition, monitor the queue depth to see if there are outstanding I/Os to a volume indicating that too many VMs may reside on that datastore. The current VMware maximum supported datastore is 64TB. The PS Series array can create a maximum volume size of 15TB.

20

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

5

Virtual machine considerations Apply best practices on a virtual machine to allow increased performance. One of these is changing the disk timeout values to 60 seconds. For Windows® guests, the VMware tools typically do this by default. For more information on the disk timeout setting recommendations, refer to the OS Initiator Guide in the PS Series Firmware download section of the support site at support.dell.com/equallogic (this site requires a login).

21

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

6

Data Drives There are four primary ways to connect data drives to VMs in a VMware and PS Series storage environment, assuming that the OS is running its C:\ or /root off of a VMFS datastore volume.    

6.1

VMDK on VMFS iSCSI in the guest (sometimes called direct connect, or storage direct) Raw Device Mapping (RDM) Virtual Volume (VVOL)

VMDK on VMFS A VMDK disk on a VMFS volume is the most commonly used practice for data drives. This scenario involves carving out additional space on a VMFS datastore and assigning it to a virtual machine. Advantages:      

6.1.1

Easy to configure: Easily add storage to a VM from free space in existing datastores or provision another datastore Viewable in vCenter: This keeps the administration overhead light, the data drive location is known and it is easier to integrate vCenter commands, such as cloning, templates and Storage vMotion Does not require storage team interaction: As long as the datastore has free space, data drives can be added to a VM without any storage interaction Allows for tiering of data drives to different VMFS volumes/pools based on workload: This allows a datastore on a 10k R50 volume and a database drive residing on a 15k R10 volume to be kept Uses vSphere MPIO to maintain the same performance benefits Able to leverage VSM to take hypervisor consistent snapshots of the VM and its associated data

VMDK on VMFS best practices When configuring a VMDK on VMFS, use the following best practices to maintain performance. Using vSphere Web Client: 1. Right-click on the virtual machine where the data drive needs to be added. 2. Select Edit Settings. 3. Under Virtual Hardware, New Device, select SCSI Controller from the list and click Add. As many as three additional SCSI controllers can be added across which new hard disks can be spread. 4. Under Virtual Hardware, New Device, select New Hard Disk from the list and click Add.

22

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

5.

For VMs that are supported, change the controller type from LSI to a Paravirtual SCSI adapter for better performance.

Note: Paravirtual SCSI adapter support and configuration information is located in the VMware Knowledge Base article 1010398. 6. Expand the details of the newly listed SCSI controller by clicking >. 7. In Change Type, select VMware Paravirtual from the list.

8. Expand the details of the newly listed hard disk by clicking >. 9. Select details, including desired capacity, Disk Provisioning method, and Location (datastore). 10. Change the Virtual Device Node to one of the new SCSI controllers.

23

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

11. Change the SCSI ID to the next available ID for that controller – shown in the example: SCSI(1:0). When adding multiple hard disk devices, spread them across the three added in SCSI controllers.

The Dell support staff has seen tremendous performance improvements by making these few changes. Note: Virtualized RDMs and VMDKs prior to ESXi v5.5 are limited to 2TB-512b in size in all versions of ESX. Using vSphere Client GUI: 1. 2. 3. 4. 5.

Click on the virtual machine where the data drive needs to be added. Click Edit Virtual Machine Settings. Under hardware choose Add > Hard Disk and click Next. Choose to create a new virtual disk and click Next. Choose the size of virtual disk to be added to the VM. Set the disk provisioning options and choose whether to keep it on the same VMFS Datastore as the VM or browse for another one. 6. The next screen has an important step that is often overlooked. Change the virtual SCSI adapter to be different from the one where the base OS drive is residing. 7. Change the Virtual Device Node to 1:0, 2:0 or 3:0 for each additional device past the OS drive sitting on 0:0. The Dell support staff has seen tremendous performance improvements by making these few changes.

24

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

This also adds a new SCSI Controller to the virtual machine. 8. For VMs that are supported, change the SCSI Controller Type from LSI to Paravirtual for better performance.

6.2

iSCSI in the guest VM You can take advantage of iSCSI in the guest VM by using the guest VM iSCSI software initiator. This also allows vMotion and all of the other tools to work. Advantages: 



25

Overcome 2TB Limits of VMFS/RDM in vSphere 4.1. If you are using large file systems and still on vSphere 4.1 this ability allows VMs to talk to a volume much larger than a 2TB volume or avoid using extents. Isolates the data drives for Data Protection/DR strategies. This technique also allows you to isolate your data drives to a volume that may have different replication and snapshot strategies than the volume that houses the parent VM.

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

 



Can be mounted on physical or other virtual machines. If you have a VM crash or just want to dismount and remount the data, you can easily do this through the iSCSI initiator. Uses the same best practices as physical environments. This technique is the same technique used for the physical environment. If you perform a P2V migration of a server that has iSCSI attached volumes, you can continue to run the VM with very little change to the server. Allows tiering of data drives to different volumes and pools based on workload. Since the data drive resides directly on a SAN volume, it can perform faster and can be placed on a different SAN tier from the VMs other virtual disks.

Disadvantages: 





26

It is not visible to vCenter. Because the volume is managed from the VM itself, vCenter will not see it in the storage tab and you will not see it connected to the VM when editing the properties. This can cause some additional management overhead. Requires storage team intervention. Since it is a brand new volume, you need to create the volume to be seen by the VM not the vSphere environment. This means installing and configuring the iSCSI software initiator, connecting to the SAN with proper pathing, configuring and formatting the volume to be seen by the VM. Needs to be considered for DR plans separate from VMs. Because the volume is not seen by vCenter or any of the VMware tools (like Site Recovery Manager), they need to be put into the recovery and protection plans according to their SLAs.

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

6.2.1

iSCSI in the guest VM best practices Configuring iSCSI in the guest VM is the same as configuring it for a physical server. Install the iSCSI initiator into the guest VM, connect to the SAN and format the presented volume. There are, however, some considerations to think about on the ESX servers. 1.

Normally the iSCSI network and Public LAN network are isolated so you first have to make sure that the VM can see the PS Series storage. 2. Verify the VM is configured properly to take advantage of MPIO inside the guest VM if you need it. 3. Modify the vSwitch that is delivering iSCSI traffic to include two iSCSI Guest Port Groups. In the diagram below, the iSCSI Guest Port Groups are iSCSI Guest 1 and iSCSI Guest 2. These are installed on the vSwitch that can communicate on the SAN network. (vSphere Client GUI steps are provided after the following vSphere Web Client steps below.)

4. Once the iSCSI Guest Port Groups are created, guarantee traffic across the physical NICs. Monitoring esxtop does not guarantee the traffic across all the physical NICs from a single VM; it has to be forced. Take advantage of the vSwitch NIC Teaming so that each port group is configured to use a single NIC, enabling alternate NICs to be used only for failover.

27

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

5.

Select one of the iSCSI Guest port groups and click the Edit icon.

6. In the Settings window, click Teaming and failover and select Override under Failover order. 7. Make one of the physical NICs active and set the other to standby (not unused). Do this for the other iSCSI guest and use the other adapter. This prevents the guest OS from detecting a link failure. Host MPIO algorithms typically pause I/O for a short time before declaring a path dead. Using a standby prevents this I/O delay. For the guest OS to manage this function, set the alternate NIC to unused.

28

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

8. The last thing to do is assign new Ethernet adapters to the VM and choose the iSCSI guest networks as the new networks. vSphere Client GUI Steps: 1.

Modify the vSwitch that is delivering iSCSI traffic to include two iSCSI Guest Port Groups (iSCSI Guest 1 and iSCSI Guest 2 in the diagram below). These are installed on the vSwitch that can communicate on the SAN network.

2. Once the iSCSI Guest Port Groups are created, guarantee traffic across the physical NICs. Monitoring esxtop does not guarantee the traffic across all the physical NICs from a single VM; it

29

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

has to be forced. Take advantage of the vSwitch NIC Teaming so that each port group is configured to use a single NIC, enabling alternate NICs to be used only for failover. 3. Inside the vSwitch, select the first iSCSI guest and click properties. 4. In the NIC Teaming tab, select Override vSwitch Failover order. 5. Make one of the physical NICs active and set the other to standby (not unused). Do this for the other iSCSI guest and use the other adapter. This prevents the guest OS from detecting a link failure. Host MPIO algorithms typically pause I/O for a short time before declaring a path dead. Using a standby prevents this I/O delay. For the guest OS to manage this function, set the alternate NIC to unused.

6. The last thing to do is assign new Ethernet adapters to the VM and choose the iSCSI guest networks as the new networks.

30

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

6.3

Raw Device Mapped LUNs Raw Device Mapped LUNs (RDMs) are used to isolate the data on volumes and retain the view inside vCenter. Advantages:     

Easy to configure: Each RDM is a single volume on the SAN Viewable in vCenter under the VMs to which it is attached: The volume is displayed inside vCenter under the VM and enables vCenter integration with software such as SRM Enables tiering of data drives to different volumes or pools based on workload: It is possible to isolate the data drive RDMs to another tier of storage for performance Uses vSphere MPIO for high bandwidth Apps: Continues to leverage the MEM or Round Robin for multipathing Isolates data from OS and enables the use of 3rd party applications which work with RDMs: There are some 3rd party applications that require some form of RDM for either backup or protection or tools

Disadvantages:   

No guest integration with Auto Snapshot Manager/Microsoft ® Edition: ASM/ME cannot recognize RDM volumes for Smart Copy operations No VM integration with VSM Smart Copies: VSM does not recognize RDM volumes for Smart Copy operations for VM snapshots Each ESXi server needs connectivity; connection counts need to be evaluated: Since each ESX server needs to see the RDM volume, there are multiple connections to the RDM even though only one server is using the volume at a time.

Configure RDMs with the same best practices as VMDK on VMFS with the new SCSI adapter and paravirtualized adapter if supported.

6.3.1

VM data drive formatting options Dell has found improvements in performance by using a 64k partition alignment. This is necessary in Windows Operating Systems before Windows 2008, Oracle® Solaris, and Red Hat® Enterprise Linux® guest OSs. Additionally, Microsoft® SQL, Exchange®, and other I/O intensive applications benefit from formatting the filesystem with a 64K cluster size. This format achieves a better alignment of I/Os with the 64K stripe size of the PS Series arrays.

7

Conclusion As more environments become virtualized, it is necessary to follow best practices to maintain not only the performance of these environments, but also the high availability. Dell recommends following these practices even if there is no apparent performance issue because it allows the environment to be in a known healthy state.

31

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

A

ESX 4.1 and ESX 5.0/5.1 information This appendix contains information specific to older versions of VMware vSphere/ESX and is no longer applicable to the versions covered in the main portion of this document.

A.1

Host based connectivity

A.1.1

PS Series storage heartbeat VMkernel port In the VMware virtual networking model, certain types of VMkernel network traffic are sent out on a default VMkernel port for each subnet. Typically, this is the lowest numbered VMkernel port on that subnet. The iSCSI multipathing network configuration requires that each iSCSI VMkernel ports be bound to a single physical NIC as an uplink. As a result, if the physical NIC that is being used as the uplink for the default VMkernel port goes down, network traffic using the default VMkernel port will fail. This includes vMotion traffic, SSH access, and ICMP ping replies. Although existing iSCSI connections are not directly affected by this condition, a side effect of the suppressed ping replies is that the PS Series storage group will not be able to accurately determine connectivity during the iSCSI login process, the initiator will attempt to reestablish the lost sessions. In some scenarios, depending on array, server and network load, logins may not be completed in a timely manner. This will result in not all paths to the volume being properly established or an all paths down (APD) event to occur. To prevent this from occurring, Dell recommends creating a highly available VMkernel port, or storage heartbeat VMkernel port, on the iSCSI subnet serving as the default VMkernel port for such outgoing traffic. Because this storage heartbeat has access to all of the available physical network cards, the ICMP ping replies will continue to work even if one of the physical network cards has lost connectivity.

Figure 6

Conceptual Image of iSCSI connectivity with storage heartbeat

ESX 5.0: The storage heartbeat is recommended for any ESX hosts running version 5.0. For more information on how to configure connectivity including the storage heartbeat, refer to technical report Configuring iSCSI Connectivity with VMware vSphere 5 and Dell EqualLogic PS Series storage at http://en.community.dell.com/dell-groups/dtcmedia/m/mediagallery/19997606.

32

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

ESX 4.x: The storage heartbeat is recommended for all ESX hosts running any version of 4.x software. For more information on how to configure connectivity including the storage heartbeat, refer to technical report Configuring VMware vSphere Software iSCSI With Dell EqualLogic PS Series storage at http://en.community.dell.com/dell-groups/dtcmedia/m/mediagallery/19861454. Note: If you are using jumbo frames, the storage heartbeat VMkernel port must also be configured for jumbo frames. Otherwise, proper negotiation of jumbo frames will likely not occur.

A.1.2

VMware Round Robin Note: In ESX/ESXi version 4.x the I/Os per path parameter default cannot be changed. This means that new volumes will not inherit the three I/Os per path setting. As you add Datastores and RDMs, you must go to each ESX host and reset the I/Os per path parameter. Once configured, the settings will persistent across reboots/startups. There are scripts available that automate this process. These can be copied onto the ESX server. As you add new volumes, including RDM volumes, re-run the script to insure that all the PS Series storage volumes are set. ESX 4.x: In VMware ESX 4, the following commands can be run to set the default policy for all Dell PS Series storage volumes to Round Robin. This also automatically sets new PS Series storage volumes to Round Robin. The I/Os per path parameter must still be set on these new Datastores when they are added. To configure the default policy for all new and future PS Series storage volumes to Round Robin, type in the following three commands and press [Enter] after each one: esxcli nmp satp setdefaultpsp –-satp VMW_SATP_EQL –-psp VMW_PSP_RR esxcli corestorage claimrule load esxcli corestorage claimrule run In order to change existing Datastore volumes to Round Robin the following script can be run: for i in `esxcli nmp device list | grep -i -B1 "ay Name: EQLOGIC" | grep -i "naa." | grep -i -v "ay Name"` ; do esxcli nmp device setpolicy --device $i -psp VMW_PSP_RR; done Once existing Datastores and all new Datastores have been configured for Round Robin, the following script must be executed to change the IOs per path to 3: for i in `esxcli nmp device list | grep -i -B1 "ay Name: EQLOGIC" | grep -i "naa." | grep -i -v "ay Name"` ; do esxcli nmp roundrobin setconfig --device $i --iops 3 --type iops; done Any new Datastores will automatically inherit the Round Robin setting but still need the I/Os per path script to change it back to 3. To verify the settings, type in the following command: esxcli nmp device list

33

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

The output should look similar to the following: naa.6090a098604ebfe262d9740100000064 Device Display Name: EQLOGIC iSCSI Disk (naa.6090a098604ebfe262d9740100000064) Storage Array Type: VMW_SATP_EQL Storage Array Type Device Config: SATP VMW_SATP_EQL does not support device configuration. Path Selection Policy: VMW_PSP_RR Path Selection Policy Device Config: {policy=iops,iops=3,bytes=10485760,useANO=0;lastPathIndex=0: NumIOsPending=0,numBytesPending=0} Working Paths: vmhba41:C1:T3:L0, vmhba41:C0:T3:L0 Use the Storage Array Type VMW_SATP_EQL to indicate the PS Series storage volume and how the Path Selection Policy is VMW_PSP_RR for Round Robin. In addition the policy=iops, iops=3 verifies that the number of I/Os per path has been changed from 1000 to 3. Note: If you later decide to install the Dell MEM, remove this claim rule. To do this, run the following command on each ESXi node. esxcli storage nmp satp rule remove -s VMW_SATP_EQL -V "EQLOGIC" -M "100E-00" -P "VMW_PSP_RR" -O "iops=3"

A.1.3

Default iSCSI timeout values ESX 5.0: The default timeout value in ESX 5.0 is 5 seconds. This field is not changeable until the ESX 5.0 server has been updated to patch ESXi500-201112001 or higher. Once this patch is installed, the field for LoginTimeout is editable and can be changed to 60 seconds from the Advanced options on the iSCSI Software Adapter using the vSphere Client GUI. ESX 4.1: The default timeout value in ESX 4.1 is 15 seconds. This option is not editable until applying update 3. Once the field can be modified, the LoginTimeout can be changed to 60 seconds from the Advanced options on the iSCSI Software Adapter using the vSphere Client GUI.

A.2

Datastore size and queue depth With ESX 4.1, the maximum volume size is 2TB-512b. The ESX servers will be unable to find and format a 2048GB volume. A 2TB volume that is created using the EQL GUI will produce a LUN size of 2048GB, which is slightly too large. Use a 2047GB LUN instead. A volume that is resized past 2047GB and then reread by ESX, no longer has an accessible Datastore or RDM until the volume size is returned to less than 2048GB. Should this happen, contact Dell Support to receive assistance with shrinking the volume while retaining data.

34

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

B

Technical support and customer service Dell support service is available to answer your questions about PS Series storage arrays.

B.1

Contacting Dell 1.

If you have an Express Service Code, have it ready. The code helps the Dell automated support telephone system direct your call more efficiently. 2. If you are a customer in the United States or Canada in need of technical support, call 1-800-945-3355. If not, go to Step 3. 3. Visit support.dell.com/equallogic. 4. Log in, or click “Create Account” to request a new support account. 5. At the top right, click “Contact Us,” and call the phone number or select the link for the type of support you need.

B.2

Warranty information The MODEL array warranty is included in the shipping box. For information about registering a warranty, visit http://support.dell.com/EqualLogic.

B.3

Related documentation To learn more about Dell PS Series storage products and new releases being planned, visit the Dell TechCenter site: http://delltechcenter.com/page/EqualLogic. Articles, demos, technical documentation and more details about the PS Series storage product family are available at http://en.community.dell.com/techcenter/storage/w/wiki/2660.equallogic-technical-content. For an updated Dell PS Series storage Compatibility Matrix visit the following URL: http://en.community.dell.com/techcenter/storage/w/wiki/2661.equallogic-compatibility-matrix. For detailed information about PS Series storage arrays, groups, volumes, array software, and host software, log in to the Documentation page at the customer support site. Table 2

35

Referenced documentation

Vendor

Document title

Dell

Dell Virtual Storage Manager 3.5: Installation Considerations and Datastore Manager

Dell

Virtual Machine Protection with Dell EqualLogic Virtual Storage Manager v3.5

Dell

Dell PS Series Arrays: Advanced Storage Features in VMware vSphere

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3

Vendor

Document title

Dell

Configuring and Installing the PS Series Multipathing Extension Module for VMware vSphere and PS Series Arrayss

Dell

Dell EqualLogic PS Arrays – Scalability and Growth in Virtual Environments

Dell

Configuring iSCSI Connectivity with VMware vSphere 5 and Dell EqualLogic PS Series storage

VMware

KB 1009524 – “Routing iSCSI Not Supported” A multivendor post on using iSCSI with VMware vSphere: http://virtualgeek.typepad.com/virtual_geek/2009/09/a-multivendor-post-onusing-iscsi-with-vmware-vsphere.html

36

Best Practices for Implementing VMware vSphere in a Dell PS Series Storage Environment | TR1091 | v1.3