Performance issues are reported on Storage Array controllers/directors/ports connected to Windows Hosts caused by SCSI 0x5e commands

book

Article ID: 100005530

calendar_today

Updated On:

Cause

This can be caused by a large number of 0x5E SCSI IOCTLs coming from Windows Cluster nodes with SFW.  This can lead to reduced performance on other Hosts and increased processor usage levels on Storage Array Controllers/Directors/Ports.

In Clusters, SFW uses a Challenger and Defender protocol to verify reservations stay on disks to avoid data corruption via split brain. Therefore Storage Foundation maintains a SCSI reservation thread within the Vxio driver that checks the SCSI reservation of imported disks every 3 seconds using a 0x5E IOCTL.

Why is SCSI command 0x5E used?


SCSI 0x5e (0x5E = SCSIOP_PERSISTENT_RESERVE_IN) is used to verify that the host still has a persistent reservation on the disk. The frequency of these commands are: (total time of perform SCSI reads (read reservation, read keys) for all disks) + 3 seconds.

Note - The frequency of this operation is not configurable.

Specifically for SCSI-3, Vxio sends SCSI-3 threads to every disk in the following manner:

  1. Read the SCSI-3 reservation key: Opcode (0x5e, Service Action 01)
  2. If the read SCSI-3 reservation key returns success, Vxio will issue a read SCSI-3 keys (0x5e, Service Action: 0)

Resolution

  1. Avoid overloading a single node
    Every Cluster Dynamic Disk Group maintains one SCSI reservation thread. Performance degradation might show up when all Cluster Dynamic Disk Groups are online on the same cluster node. Simply switching some of the Cluster Disk resources over to other nodes can reduce contention on a set of paths as well as reducing the on node load.
     
  2. Reduce the number of LUNs (Disks)
    Even though usage of smaller LUNs is recommended by several storage vendors for performance reasons; it is necessary to find a optimized balance between IO performance and IOCTL overhead.
  1. Work with Storage Array Vendor to optimize IOCTL handling
    Each Storage Array vendor has specific best practice guides for Windows clusters.  Veritas recommends checking and applying these settings where appropriate.
    For example: It is not Best Practice to have a mixture of Windows, UNIX and ESX/VSphere on the same FA Ports for an EMC DMX Array; however, an EMC VMax Array allows this type of configuration.
    Please contact your storage vendor for further details.

Issue/Introduction

Customers with Storage Foundation for Windows (SFW) or Storage Foundation High Availability for Windows (SFWHA) clusters that use a large number of LUNs can suffer from performance degradation at the Storage Array level.