During LUN addition or removal, the following sequence of commands are executed by user/admin:

1. cfgadm configure command is run

When cfgadm is executed and completes, indicating device state changes, either additions or removals, syseventd notifies vxesd. Upon receiving notification from systeventd, vxesd will trigger an automatic device discovery.

     2.  powermt commands are run by user/admin.

Powerpath performs device discovery and creates TPD metanodes.

     3.   vxdctl enable or vxdisk scandisks command is run by user/admin

DMP will perform device discovery and claim the device correctly as being managed by powerpath.

The problem occurs when vxesd triggers an automatic device discovery while PowerPath has not yet discovered or is in process of discovering newly added devices. A small window of opportunity exists where some paths of a disk are claimed as managed by powerpath (powermt commands followed by vxdctl enable) and some paths are claimed as being managed by DMP (vxesd triggered automatic discovery). This is not the expected sequence of events and thus leads to an inconsistent DMP database and could lead to a potential system panic.

Workarounds:

Veritas recommends that vxesd be permanently disabled in configurations where Powerpath is the multipathing driver of choice.

Check if vxesd daemon is running on the system:
# ps -eaf | grep -i vxesd
    root   624     1   0   Jun 24 ?           3:16 /sbin/vxesd
#

From command line - disable vxesd prior to LUN addition or removal using command:
# vxddladm stop eventsource
# ps -eaf | grep -i vxesd
#

To permanently disable vxesd and make the change persistent across reboot - comment out the pertinent entry in VxVM startup script. Edit the vxvm-startup file, and comment out the following line:
/usr/sbin/vxddladm start eventsource


Depending on the OS/version, the file will be in different locations:

Solaris 10: /lib/svc/method/vxvm-startup2
Solaris 8,9: /etc/init.d/vxvm-startup2
 
Formal Resolution:

Veritas will be implementing the following code change and enhancement in upcoming releases.

1. Code modification will result in vxesd being automatically disabled when Powerpath is detected.

This will be available in version 5.1RP2

Etrack: 2109397: Need to disable vxesd when PP is present on the system

      2.  vxesd behavior will be enhanced to operate in limited feature mode when Powerpath is present.

This feature will be implemented in 5.1SP1

Etrack: 2016129 : DDL: Have a tunable to disable OS event monitoring by vxesd

 

Applies To

 

Solaris 8,9 and 10 (sparc and X64 platforms)

VxVM 4.0, 4.1, 5.0 and 5.1 versions

EMC array and multipathing managed by EMC powerpath.

Background:

Volume Manager’s (VxVM) event source daemon (vxesd) is a process first introduced in VxVM 4.0 on Solaris and it receives notifications of device related events which are used to take appropriate actions. It registers for events from the OS SCSI driver via the syseventd framework available on Solaris 8 and above. When a new device arrives and the OS SCSI driver configures it into the system, the SCSI driver puts an event in the syseventd daemon's queue. All registrants for the event, including vxesd would be notified either via a callback mechanism or by running a specific script. In case of vxesd, the file /etc/sysevent/config/EC_devfs/sysevent.conf contains the registration for the vxesd's script "/etc/vx/es_scripts/es_devfs.pl". The script communicates to vxesd the physical path of the device that has arrived. The vxesd daemon waits for a few seconds to pull in all the events and initiate DDL's (Device Discovery layer) automatic device discovery to add the new device into DDL/DMP database. vxesd helps in logging the DMP related activities in /etc/vx/dmpevents.log and this file could be useful for debugging any DMP related issues.
In an environment where DMP handles the multipathing of the devices, the expected sequence of events while executing DR operations are:

   1. The cfgadm module attaches the given device to the system's device heirarchy and raises an event to denote the arrival of a new device.
   2. The event gets queued into syseventd daemon's queue which inturn invokes all the scripts that have registered for the event. The event argument gives the physical path of the device.
   3. Incase of VxVM, the vxesd daemon registers the script es_devfs.pl with syseventd daemon for device arrival events. So this script is executed in response to the device arrival event from the DR operation.
   4. The script establishes a socket connection with vxesd and transfers the event parameter to it.
   5. The vxesd daemon, inturn connects to vxconfigd to initiate DDL device discovery for the device that has arrived. Array-specific ASLs such as libvxemc.so or libvxCLARiiON.so claims the newly added device.

•   An important note is that the whole operation takes place *asynchronously* and cfgadm returns to shell right after adding the event to the syseventd's queue.
•   When EMC PowerPath (TPD – Third Party Driver) is doing the multipathing for the EMC array, Volume Manager’s DMP(Dynamic Multi-pathing) does not perform multipathing for those arrays.

If EMC PowerPath is performing the multipathing of the devices, the correct sequence of events are:

   1. The cfgadm module attaches the given device to the system's device heirarchy.
   2. EMC powerpath discovers the new devices when powermt commands are run. Powerpath provides the TPD metanodes.
   3.  vxdctl enable or vxdisk scandisks is run to initiate VxVM device discovery.  
   4. These TPD metanodes are further discovered by DMP's TPD ASL (Array Support Library) such as libvxpp.so. Once the device is claimed by libvxpp.so, it will be ignored by array-specific ASLs, resulting in powerpath properly managing the multipathing for the device and a successful device discovery operation.