How to configure event logging with Dynamic Multi-Pathing 5.1 SP1
Article ID: 100045842
Updated On:
Description
Description
This document includes the following topics:
About DMP event logging
The event source daemon (vxesd) is a Veritas Dynamic Multi-Pathing (DMP) component process that receives notifications of any device-related events that are used to take appropriate actions.
DMP notifies vxesd of major events, and vxesd logs the event in a log file (/etc/vx/dmpevents.log). These events include:
■ Enabling paths or DMP nodes
■ Disabling paths or DMP nodes
■ Throttling of paths
■ I/O error analysis
■ Host Bus Adapter (HBA) and Storage Area Network (SAN) events
The log file is located in /var/adm/vx/dmpevents.log but is symbolically linked to
/etc/vx/dmpevents.log. When the file reaches 10,000 lines, the log is rotated. That is, dmpevents.log is renamed dmpevents.log.X and a new dmpevents.log file is created.
Event log messages are categorized by log level, as follows:
Level 1 Critical DMP log messages.
Level 2
Messages that relate to I/O throttling, suspected path failures, repeated path failures, and DMP node migration.
Level 3 Messages that relate to I/O errors, I/O error analysis, and path media errors.
Level 4 Messages that relate to setting or changing attributes.
Greater than level 4
Messages available for troubleshooting by Symantec support.
By default, only level 1 event log messages are displayed. You can change the level of detail in the event log file using the tunable dmp_log_level.
See the vxdmpadm(1M) manual page.
DMP event log messages
Veritas Dynamic Multi-Pathing (DMP) records messages to the console and to a log file on certain events.
Table 1-1 lists the event log messages and their descriptions.
Table 1-1 DMP (Veritas Dynamic Multi-Pathing) Event Log Messages
|
Log Level |
Log Message |
Meaning |
Scenarios |
|
2 |
Removed Path pathname belonging to Dmpnode dmpnodename |
DMP deletes the path from the DMP node. |
■ A Logical Unit Number (LUN) is removed from an array. All of the paths to the LUN and the DMP node are removed from the DMP database. ■ A LUN is physically migrated from one array to another. DMP detaches the paths from one enclosure and attaches them to the other. |
|
2 |
I/O error occured on Path pathname belonging to Dmpnode dmpnodename |
I/O did not succeed on the path |
■ The path is not available physically due to errors such as an HBA controller failure, a failed LUN, or a network failure. ■ In an A/P (Active/Passive) array, the active processor is disabled. |
Table 1-1 DMP (Veritas Dynamic Multi-Pathing) Event Log Messages
(continued)
|
Log Level |
Log Message |
Meaning |
Scenarios |
|
3 |
Throttled Path pathname belonging to Dmpnode dmpnodename |
The number of I/Os simultaneously processed for the DMP node has exceeded the throttle parameters. |
■ If an array is slow to process concurrent I/Os, the number of I/Os may exceed the throttle level set for the DMP nodes. DMP throttling is controlled by the following tunable parameters: ■ The queuedepth tunable parameter indicates the maximum number of I/O requests for each DMP node. ■ The iotimeout tunable parameter indicates the maximum processing time for I/Os on each path. ■ DMP throttles the path if the pending I/Os exceed either of these parameters. To tune the throttle values, use the vxdmpadm setattr command to set the queuedepth tunable parameter or the iotimeout tunable parameter. |
|
3 |
Marked as idle Path pathname belonging to Dmpnode dmpnodename |
An in-use path has no I/O activity for 60 seconds |
■ The DMP kernel statistics daemon monitors each path’s I/O activities every 60 seconds. If the daemon does not detect I/O on a path since the last check, then the path is marked as idle. |
|
3 |
Enabled Path pathname belonging to Dmpnode dmpnodename |
DMP starts using the path for its operations. |
■ A path is enabled manually using the vxdmpadm enable command. ■ A new path is physically added due to the addition of an HBA or a LUN. The DMP device discovery code enables the path. ■ The DMP restore daemon enables a previously disabled path, if the inquiry and open succeeds for that path. |
Table 1-1 DMP (Veritas Dynamic Multi-Pathing) Event Log Messages
(continued)
|
Log Level |
Log Message |
Meaning |
Scenarios |
|
3 |
Disabled Path pathname belonging to Dmpnode dmpnodename |
DMP stops using the path for its operations. |
■ A path is disabled manually using the vxdmpadm enable command. ■ If DMP detects an error on a path such as IO failure, inquiry or open IOCTL failure, then the path is disabled. ■ When you disable a host bus adapter (HBA) or remove a LUN, DMP disables all of the paths through the HBA or LUN. |
|
3 |
SCSI error occurred on Path pathname: opcode=SCSI_operation_cCode reported transport failure (status=request_status key=sense_key, asc=additional_sense _code, ascq=additional_ sense_code_qualifier) |
DMP bypasses the SCSI strategy to issue SCSI commands to disk and the SCSI command fails with an error for the opcode |
■ SCSI commands can fail if the command cannot reach the disk due to a bus error, a network error, or a SCSI adapter hardware error. ■ DMP SCSI commands like inquiry, open or failover (change from Primary to Secondary controller in an A/P or A/A-A array ) can fail with this error message. ■ There are many types of SCSI errors. Transport failure is just one of them. For complete information about the SCSI errors, refer to the following URL: http://www.isobuster.com/help.php?help=305. Uee the values of the "key", "asc" and "ascq" in the message to look up the failure in the table. Note: For details on some typical errors, refer to the conv_scsi_tbl[] array in src/common/lib/vxvm/libvxvm/dmpapi.c |
|
3 |
Marked as suspect Path pathname belonging to Dmpnode dmpnodename |
DMP (vxesd daemon) has detected a change in the network configuration for the path |
■ Any change in the fabric connection between the host and the array can trigger this event. The vxesd daemon issues the VOLOP_DMP_RECONFIG_FABRIC event to notify the vxconfigd daemon that the fabric layout has been changed which initiates device discovery. |
Table 1-1 DMP (Veritas Dynamic Multi-Pathing) Event Log Messages
(continued)
|
Log Level |
Log Message |
Meaning |
Scenarios |
|
3 |
Initiated SAN topology discovery |
DMP (vxesd daemon) detects the attach or reattach of a SAN array, and initiates the device discovery |
■ The vxesd daemon uses HBA-specific libraries to detect any changes in the SAN, including the addition of a LUN, removal of a LUN, or the complete removal of the SAN. The vxesd daemon logs this message when the device discovery is started . |
|
3 |
Completed SAN topology discovery |
DMP (vxesd daemon) detects the attach or reattach of a SAN array, and completes the device discovery. |
■ The vxesd daemon uses HBA-specific libraries to detect any changes in the SAN, including the addition of a LUN, removal of a LUN, or the complete removal of the SAN. The vxesd daemon logs this message when the device discovery is completed. |
|
3 |
Failover initiated for Dmpnode dmpnodename without/with quiescing |
For A/P or A/A-A array, DMP re-directs the I/O from one storage controller to another for the DMP node |
■ The Active/Primary path on the host is manually disabled. ■ An I/O error, inquiry and open IOCTL fails on the Active/Primary path on the host. ■ The Active storage controller on the array is rebooted. |
|
3 |
Reconfiguration is in progress |
The vxconfigd daemon ( especially Device Discovery layer) begins the device discovery. |
■ Booting the system or invoking commands like vxdisk scandisks causes vxconfigd to initiate device discovery. ■ The VOL_NOTE_RECONFIG_START event is sent to the vxesd daemon to log this event |
Table 1-1 DMP (Veritas Dynamic Multi-Pathing) Event Log Messages
(continued)
|
Log Level |
Log Message |
Meaning |
Scenarios |
|
3 |
Reconfiguration has finished |
The vxconfigd daemon ( especially Device Discovery layer) completes the device discovery. |
■ Booting the system or invoking commands like vxdisk scandisks causes vxconfigd to initiate device discovery. ■ When device discovery is completed, the VOL_NOTE_RECONFIG_FINISH event is sent to the vxesd daemon to log this event |
|
3 |
SCSI error occurred on Path pathname: opcode=SCSI_operation_cCode reported reservation conflict (status=request_status key=sense_key, asc=additional_sense _code, ascq=additional_ sense_code_qualifier) |
DMP SCSI command for PGR operations such as read, reserve, register, or clear failed on the path for that opcode. |
■ DMP registers PGR keys on paths for Active/Active or Active/Passive cluster applications . If DMP fails to register the PGR keys, this message is logged. ■ DMP performs PGR operations for operations such as the vxdg init command, the vxdg import command, and failover ( I/O re-direction from Primary path to Secondary path). |
|
3 |
I/O retry(%u) on Path pathname belonging to Dmpnode dmpnodename |
DMP error analysis detects that the path is not faulty but I/O is not successful. DMP retries the I/O on the path |
■ If I/O fails due to a device reset or busy status of disk , DMP detects that the path is not at fault. DMP receives the action DMP_PATH_OKAY from its error analysis code path. DMP then retries the I/O on the path. The number of retries is controlled by the fixed retry tunable ( dmp_retry_count) or timebound tunable (iotimeout) which can be set using the vxdmpadm setattr command. |
Table 1-1 DMP (Veritas Dynamic Multi-Pathing) Event Log Messages
(continued)
|
Log Level |
Log Message |
Meaning |
Scenarios |
|
3 |
I/O analysis done as error on Path pathname belonging to Dmpnode dmpnodename |
DMP detects an I/O error on the path. DMP analyzes the error to determine the necessary action. |
■ DMP detects an I/O error if it could not send the I/O successfully to the underlying disks. This could be due to the LUN not available, network failure, or other errors. ■ Based on the I/O error, DMP analyzes the error to determine the necessary action. For the detailed analysis error code, refer to our source code in src/common/kernel/vxdmp/dmpdefs.h |
|
4 |
Unmarked as idle Path pathname belonging to Dmpnode dmpnodename |
When an I/O is sent to an idle path (path that has no I/O activity), DMP removes the idle state from the path. |
■ The DMP kernel statistics daemon monitors each path’s I/O activities every 60 seconds. If there are I/Os on idle paths, then DMP removes the idle state from that path. |
Issue/Introduction
