Error Message

Also the output of ps -elf shows the gablogd is in a D state.
 

# ps -elf|grep gablogd
1 D root      5266     1  0  85   0 -  2543 gab_li Jan22 ?        00:00:00 /opt/VRTSgab/gablogd    <<<
0 S root     31669 31542  0  78   0 -  1507 pipe_w 11:30 pts/1    00:00:00 grep gablogd

# lsof | grep 5266
gablogd    5266     root  cwd       DIR              253,3      4096     164321 /var/log/gab_ffdc
gablogd    5266     root  rtd       DIR              253,0      4096          2 /
gablogd    5266     root  txt       REG              253,0     14764     394693 /opt/VRTSgab/gablogd
gablogd    5266     root  mem       REG              253,0    129832    1705181 /lib/ld-2.5.so
gablogd    5266     root  mem       REG              253,0   1689388    1706744 /lib/libc-2.5.so
gablogd    5266     root  mem       REG              253,0    125177     478921 /usr/lib/libgab.so
gablogd    5266     root  mem       REG              253,0    216544    1704004 /lib/libm-2.5.so
gablogd    5266     root  mem       REG              253,3   8388608     164044 /var/log/gab_ffdc/gablog.1
gablogd    5266     root    0u      CHR                1,3                 2120 /dev/null
gablogd    5266     root    1u      CHR                1,3                 2120 /dev/null
gablogd    5266     root    2u      CHR                1,3                 2120 /dev/null
gablogd    5266     root    3w      REG              253,3       944     164322 /var/log/gab_ffdc/glgd_A.log
gablogd    5266     root    4u      CHR              250,0                10167 /dev/gab_0

#  cat /var/log/gab_ffdc/glgd_A.log
Fri Jan  7 10:13:49 2011:[25172]: GAB INFO V-15-1-26004 Keeping 20 log files of 8388608 bytes each in |/var/log/gab_ffdc| directory. Daemon  log size limit 8388608 bytes
Sat Jan  8 08:04:16 2011:[25174]: GAB INFO V-15-1-26017 Exiting on request from GAB driver
Sat Jan  8 08:09:03 2011:[5235]: GAB INFO V-15-1-26004 Keeping 20 log files of 8388608 bytes each in |/var/log/gab_ffdc| directory. Daemon log size limit 8388608 bytes
Sat Jan 15 08:03:34 2011:[5237]: GAB INFO V-15-1-26017 Exiting on request from GAB driver
Sat Jan 15 08:08:13 2011:[5262]: GAB INFO V-15-1-26004 Keeping 20 log files of 8388608 bytes each in |/var/log/gab_ffdc| directory. Daemonlog size limit 8388608 bytes
Sat Jan 22 08:03:29 2011:[5264]: GAB INFO V-15-1-26017 Exiting on request from GAB driver
Sat Jan 22 08:08:07 2011:[5264]: GAB INFO V-15-1-26004 Keeping 20 log files of 8388608 bytes each in |/var/log/gab_ffdc| directory. Daemonlog size limit 8388608 bytes

From e2139883:
----------------------------------------------------------------------
These messages are thrown due to gablogd sleeping in an ioctl context.
The sleep, by design is unbounded, since it will be woken up only in
critical events like iofence or death of the master of any GAB port.
 

What is gablogd - GAB logging daemon?

 

GAB implements a distributed network protocol. For situations when GAB decides to take the drastic action of killing its userland client process or panicking a node to resolve an issue, data from the affected node alone may not suffice for a meaningful support analysis. The new gablogd daemon attempts to address this issue. GAB starts this daemon by default at GAB configuration time.

 

If GAB encounters some problem, then First Failure Data Capture (FFDC) logs are also generated and dumped along with other core dumps and stack traces.

 

When you have configured GAB, GAB also starts a GAB logging daemon (/opt/VRTSgab/gablogd). GAB logging daemon is enabled by default. You can change the value of the GAB tunable parameter gab_ibuf_count to disable the GAB logging daemon.

 

To disable gablogd, you can add the following line in /etc/sysconfig/gab:

 

gab_ibuf_count=0

 

On reboot, gablogd will not be started.

 

On a running system, you could run the command ‘gabconfig –K’ (uppercase ‘K’) to disable this.

1) From e2139883:

Workaround: As the operating system message indicates, set the following:
echo 0 > /proc/sys/kernel/hung_task_timeout_secs

2) Turn gablogd off by setting gab_ibuf_count to 0 in /etc/sysconfig/gab.

See VCS Admin Guide for 5.1SP1, starting at the bottom of page 616.
gab_ibuf_count is mentioned explicitly in the table on page 618.

--Add this line to /etc/sysconfig/gab:

GAB_IBUF_COUNT=0

and either reload the gab module or reboot the system.

# cat /etc/sysconfig/gab
# This file is sourced from /sbin/init.d/gab.
#
# Set the two environment variables below as follows:
#
#       1 = start or stop gab
#       0 = do not start or stop gab
#
GAB_START=1
GAB_STOP=1
GAB_IBUF_COUNT=0    <<<<<<

--Verification for D state of gablogd:

# ps -elf | grep gablog
0 S root      7147  7029  0  78   0 - 15284 pipe_w 10:51 pts/0    00:00:00 grep gablog

 

 

Applies To

VCS 5.1SP1 running on RHEL 5 Update 5.

On RHEL5 Update 5 or later, the gablogd daemon prints informational and trace messages similar to the following on console, or in /var/log/dmesg:
  INFO: task gablogd:3423 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
gablogd       D ffff81007c65e9a0     0  3423      1          3601  3420 (NOTLB)
ffff810066d81e38 0000000000000082 0000000000000000 00003ffffffff000
ffff810066d81de8 0000000000000001 ffff810069126040 ffff810072bb70c0
00000018c423b106 0000000000034354 ffff810069126228 00000000f76f8004
Call Trace:
[<ffffffff8005ede9>] error_exit+0x0/0x84
[<ffffffff8850e78a>] :gab:gab_linux_sv_wait+0x53/0x68
[<ffffffff8008e16d>] default_wake_function+0x0/0xe
[<ffffffff884f8538>] :gab:gab_daemonlog+0xb1d/0xc8e
[<ffffffff8850e36c>] :gab:gab_linux_ioctl+0x10e/0x1a3
[<ffffffff8850e41d>] :gab:gab_linux_compat_ioctl+0x1c/0x20
[<ffffffff800fb8d4>] compat_sys_ioctl+0xc5/0x2b2
[<ffffffff8006249d>] sysenter_do_call+0x1e/0x76

ETrack: 2139883