How to collect data related to usage of VxVM, VxFS, & VCS in general
Article ID: 100005983
Updated On:
Description
Description
List of the steps for collecting data in the perspective of VxVM, VxFS, & VCS in general.
1) Collect date and time as soon as problem is observed.
2) Check what all operations are going on when performance degradation is observed, check if VERITAS operations are going on (look for VxVM/VxFS/VCS command if running).
3) Collect all the process currently running.
# time ps -elf
4) Collect kernel threadlist every 2 minutes, when performance degradation is observed. Threadlist will be logged into /var/log/messages file.
To avoid a system hang (it can cause problems with VCS/CFS) while collecting kernel threadlist using the Linux sysrq interface follow these steps.
# cat /proc/sys/kernel/printk > /tmp/printk
# echo 0 > /proc/sys/kernel/printk
# echo t > /proc/sysrq-trigger
# cat /tmp/printk > /proc/sys/kernel/printk
5) Collect top, meminfo and vmstat command output at regular interval (top 1 sec, vmstat 5 sec).
In one session watch continuously the top 20 process:
# watch -n 1 "top -b -d 1 | head -n 20"
# top –b –d 1 /* 1 sec interval */
If interested to look at any specific process (like vxconfigd) then use the following command
# top -b -d 5 -p `ps -ef |grep vxconfigd |grep -v grep|awk '{print $2}'`
# vmstat 5 /* 5 seconds interval */
# while true
> do
> date
> cat /proc/meminfo | egrep 'MemTotal|MemFree|Buffers|Dirty'
> sleep 2
> done
6) Collect VxVM i/o statistics if there is an application i/o load.
# vxstat -i 2 –vps
7) Collect the output from the following commands at regular intervals (10 seconds) in case if any VxVM command is taking time.
# pmap `ps aux | grep vxconfigd | grep -v grep | awk '{print $2}'` | grep -i total
# cat /proc/`pgrep vxconfigd`/status
# /usr/sbin/lsof -a -p `pgrep vxconfigd`
# pstack `pgrep vxconfigd`
Collect the stack of vxconfigd using 'gdb' if pstack command doesn’t work.
# gdb -p `pgrep vxconfigd`
(gdb) bt
(gdb) info threads
(gdb) info sharedlibrary
(gdb) info dcache
(gdb) quit
8) Please run the following command only if any VxVM commands are not working as expected or any VxVM/vxconfigd related issue is observed. Running vxconfigd in debug mode at
level 9 can slow down the vxconfigd.
How to start vxconfigd in debug mode at various labels (min 1, max 9)
# vxdctl debug
Once done, disable the logging using following command.
# vxdctl debug 0
Generate a gcore of vxconfigd when high memory usage is observed:
# gcore -o /usr/tmp/vxconfigd.core `pgrep vxconfigd`
VxFS specific commands (9,10) :
--------------------------------
9) Run the following command on VxFS mount points and get the "statfile" for each mount point.
# vxfsstat -w "statfile" -t 10 -c 2 /mount_point
10) If you experience any performance issues with a VxFS files system, please collect the metasave of the VxFS filesystem in question.
# cd /opt/VRTSspt/FS/MetaSave/
# ls
metasave_rhel4_x86 metasave_sles9_x86 README.metasave metasave_rhel4_x86_64 metasave_sles9_x86_64 Please follow the steps in README.metasave.
---------------------------------------------------------------------------
11) Please collect the output of FirstLook
Please see the article for FirstLook (Please skip downloading it):
To execute FirstLook, invoke start_look. For example, executing FirstLook on Solaris:
# cd /opt/VRTSspt/FirstLook
# ./start_look -l
Note:
1) Always use the -l option to get lockstat data unless system is VERY heavily loaded.
2) Note - do not run more than one hour, the logs will overwrite after 60 minutes.
3) Try to start the script 5 minutes before the performance issue is expected to be seen.
4) For example, files will be saved in /opt/VRTSspt/FirstLook/Flook_Apr_8_11:58:00_logs.
Optionally, it is possible to confirm that the FirstLook process is running:
# ps -ef | grep look
root 19984 1 0 08:58:19 pts/1 0:00 /bin/ksh /vx_install/FirstLook/Flook_sol After enough time to collect useful information, or if the performance issue is intermittent and has stopped, stop FirstLook:
# ./stop_look
Please compress the output using gzip, e.g., '/opt/VRTSspt/FirstLook/Flook_04_08_11_12:00:07_logs.tar'
12) Please collect the output of new version of VRTSexplorer from the affected node, in question:
# cd /opt/VRTSspt/VRTSexplorer
# ./VRTSexplorer
13) Please collect the output of VRTSexplorer from all other nodes in the VCS cluster.
14) Collect system dump from all nodes in the cluster around the same time. Please set peerinact timeout as shown in the article 100029707.
Issue/Introduction
How to collect data related to usage of VxVM, VxFS, & VCS in general
Was this article helpful?
Yes
No
Powered by
