Performance issue after Oracle Database upgrade from 11 to 19c - read and write operations are taking longer due to Veritas ODM not linked
Article ID: 100052946
Updated On:
Description
Error Message
When running commands: "ps -aef | grep dbw" & "lsof -p
Linux based example
The following VxVM devices are reporting issues: VxVM22000, VxVM10000, VxVM4000 & VxVM19000 corresponds to the impacted Veritas File Systems (VxFS).
The following lsblk syntax is a useful tool on Linux to display related information in a concise view:
# /bin/lsblk -o 'NAME,KNAME,MAJ:MIN,FSTYPE,LABEL,RO,RM,MODEL,SIZE,OWNER,GROUP,MODE,ALIGNMENT,MIN-IO,OPT-IO,PHY-SEC,LOG-SEC,ROTA,SCHED,MOUNTPOINT,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO'
NAME KNAME MAJ:MIN FSTYPE LABEL RO RM MODEL SIZE OWNER GROUP MODE ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED MOUNTPOINT DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO
VxVM4000 VxVM4000 199:4000 0 0 4.3T brw-rw---- 0 512 0 512 512 1 bfq /oracle/PIN/sapdata1 0 0B 0B 0
VxVM10000 VxVM10000 199:10000 0 0 4.6T brw-rw---- 0 512 0 512 512 1 bfq /oracle/PIN/sapdata2 0 0B 0B 0
VxVM19000 VxVM19000 199:19000 0 0 5T brw-rw---- 0 512 0 512 512 1 bfq /oracle/PIN/sapdata3 0 0B 0B 0
VxVM22000 VxVM22000 199:22000 0 0 5.1T brw-rw---- 0 512 0 512 512 1 bfq /oracle/PIN/sapdata4 0 0B 0B 0
VxVM22000, VxVM10000, VxVM4000 and VxVM19000 are consistently highly utilized and high wait times.
# /usr/bin/iostat -x 1 4
...
Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
VxVM22000 1435.00 44.00 11552.00 1368.50 0.00 0.00 0.00 0.00 10.66 24.32 15.15 8.05 31.10 0.55 81.20
VxVM10000 866.00 22.00 7032.00 216.00 0.00 0.00 0.00 0.00 4.45 0.77 3.18 8.12 9.82 0.87 77.20
VxVM4000 1378.00 46.00 33552.00 1272.00 0.00 0.00 0.00 0.00 31.50 70.15 45.02 24.35 27.65 0.65 93.20
VxVM19000 1841.00 42.00 14784.00 665.50 0.00 0.00 0.00 0.00 5.70 36.21 10.52 8.03 15.85 0.51 96.80
VxVM22000 1299.00 70.00 10456.00 2200.00 0.00 0.00 0.00 0.00 2.72 3.90 2.74 8.05 31.43 0.60 82.80
VxVM10000 1064.00 41.00 8640.00 481.00 0.00 0.00 0.00 0.00 5.04 2.00 4.55 8.12 11.73 0.70 76.80
VxVM4000 2222.00 69.00 42696.00 1153.00 0.00 0.00 0.00 0.00 13.70 63.41 32.93 19.22 16.71 0.43 97.60
VxVM19000 1443.00 59.00 11584.00 1266.00 0.00 0.00 0.00 0.00 28.34 37.03 41.52 8.03 21.46 0.63 94.80
VxVM22000 1411.00 31.00 11296.00 1551.00 0.00 0.00 0.00 0.00 4.13 1.68 4.80 8.01 50.03 0.58 84.00
VxVM10000 1326.00 19.00 10808.00 308.00 0.00 0.00 0.00 0.00 28.01 8.74 35.77 8.15 16.21 0.72 96.80
VxVM4000 1922.00 26.00 45824.00 716.00 0.00 0.00 0.00 0.00 3.15 5.62 4.69 23.84 27.54 0.49 94.80
VxVM19000 2291.00 30.00 18352.00 356.00 0.00 0.00 0.00 0.00 18.29 38.20 40.88 8.01 11.87 0.40 93.60
Cause
Without the Veritas ODM library correctly linked to the Oracle Database library, the system may show longer read & write operations.
Verify the ODM linking steps were done properly and in all the cluster nodes.
Oracle User:
$ ls -l $ORACLE_HOME/rdbms/lib/odm/libodm19.so
The environment may report unexpected events for:
read & write operations
vxi_flush_throttle reporting higher values than before
Volume Manager (VxVM) reporting write delays
NOTE: When Veritas ODM is used with Oracle 19c it requires Oracle patch 30404521.
Oracle Patch: 30404521 is included in the Oracle 19.14 version.
Resolution
Confirm the ODM linking has been done correctly following InfoScale & Oracle Database upgrades.
Enable & relink Veritas ODM (libodm.so)
Note: In 19c, create a new link inside
For example, in Oracle 19c:
1. Log in as RDBMS Home Owner OS user.
2. Shut down the database.
3. Change directories:
$ cd $ORACLE_HOME/rdbms/lib
4. Take a copy of the existing original ODM library.
$ mv $ORACLE_HOME/rdbms/lib/odm/libodm[software_version].so $ORACLE_HOME/rdbms/lib/odm/libodm[software_version].so.org
5. Check the odm directory exists, if not create it:
$ mkdir –p odm
$ cd odm
6.] In 19c, create a new link inside $ORACLE_HOME/rdbms/lib/odm/libodm19.so linked with /opt/VRTSodm/lib64/libodm.so.
For example, in 19c
$ ln -s /opt/VRTSodm/lib64/libodm.so $ORACLE_HOME/rdbms/lib/odm/libodm19.so
$ ls -l $ORACLE_HOME/rdbms/lib/odm/libodm19.so
NOTE: No changes are required for the default link present inside /lib/libodm19.so, which points to $ORACLE_HOME/lib/libodmd19.so.
7. Start the database and check the alert.log for the below message:
In the Oracle alert log we should see something like
Sample message
Oracle instance running with ODM: Veritas 7.4.2.1800 ODM Library, Version 2.0
Once Veritas ODM had been configured correctly, the Oracle Database performance issue was addressed and normal service was resumed.
Issue/Introduction
The system is reporting performance issues with the Oracle Database upgraded to 19c. Read and write operations are now taking dramatically longer than before the Oracle Database & Veritas InfoScale upgrade.
Veritas ODM (Oracle Disk Manager) was used prior to upgrading the Oracle Database & Veritas InfoScale software.
