How to install and configure Storage Foundation RAC 6.0.1 for Oracle RAC 11gR2 (11.2.0.3)

book

Article ID: 100010041

calendar_today

Updated On:

Description

Starting with Oracle RAC 11gR2 (11.2.0.2 and 11.2.0.3), Oracle has changed a bit internally, e.g. introducing some new features like HAIP, which makes part of Veritas Storage Foundation RAC (SFRAC) 6.0.1 installation guide impractical, especially the installer script of SFRAC 6.0.1 is oriented for Oracle 10g and 11g, but nowadays more customers are beginning to install the latest Oracle version 11gR2 (specifically 11.2.0.3).

This article attempts to provide a concise end-to-end procedure for how to install & configure SFRAC 6.0.1 for Oracle RAC 11gR2 (11.2.0.3), and can compliment the document below:

- Veritas Storage Foundation for Oracle RAC Installation and Configuration Guide (November 2012)

https://sort.veritas.com/DocPortal/pdf/ka6j0000000BNlEAAW

- Configuring interconnect redundancy with Oracle RAC 11.2.0.2 or later

Note: This article is for demo purposes only, Veritas recommends customers engage our consulting service for complex and customized configurations.

The architecture and configuration of SFRAC demonstrated in this example will be:

Using NIC bonding as to provide redundancy for both Oracle RAC and SFRAC(VCS) private link, that is Oracle Clusterware interconnect over SFRAC LLT link
Oracle RAC OCR and Voting disks are mirrored on Veritas CFS
Oracle RAC binaries are installed locally to each node
Oracle RAC database datafile and recovery data are located on Veritas CFS
Using 3 LUNs as SFRAC I/O fencing to take care of split-brain before Oracle RAC Clusterware
Using Veritas ODM with CFS for better performance
Using CSSD agent in SFRAC mandatory to handle inter-dependencies with Oracle RAC

Basically the installation & configuration of SFRAC 6.0.1 on Oracle RAC 11.2.0.3 includes 12 steps:

1. Configuring the interfaces and networking
2. Installing SFRAC
3. Configuring SFRAC
4. Completing Oracle RAC installation prerequisites
5. Preparing to install Oracle RAC
6. Installing and configuring Oracle RAC Grid Infrastructure (Clusterware)
7. Installing Oracle RAC database
8. Using SFRAC to perform Oracle RAC post-installation tasks (configure OSSD and relink libraries)
9. Creating the database storage on SFRAC CFS
10. Creating the Oracle RAC database via DBCA
11. Configuring SFRAC to manage Oracle database
12. Verifying the installation and configuration

Here's the sample highlight for end-to-end procedure:

1. Configure the interfaces and networking

Configuring NIC bonding as both Oracle RAC and SFRAC's private network link

According to the exact setup of network switch and OS, set attributes of bonded NIC correctly; If necessary, engage OS vendor and Oracle to verify what best value it should be. One possible setting could be:

BONDING_OPTS="mode=4 miimon=1000"

Following Oracle instructions to setup DNS and local /etc/hosts file

2. Installing SFRAC

Using one of the SFRAC installer scripts either "installer" or "installsfrac" to install SFRAC

# ./installsfrac

Veritas Storage Foundation for Oracle RAC 6.0.1 Install Program

1) Install minimal required rpms - 536 MB required 2) Install recommended rpms - 774 MB required 3) Install all rpms - 798 MB required 4) Display rpms to be installed for each option

Select the rpms to be installed on all systems? [1-4,q,?] (2) 3 ... Would you like to configure SF Oracle RAC on rhel63-01 rhel63-02? [y,n,q] (n)

Veritas Storage Foundation for Oracle RAC cannot be started without configuration.

Run the '/opt/VRTS/install/installsfrac601 -configure' command when you are ready to configure Veritas Storage Foundation for Oracle RAC. ... Would you like to view the summary file? [y,n,q] (n)

3. Configure SFRAC

Configuring SFRAC like normal VCS, the bonded NIC will be only 1 private link

# /opt/VRTS/install/installsfrac601 -configure

Veritas Storage Foundation for Oracle RAC 6.0.1 Configure Program

Enter the 64 bit RHEL6 system names separated by spaces: [q,?] (rhel63-01 rhel63-02)

Veritas Storage Foundation for Oracle RAC 6.0.1 Configure Program rhel63-01 rhel63-02

1) Configure SF Oracle RAC sub-components 2) Prepare to Install Oracle 3) Install Oracle Clusterware/Grid Infrastructure and Database 4) Post Oracle Installation Tasks 5) Exit SF Oracle RAC Configuration

Choose option: [1-5,q] (1) 1

To configure VCS for SFRAC the following information is required:

A unique cluster name Two or more NICs per system used for heartbeat links A unique cluster ID number between 0-65535

All systems are being configured to create one cluster.

Enter the unique cluster name: [q,?] sfrac

1) Configure heartbeat links using LLT over Ethernet 2) Configure heartbeat links using LLT over UDP 3) Automatically detect configuration for LLT over Ethernet b) Back to previous menu

How would you like to configure heartbeat links? [1-3,b,q,?] (1)

Discovering NICs on rhel63-01 .......................... Discovered bond0 eth0 eth2 eth3

Enter the NIC for the first private heartbeat link on rhel63-01: [b,q,?] (bond0) Would you like to configure a second private heartbeat link? [y,n,q,b,?] (y) n Enter the NIC for the low-priority heartbeat link on rhel63-01: [b,q,?] (eth0) Are you using the same NICs for private heartbeat links on all systems? [y,n,q,b,?] (y) Checking media speed for bond0 on rhel63-01 ............... Not Applicable (Bonding NIC) Checking media speed for bond0 on rhel63-02 ............... Not Applicable (Bonding NIC)

Enter a unique cluster ID number between 0-65535: [b,q,?] (108) ... Veritas Storage Foundation for Oracle RAC Startup completed successfully

- Add the following line in the /etc/llttab file:

set-dbg-minlinks 2

Configuring SFRAC I/O fencing

# /opt/VRTS/install/installsfrac601 -fencing

4. Complete Oracle RAC installation prerequisites

Follow the Oracle instructions to complete installation prerequisites such as installing OS packages, changing /etc/sysctl.conf file, ntp, creating Oracle groups and users (grid and oracle), creating Oracle binaries directories manually, then reboot if necessary

5. Preparing to install Oracle RAC

Configuring Oracle OCR/voting disk via SFRAC installer

# /opt/VRTS/install/installsfrac601 -configure

Veritas Storage Foundation for Oracle RAC 6.0.1 Configure Program

Enter the 64 bit RHEL6 system names separated by spaces: [q,?] (rhel63-01 rhel63-02)

Veritas Storage Foundation for Oracle RAC 6.0.1 Configure Program rhel63-01 rhel63-02 1) Configure SF Oracle RAC sub-components 2) Prepare to Install Oracle 3) Install Oracle Clusterware/Grid Infrastructure and Database 4) Post Oracle Installation Tasks 5) Exit SF Oracle RAC Configuration

Choose option: [1-5,q] (1) 2 1) Create Oracle Users and Groups 2) Create Storage for OCR and Voting disk 3) Oracle Network Configuration 4) Exit SF Oracle RAC Configuration b) Back to previous menu

Choose option: [1-4,b,q] (1) 2

Create Storage for OCR and Voting disk

Ensure that SF Oracle RAC is running for creating disk groups, volumes and file systems for Oracle

Press [Enter] to continue:

Do you want the installer to assist you in creating disk groups, volumes and file systems for Oracle? [y,n,q] (n) y

1) Create a disk group 2) Use an existing disk group b) Back to previous menu

Choose option: [1-2,b,q] 1

Do you want to enable mirroring? [y,n,q] (y)

Select the serial numbers of at least two disks separated by spaces for creating storage for OCR and Voting disk, excluding disks used for the Operating System and data: [1-18,b,q] 1 2 Enter the disk group name: [b] (ocrvotedg)

Create Storage for OCR and Voting disk

Storage scheme for OCR and Voting disk storage: 1) CVM Raw Volume 2) Clustered File System b) Back to previous menu

Select the storage scheme to be used: [1-2,b,q] 2

Do you want to create separate filesystems for ocr and vote? [y,n,q] (y) n Enter the volume name for OCR and Voting disk: [b] (ocrvotevol) Enter the volume size for OCR and Voting disk (in MB): [b] (640) Enter Oracle UNIX user name: [b] (oracle) grid Enter Oracle UNIX group name: [b] (oinstall)

Enter the mount point location for CFS (common for all the nodes): [b] (/ocrvote)

Veritas Storage Foundation for Oracle RAC 6.0.1 Configure Program rhel63-01 rhel63-02

1) Create Oracle Users and Groups 2) Create Storage for OCR and Voting disk 3) Oracle Network Configuration 4) Exit SF Oracle RAC Configuration b) Back to previous menu

Choose option: [1-4,b,q] (1) 4

- Setting the disk detach policy setting to (local) with ioship off for OCR and voting disk

# vxdg -g ocrvotedg set diskdetpolicy=local

6. Install and configure the Oracle RAC Grid Infrastructure (Clusterware)

- Run the Oracle installer and choose the installation option as "Install and Configure Oracle Grid Infrastructure for a Cluster"

$ ./runInstaller Starting Oracle Universal Installer...

Note:
+) "Cluster Registry Storage Type" is "File System"
+) "Cluster Registry Location" is "/ocrvote"

- Follow the Oracle installer's prompts to run the below scripts seperately on each node in the cluster, ensure they are executed successfully

# /u01/app/oraInventory/orainstRoot.sh

# /u01/app/grid/product/11.2.0.3/grid/root.sh ... Configure Oracle Grid Infrastructure for a Cluster ... succeeded

7. Install the Oracle RAC database software

Run the Oracle installer and choose the installation option "Install database software only"

$ ./runInstaller Starting Oracle Universal Installer...

Following Oracle installer's prompt to run the below script seperately on each nodes in the cluster

# /u01/app/oracle/product/11.2.0.3/db_1/root.sh

8. Use SFRAC to perform Oracle RAC post-installation tasks (configure OSSD and relink libraries)

Verifying the status of SFRAC and Oracle RAC like below

# hastatus -sum

-- SYSTEM STATE -- System State Frozen

A rhel63-01 RUNNING 0 A rhel63-02 RUNNING 0

-- GROUP STATE -- Group System Probed AutoDisabled State

B ClusterService rhel63-01 Y N ONLINE B ClusterService rhel63-02 Y N OFFLINE B cvm rhel63-01 Y N ONLINE B cvm rhel63-02 Y N ONLINE B vxfen rhel63-01 Y N ONLINE B vxfen rhel63-02 Y N ONLINE

$ crsctl stat res -t -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.LISTENER.lsnr ONLINE ONLINE rhel63-01 ONLINE ONLINE rhel63-02 ora.asm OFFLINE OFFLINE rhel63-01 Instance Shutdown OFFLINE OFFLINE rhel63-02 ora.gsd OFFLINE OFFLINE rhel63-01 OFFLINE OFFLINE rhel63-02 ora.net1.network ONLINE ONLINE rhel63-01 ONLINE ONLINE rhel63-02 ora.ons ONLINE ONLINE rhel63-01 ONLINE ONLINE rhel63-02 -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE rhel63-02 ora.LISTENER_SCAN2.lsnr 1 ONLINE ONLINE rhel63-01 ora.LISTENER_SCAN3.lsnr 1 ONLINE ONLINE rhel63-01 ora.cvu 1 ONLINE ONLINE rhel63-01 ora.oc4j 1 ONLINE ONLINE rhel63-01 ora.rhel63-01.vip 1 ONLINE ONLINE rhel63-01 ora.rhel63-02.vip 1 ONLINE ONLINE rhel63-02 ora.scan1.vip 1 ONLINE ONLINE rhel63-02 ora.scan2.vip 1 ONLINE ONLINE rhel63-01 ora.scan3.vip 1 ONLINE ONLINE rhel63-01

Add Oracle RAC patches or patchsets if necessary
Configure the CSSD resource manually in SFRAC as below

Note:
+) Due to PrivNIC/MultiPrivNIC issue with Oracle RAC 11.2.0.3, we are unable to do the same using SFRAC installer script

# haconf -makerw # hares -add cssd Application cvm # hares -modify cssd StartProgram /opt/VRTSvcs/rac/bin/cssd-online # hares -modify cssd StopProgram /opt/VRTSvcs/rac/bin/cssd-offline # hares -modify cssd MonitorProgram /opt/VRTSvcs/rac/bin/cssd-monitor # hares -modify cssd CleanProgram /opt/VRTSvcs/rac/bin/cssd-clean # hares -modify cssd Critical 0 # hares -override cssd OnlineWaitLimit # hares -modify cssd OnlineWaitLimit 5 # hares -override cssd OfflineWaitLimit # hares -modify cssd OfflineWaitLimit 3 # hares -modify cssd Enabled 1 # hares -link cssd ocrvote_mnt_ocrvotedg # haconf -dump -makero

Prevent the automatic startup of the oracle Clusterware/Grid Infrastructure from all nodes in the cluster

# crsctl disable crs // run on node rhel63-01 CRS-4621: Oracle High Availability Services autostart is disabled.

# crsctl disable crs // run on other nodes like rhel63-02 in the cluster CRS-4621: Oracle High Availability Services autostart is disabled.

- Relink the SF Oracle RAC libraries with Oracle RAC (VCSMM and ODM)

# /opt/VRTS/install/installsfrac601 -configure rhel63-01 rhel63-02 Veritas Storage Foundation for Oracle RAC 6.0.1 Configure Program rhel63-01 rhel63-02

Choose option: [1-5,q] (1) 4

1) Configure CSSD agent 2) Relink Oracle Database Binary 3) Exit SF Oracle RAC Configuration b) Back to previous menu

Choose option: [1-3,b,q] (1) 2

Veritas Storage Foundation for Oracle RAC 6.0.1 Configure Program rhel63-01 rhel63-02

Relink Oracle Database Binary

Enter Oracle UNIX user name: [b] (grid) oracle Enter Oracle UNIX group name: [b] (oinstall) Enter absolute path of Oracle Clusterware/Grid Infrastructure Home directory: [b] (/u01/app/grid/product/11.2.0.3/grid) Enter absolute path of Oracle Database Home directory: [b] (/u01/app/oracle/product/11.2.0.3/db_1) Oracle Version Detected: 11.2.0.3 Do you want to continue? [y,n,q] (y)

Veritas Storage Foundation for Oracle RAC 6.0.1 Configure Program rhel63-01 rhel63-02

Relink Oracle Database Binary ... Verifying skgxn library on rhel63-01 ............................................................................................ Done Verifying skgxn library on rhel63-02 ............................................................................................ Done Verifying Oracle Clusterware/Grid Infrastructure status on rhel63-01 ......................................................... Running Changing css misscount to 600 ................................................................................................... Done

Verifying Oracle Database status ................................................................................................ Done

rhel63-01 Copying SF Oracle RAC ODM library on rhel63-01 .................................................................................. Done Setting permissions oracle:oinstall for Oracle ODM library ...................................................................... Done

rhel63-02 Copying SF Oracle RAC ODM library on rhel63-02 .................................................................................. Done Setting permissions oracle:oinstall for Oracle ODM library ...................................................................... Done

Oracle relinking completed successfully

9. Create the database storage on SFRAC CFS

Use Veritas commands to manually create the storage for database, below is an example of only oradatadg/oradatavol, orafradg/orafravol:

# vxdg -s init oradatadg emc0_dd6 emc0_dd7 emc0_dd8 emc0_dd9 # vxassist -g oradatadg maxsize # vxassist -g oradatadg make oradatavol 8047m # vxvol -g oradatadg startall # mkfs -t vxfs /dev/vx/rdsk/oradatadg/oradatavol # mkdir /oradata # mount -t vxfs -o cluster /dev/vx/dsk/oradatadg/oradatavol /oradata # chown oracle:oinstall /oradata # chmod 755 /oradata

# ssh rhel63-02 mkdir /oradata # ssh rhel63-02 mount -t vxfs -o cluster /dev/vx/dsk/oradatadg/oradatavol /oradata

10. Create the Oracle RAC database via Oracle DBCA

Run the Oracle installer to create the Oracle RAC database

$ dbca

Note: +) "Configuration Type" is "Admin-Managed" in this example +) "Storage Type" is "Cluster File System" +) "Use Common Location for All Database files" is "/oradata"

11. Configure SFRAC to manage Oracle database

Configuring SFRAC VCS service groups manually for Oracle database

# haconf -makerw # hagrp -add racsg # hagrp -modify racsg Parallel 1 # hagrp -modify racsg SystemList rhel63-01 0 rhel63-02 # hagrp -modify racsg AutoStartList rhel63-01 rhel63-02 # hares -add oradatadg_cvmvoldg CVMVolDg racsg # hares -modify oradatadg_cvmvoldg CVMDiskGroup oradatadg # hares -modify oradatadg_cvmvoldg CVMActivation sw # hares -modify oradatadg_cvmvoldg CVMVolume oradatavol # hares -add oradata_cfsmount CFSMount racsg # hares -modify oradata_cfsmount MountPoint "/oradata" # hares -modify oradata_cfsmount BlockDevice "/dev/vx/dsk/oradatadg/oradatavol" # hares -add orafradg_cvmvoldg CVMVolDg racsg # hares -modify orafradg_cvmvoldg CVMDiskGroup orafradg # hares -modify orafradg_cvmvoldg CVMActivation sw # hares -modify orafradg_cvmvoldg CVMVolume orafravol # hares -add orafra_cfsmount CFSMount racsg # hares -modify orafra_cfsmount MountPoint "/orafra" # hares -modify orafra_cfsmount BlockDevice "/dev/vx/dsk/orafradg/orafravol" # hares -add racdb Oracle racsg # hares -modify racdb Owner oracle # hares -modify racdb Home "/u01/app/oracle/product/11.2.0.3/db_1" # hares -modify racdb StartUpOpt SRVCTLSTART # hares -modify racdb ShutDownOpt SRVCTLSTOP # hares -local racdb Sid # hares -modify racdb Sid racdb1 -sys rhel63-01 # hares -modify racdb Sid racdb2 -sys rhel63-02 # hares -link oradata_cfsmount oradatadg_cvmvoldg # hares -link orafra_cfsmount orafradg_cvmvoldg # hares -link racdb oradata_cfsmount # hares -link racdb orafra_cfsmount # hagrp -link racsg cvm online local firm # hagrp -enableresources racsg # hares -modify oradatadg_cvmvoldg Critical 0 # hares -modify orafradg_cvmvoldg Critical 0 # hares -modify oradata_cfsmount Critical 0 # hares -modify orafra_cfsmount Critical 0 # hares -modify racdb Critical 0 # haconf -dump -makero

Prevent SFRAC/VCS from faulting the resources until Oracle Clusterware/Grid Infrastructure exhausts its restart attempts

# crsctl status resource ora.racdb.db -f | grep RESTART_ATTEMPTS RESTART_ATTEMPTS=2 # haconf -makerw # hatype -modify Oracle ToleranceLimit 2 # haconf -dump -makero

Preventing automatic database startup

$ srvctl modify database -d racdb -y manual

12. Verify the installation and configuration for SFRAC and Oracle RAC

Verifying all related service groups ONLINE from SFRAC side

# hastatus -sum

-- SYSTEM STATE -- System State Frozen

A rhel63-01 RUNNING 0 A rhel63-02 RUNNING 0

-- GROUP STATE -- Group System Probed AutoDisabled State

B ClusterService rhel63-01 Y N OFFLINE B ClusterService rhel63-02 Y N ONLINE B cvm rhel63-01 Y N ONLINE B cvm rhel63-02 Y N ONLINE B racsg rhel63-01 Y N ONLINE B racsg rhel63-02 Y N ONLINE B vxfen rhel63-01 Y N ONLINE B vxfen rhel63-02 Y N ONLINE

Verifying all related Oracle services, database, listener ONLINE from Oracle RAC side

# crsctl check cluster -all ************************************************************** rhel63-01: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** rhel63-02: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** # crsctl status res -t -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.LISTENER.lsnr ONLINE ONLINE rhel63-01 ONLINE ONLINE rhel63-02 ora.asm OFFLINE OFFLINE rhel63-01 Instance Shutdown OFFLINE OFFLINE rhel63-02 ora.gsd OFFLINE OFFLINE rhel63-01 OFFLINE OFFLINE rhel63-02 ora.net1.network ONLINE ONLINE rhel63-01 ONLINE ONLINE rhel63-02 ora.ons ONLINE ONLINE rhel63-01 ONLINE ONLINE rhel63-02 -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE rhel63-01 ora.LISTENER_SCAN2.lsnr 1 ONLINE ONLINE rhel63-02 ora.LISTENER_SCAN3.lsnr 1 ONLINE ONLINE rhel63-02 ora.cvu 1 ONLINE ONLINE rhel63-02 ora.oc4j 1 ONLINE ONLINE rhel63-02 ora.racdb.db 1 ONLINE ONLINE rhel63-01 Open 2 ONLINE ONLINE rhel63-02 Open ora.rhel63-01.vip 1 ONLINE ONLINE rhel63-01 ora.rhel63-02.vip 1 ONLINE ONLINE rhel63-02 ora.scan1.vip 1 ONLINE ONLINE rhel63-01 ora.scan2.vip 1 ONLINE ONLINE rhel63-02 ora.scan3.vip 1 ONLINE ONLINE rhel63-02

Transfer to DBA team to check further if needed

Applies To

Oracle RAC 11gR2 (specially 11.2.0.3)
Veritas Storage Foundation for Oracle RAC 6.0.1
Red Hat Enterprise Linux Server release 5 or 6

Issue/Introduction

How to install and configure Storage Foundation RAC 6.0.1 for Oracle RAC 11gR2 (11.2.0.3)

Was this article helpful?

thumb_up Yes

thumb_down No

Welcome to "KB Articles"