oracle-rhcs4-notes v.5 - BETA - (C) 2006 Red Hat, Inc.

Oracle is a registered trademark of Oracle Corporation.
Oracle9i is a trademark of Oracle Corporation.
Oracle10g is a trademark of Oracle Corporation.
All other trademarks are property of their respective owners.

Warning: BETA / WORK IN PROGRESS

The information contained herein may be incomplete. If anything is missing or requires clarification, please add a comment with the missing information to: Bug #182423. Oracle Resource Agent

Introduction & Scope

This is a brief HOWTO which should help an experienced system administrator get Oracle(R) 10g Release 2 installed for cold failover cluster operation on Red Hat Cluster Suite 4. The installation of the database is performed only once, rather than once per node.

This documentation is not intended as a replacement for any documentation concerning Red Hat Enterprise Linux, Red Hat Cluster Suite, or Oracle 10g Release 2 operations or installation. In order for this HOWTO to be beneficial, you must have knowledge about how to administer Red Hat Enterprise Linux 4, Red Hat Cluster Suite 4, and Oracle 10g Release 2

Documentation on Oracle Database 10g Release 2 may be found at the following address:
http://www.oracle.com/pls/db102/homepage

Documentation on configuring and installing Red Hat Cluster Suite 4 may be found at the following address:
http://www.redhat.com/docs/manuals/csgfs/

Red Hat Enterprise Linux Documentation may be found here:
http://www.redhat.com/docs/manuals/enterprise/

Part 1: Individual Cluster Node Configuration

Perform the following steps on all nodes in the cluster. Also perform any related tasks as referenced by the Oracle installation documentation. If this HOWTO and the Oracle installation documentation conflict, the Oracle documentation should be followed, especially with respect to configuring sysctl parameters and RPM installations.

STOP ALL CLUSTER DAEMONS AND SERVICES.
Create an Oracle user. The installation will not work if run as root.
```
    # useradd oracle
    # passwd oracle
  
```
The oracle UIDs and GIDs must be the same on all cluster nodes. Make sure they match by looking at /etc/passwd and /etc/groups. If the UID/GIDs do not match, Oracle will not run. The password should be the same for the Oracle user on all nodes as well.

Edit /etc/sysctl.conf and add these lines:

    kernel.sem = 250 32000 100 128
    kernel.shmmax = 536870912
    fs.file-max = 65536
    net.core.rmem_default = 262144
    net.core.rmem_max = 262144
    net.core.wmem_default = 262144
    net.core.wmem_max = 262144
    net.ipv4.ip_local_port_range = 1024 65000

Run:

    # sysctl -f /etc/sysctl.conf

Repeat for all nodes in the cluster.

Install these RPMs if you do not have them already:
```
    compat-libstdc++-33-3.2.3
    libaio-0.3.*
    glibc-devel-2.3.4
  
```
There may be more which are not on this list. REFER TO THE ORACLE INSTALLATION DOCUMENTATION FOR MORE INFORMATION Ensure all of the development tools (gcc, binutils, etc.) installed as well.
Create the Oracle mountpoint(s); e.g.:
```
  mkdir /mnt/oracle
  
```
Install the oracledb.sh resource agent in to /usr/share/cluster

Part 2: Oracle Installation and Basic Tests

The following steps should be performed on one node only.

Choose the partition(s) you want to use for Oracle and format them using the file system of your choice. In my example, I used /dev/sdb7 (a partition on a shared Winchester OpenRAID array) formatted with the ext3 file system.
```
    # mke2fs -j /dev/sdb7
  
```
Add the virtual IP you intend to use for the system and reconfigure your hostname to the FQDN matching the IP address. As with all virtual IPs, the virtual IP used for Oracle must be in the same subnet as an existing IP address on the system. In my case, I chose 192.168.1.20 (my main eth0 IP is 192.168.1.40).
```
    # ip addr add 192.168.1.20/24 dev eth0
    # hostname svc0.foo.test.com
  
```
Mount the target Oracle partition(s). I used only /dev/sdb7, and I chose /mnt/oracle as the target for Oracle. Change the ownership of the root mount point(s) or the subdirectory in which you intend to install Oracle and/or its data.
```
    # mount -t ext3 /dev/sdb7 /mnt/oracle
    # chown oracle.oracle /mnt/oracle
  
```
Run the Oracle installer. I am not a DBA, and I do not know anything about what problems you might hit during install. I just accepted the defaults (except for the Oracle Home and Oracle Inventory directories) and used "oracle" as the database password. If you have any trouble with the installation, refer to the Oracle documentation.

The Oracle installer must be run as the Oracle user. Note: You must use the OUI_HOSTNAME=xxx command line option!
```
    # su - oracle
    $ cd /mnt/oracle_install
    $ ./runInstaller OUI_HOSTNAME=svc0.foo.test.com
  
```
Here are some screenshots to give you an idea of how I installed it.

Install everything on shared storage - even the oraInventory parts

I used /mnt/oracle/home as my Oracle Home Directory

Defaults.

Defaults.

Defaults.

Defaults.

Defaults.

Defaults.

Defaults.

Specify passwords for administration accounts.

Make sure these match your OS group names.

Ready to install.

Installation complete.

Note: If installation fails for any reason, please consult Oracle's Documentation
Test the database by logging in to the Database's Enterprise Management Console to make sure everything is working. You will need to refer to the URL you were given by the Oracle installer. In my case, it was:
```
     http://svc0.foo.test.com:1158/em

     Login: SYS
     Password: (your password)
     Connect As: SYSDBA
  
```

Create a cluster service the resource agent. It should look something like the following example. You should use a text editor to do this for now, as there is no GUI support:

	<service domain="redpref" name="oracle10g">
		<fs device="/dev/sdb7" force_unmount="1"
		    fstype="ext3" mountpoint="/mnt/oracle"
		    name="Oracle Mount"/>
		<oracledb name="orcl" user="oracle" home="/mnt/oracle/home"
		    type="10g" vhost="svc0.foo.test.com"/>
		<ip address="192.168.1.20" />
	</service>

If you prefer the script interface (so that you can add the Oracle script with the GUI), you need to customize ORACLE_HOME, ORACLE_USER, ORACLE_SID, ORACLE_LOCKFILE, and ORACLE_TYPE to match your environment. If you do it this way, the service should look like this:

	<service domain="redpref" name="oracle10g">
		<fs device="/dev/sdb7" force_unmount="1"
		    fstype="ext3" mountpoint="/mnt/oracle"
		    name="Oracle Mount"/>
		<script name="oracle10g" path="/path/to/oracledb.sh"/>
		<ip address="192.168.1.20" />
	</service>

(with the <oracledb> line changed to a <script> line if you used the script method...)

Run a test-stop the service using rg_test (RHCS4U2 and later; Never run this command when the cluster is running!):


	[root@cyan ~]# rg_test test /etc/cluster/cluster.conf stop \
				    service oracle10g
	Running in test mode.
	Stopping oracle10g...
	Restarting /usr/share/cluster/oracledb.sh as oracle.
	Stopping Oracle EM DB Console:                             [  OK  ]
	Stopping iSQL*Plus:                                        [  OK  ]
	Stopping Oracle Database:                                  [  OK  ]
	Stopping Oracle Listener:                                  [  OK  ]
	Waiting for all Oracle processes to exit:                  [  OK  ]
	<info>   Removing IPv4 address 192.168.1.20/22 from eth0
	<info>   unmounting /mnt/oracle
	Stop of oracle10g complete

Set your hostname back to what it should be. If everything is set up correctly, the oracledb.sh script has already generated a hostname.so file which fools Oracle into using the virtual hostname.
```
    # hostname cyan.foo.test.com
   
```

Start the service using rg_test (Never run this command when the cluster is running!):


	[root@cyan home]# rg_test test /etc/cluster/cluster.conf \
				       start service oracle10g
	Running in test mode.
	Starting oracle10g...
	<info>   mounting /dev/sdb7 on /mnt/oracle
	<debug>  mount -t ext3  /dev/sdb7 /mnt/oracle
	<debug>  Link for eth0: Detected
	<info>   Adding IPv4 address 192.168.1.20/22 to eth0
	<debug>  Sending gratuitous ARP: 192.168.1.20 00:02:55:54:28:6c \
					 brd ff:ff:ff:ff...
	Restarting /usr/share/cluster/oracledb.sh as oracle.
	Starting Oracle Database:                                  [  OK  ]
	Starting Oracle Listener:                                  [  OK  ]
	Starting iSQL*Plus:                                        [  OK  ]
	Starting Oracle EM DB Console:                             [  OK  ]
	Start of oracle10g complete

Repeat step 5 for sanity checking / testing. If you can not log in to the EM Database Console, file a Bugzilla.
Repeat step 7 to stop the Oracle database.

Part 3: Finishing up on the other cluster node(s)

Perform the following steps on all nodes in the cluster:

Mount the target Oracle partition(s). I used only /dev/sdb7, and I chose /mnt/oracle as the target for Oracle. During installation, Oracle asked you to run a script which set up symbolic links and/or copy files. We must Run this script on each node as the root user:
```
    # mount -t ext3 /dev/sdb7 /mnt/oracle
    # /mnt/oracle/home/root.sh
    # umount /mnt/oracle
  
```
Copy /etc/oratab and /etc/oraInst.loc from the install node on which to /etc using scp or another utility.
Since the cluster daemons are still not running, copy /etc/cluster/cluster.conf from the install node in to /etc/cluster using scp.
At your option, you may wish to perform basic sanity checks on the Oracle installation. See Part 2, steps 8-10; the procedure is the same. DO NOT START THE ORACLE SERVICE ON MORE THAN ONE NODE.

Part 4: Firing up the cluster and enabling the Oracle service

Refer to the Red Hat Cluster Suite documentation.

Hacks in oracledb.sh

Automatically switches to the correct UID at the right time. RGmanager runs as root, while Oracle must run as the oracle user.
Automatically creates and sets LD_PRELOAD environment variables so that Oracle will always get the right information from the gethostname(2) call. This means that you can not start or stop the clustered Oracle instance outside of the oracledb.sh resource agent/script without also setting the LD_PRELOAD environment variable!, i.e.:
```
   # export LD_PRELOAD=$ORACLE_HOME/hostname.so
   
```
This library is not created until the oracle.sh script is run for the first time.
The oracledb.sh script is designed to be used outside of RHCS4 as a regular init script, too. Change the environment variables, and it should work.