HP StorageWorks XP Business Copy (HP StorageWorks XP10000/XP12000/XP24000 Disk Arrays) with Oracle 10gR2 RAC/ASM white paper

Introduction......................................................................................................................................... 3 Features and benefits........................................................................................................................ 3 Example environment overview ............................................................................................................. 4 Hardware configuration.................................................................................................................... 4 HP-UX configuration ......................................................................................................................... 5 Storage configuration ....................................................................................................................... 5 Load testing..................................................................................................................................... 7 Installation and configuration ................................................................................................................ 7 Configure the operating system.......................................................................................................... 7 Install the Oracle Clusterware environment.......................................................................................... 7 Install the Oracle Database files......................................................................................................... 8 Configure and start a listener on both nodes ....................................................................................... 8 Configure ASM................................................................................................................................ 8 Configure a database....................................................................................................................... 9 HP StorageWorks XP RAID Manager ................................................................................................... 10 XP RAID Manager Instance Configuration Files .................................................................................. 10 horcm1.conf—Example ............................................................................................................... 11 horch2.conf—Example................................................................................................................ 12 XP RAID Manager commands used during testing .............................................................................. 12 Oracle recovery using XP Business Copy .............................................................................................. 13 Local/remote cold backup of database............................................................................................. 14 Create the XP Business Copy ........................................................................................................... 14 Splitting the pairs ........................................................................................................................... 15 Restoring with the XP Business Copy................................................................................................. 15 Testing results ................................................................................................................................ 16 Oracle recovery using XP Snapshot...................................................................................................... 17 Local/remote cold backup of database............................................................................................. 18 Create the snapshot pairs ............................................................................................................... 18 Splitting the pairs—Taking the snapshot............................................................................................ 18 Monitoring the volumes................................................................................................................... 19 Restoring with the snapshot ............................................................................................................. 19 Testing results ................................................................................................................................ 20

Conclusion........................................................................................................................................ 21 References ........................................................................................................................................ 21 For more information.......................................................................................................................... 22

Introduction There are several backup and recovery techniques that allow for fast recovery of an Oracle® Database. Some are server based, some are appliance based, and some are storage based. This document highlights the process to perform storage-based real-time backup and recovery using the HP StorageWorks XP10000, XP12000 and XP24000 Disk Arrays with HP StorageWorks XP Business Copy and XP Snapshot. XP Snapshot is an advanced feature of XP Business Copy. Both XP Business Copy and XP Snapshot are storage-based solutions. With XP Business Copy and XP Snapshot, the user can create multiple real-time online copies of logical volumes. While XP Business Copy creates complete copies of storage volumes, XP Snapshot creates space-efficient copies that replicate only the data that has changed, thus saving disk space. Both XP Business Copy and XP Snapshot enable storage solutions such as zero-downtime backup and disaster recovery. Note All references in this document to the words XP, XP array, or StorageWorks XP storage/disk array refer to the all three XP1000, XP12000 and XP24000 Disk Arrays unless otherwise specified.

Features and benefits • Efficient—OS-independent array-based mirroring solution relieves servers of replication tasks • Effective—Background copy process has no impact on primary production operations • Integrated—Seamless interoperation in zero-downtime split-mirror backup solutions • Available—Confidently deploy copy data for disaster recovery protection To demonstrate the use of both XP Business Copy and XP Snapshot, a real-world Oracle environment was created. The configuration used an HP StorageWorks XP disk array for the database storage and consists of a 2-node Oracle RAC Cluster. Oracle’s Automatic Storage Management (ASM) is used to manage the database storage on the XP disk array. The servers used in the configuration were from the HP Integrity server family running HP-UX 11iv2. Therefore, all examples given are based on UNIX®-type commands and operations. XP Business Copy and the newly release feature, XP Snapshot, are also supported under Microsoft® Windows® Server 2000/2003, VMS, MPE/iX, IBM AIX, Linux, and Sun Solaris. This document is not a step-by-step configuration and implementation guide but rather an overview of the process required to implement both XP Business Copy and XP Snapshot in an Oracle Database environment where Oracle ASM is used for the database storage management. Additionally, the intent of this paper was not to evaluate and provide performance metrics but to prove the functionality of Oracle RAC 10gR2 with ASM and XP Business Copy and XP Snapshot technology.

3

Example environment overview To demonstrate the process of using the XP disk array and XP Business Copy/XP Snapshot with Oracle 10g, a test environment consisting of two mid-range HP Enterprise servers, an HP StorageWorks SAN Switch 2/32 Fibre Channel switch, and an XP disk array was assembled. For the XP12000 and XP24000 Disk Arrays, XP Snapshot requires the installation of the optional Shared Memory Platform Board (part number AE032BU for the XP12000 Disk Array and AE155AU for the XP24000 Disk Array). XP Snapshot on the XP10000/XP24000 Disk Arrays does not have this requirement. Table 1 lists the major components used for the testing. Table 1. Hardware environment Equipment

Number used

HP Integrity rx7620 server

2

HP-UX 11.23 VxFS file system for local storage HP StorageWorks XP AutoPath software version 3.0F HP AB465 PII-X 133-MHZ FC HBA

2 per server

HP StorageWorks XP12000 Disk Array

1

HP SAN Switch 2/32 Fibre Channel Switch

1

HP ProCurve 1-GB Ethernet switches

2

HP ProLiant DL580 server with Windows 2003 Advanced Server

1

Hardware configuration The primary servers in the test environment consisted of two HP Integrity 8-way rx7620 servers, using eight 1.1-GHz Intel® Itanium®2 CPUs. The storage used was an XP12000 Disk Array. However, as previously mentioned, the functionality described in this document is applicable to the XP10000 and XP24000 Disk Arrays as well. The storage area network (SAN) was built around a single Fibre Channel switch, zoned to simulate a highly available storage environment. Additional infrastructure hardware such as Gateway and DNS servers for network connectivity were used but are not described. The rx7620 server is a partitionable server supporting two hard partitions, and in this environment both servers were configured with the two partitions—each partition with four CPUs and 16 GB RAM. For this test environment, one partition on two separate servers was used. Two 2-GB Fibre Channel host bus adapters (HBAs) were used in each server partition, allowing for high-availability (HA) operations between the servers and the storage array. The switch was zoned to support the HA configuration. All Ethernet connections are 1 Gb including the private LAN cluster interconnects. A DL580 server with four CPUs and 2 GB of RAM was loaded with Windows 2003 Advanced Server and was used as the database load generating system.

4

Figure 1 depicts the basic test environment. Note that various supporting systems and hardware are not shown.

Figure 1.

READY ALARM MESSAGE

hp

S to ra g e W o rk s x p 1 2 0 0 0 d is k a rra y

PU S H

HP-UX configuration HP-UX 11.23 for Itanium-based servers was used on both servers. The operating system was patched to equal or greater patch levels as required in the document “Oracle Database, Quick Installation Guide, 10g Release 2 (10.2) for hp-ux Itanium.”

Storage configuration For both XP Business Copy and XP Snapshot, the XP disk array was configured to present four volumes of 25 GB to both servers. This is the storage used by ASM to store the Oracle Database and recovery files. Additionally, five smaller 500-MB volumes are presented for use by the Oracle Cluster Registry (ORC) and the cluster voting disks. A tenth volume was configured within the array as a command volume that HP StorageWorks XP RAID Manager uses for in-band array communications. As Oracle is providing the storage management for the shared volumes, they are not configured or mounted by the operating system, but are rather RAW devices. HP-UX does not directly contact these volumes. This type of configuration does not allow for shared configuration or binary files, so local storage on each node must be large enough to hold the Oracle Database and Clusterware binaries along with any local configuration files.

5

For both Oracle Clusterware and ASM to access the common storage, each node must refer to each volume with a common device file name. The HP-UX mknod command was used to create “Alternative Device” file names for all common storage. Using the HP-UX commands ioscan and ls –la /dev/rdsk, the following tables were developed. Each table details the particular node’s path information to the shared storage as used for this environment. Table 2. Node: orclux1 Device file name

Alt device

Tgt

LUN

Port

CU:LDEV

Type

Major #

Minor #

file name /dev/rdsk/c36t0d0

Asm_d1

0

0

CL2E

05:02

OPEN-V

188

0x240000

/dev/rdsk/c36t0d1

Asm_d2

0

1

CL2E

05:03

OPEN-V

188

0x240100

/dev/rdsk/c36t0d2

Asm_d3

0

2

CL2E

05:22

OPEN-V

188

0x240200

/dev/rdsk/c36t0d3

Asm_d4

0

3

CL2E

05:23

OPEN-V

188

0x240300

/dev/rdsk/c36t0d6

n/a

0

6

CL2E

05:6d

OPEN-VCM

188

0x240600

/dev/rdsk/c36t0d7

ora_ocr_d1

0

7

CL2E

05:70

OPEN-V

188

0x240700

/dev/rdsk/c36t1d0

ora_ocr_d2

1

8

CL2E

05:71

OPEN-V

188

0x241000

/dev/rdsk/c36t1d1

ora_vote_d1

1

9

CL2E

05:72

OPEN-V

188

0x241100

/dev/rdsk/c36t1d2

ora_vote_d2

1

0a

CL2E

05:73

OPEN-V

188

0x241200

/dev/rdsk/c36t1d3

ora_vote_d3

1

0b

CL2E

05:74

OPEN-V

188

0x241300

Table 3. Node: orclux3 Alt device Device file name

file name

Tgt

LUN

Port

CU:LDEV

Type

Major #

Minor #

/dev/rdsk/c20t0d0

Asm_d1

0

0

CL2E

05:02

OPEN-V

188

0x140000

/dev/rdsk/c20t0d1

Asm_d2

0

1

CL2E

05:03

OPEN-V

188

0x140100

/dev/rdsk/c20t0d2

Asm_d3

0

2

CL2E

05:22

OPEN-V

188

0x140200

/dev/rdsk/c20t0d3

Asm_d4

0

3

CL2E

05:23

OPEN-V

188

0x140300

/dev/rdsk/c20t0d6

n/a

0

6

CL2E

05:6d

OPEN-VCM

188

0x140600

/dev/rdsk/c20t0d7

ora_ocr_d1

0

7

CL2E

05:70

OPEN-V

188

0x140700

/dev/rdsk/c20t1d0

ora_ocr_d2

1

8

CL2E

05:71

OPEN-V

188

0x141000

/dev/rdsk/c20t1d1

ora_vote_d1

1

9

CL2E

05:72

OPEN-V

188

0x141100

/dev/rdsk/c20t1d2

ora_vote_d2

1

0a

CL2E

05:73

OPEN-V

188

0x141200

/dev/rdsk/c20t1d3

ora_vote_d3

1

0b

CL2E

05:74

OPEN-V

188

0x141300

Note: For ease of viewing, the alternate path information is not shown.

6

The ASM storage configuration consists of two ASM disk groups. Each disk group consists of two 25-GB volumes. This configuration meets Oracle’s basic best practice of having two separate disk groups—one for online/system data and the other for backup/recovery data. The disk group assignments are as follows: • asmdb = asm_d1 + asm_d2 • asmbk = asm_d3 + asn_d4 For this example, the Oracle Cluster Registry storage is configured to use internal redundancy, so two smaller 500-MB volumes are used: ora_ocr_d1 and ora_ocr_d2. Likewise, the voting disk is configured for internal redundancy and consists of the volumes ora_vote_d1, ora_vote_d2, and ora_vote_d3. As the array provides for redundancy, the CRS and voting disks can also be created to support external redundancy where a single device is used for each.

Load testing To effect change on the test database, Benchmark Factory developed by Quest Software was used. An OLTP type of workload with users ranging from 500 to 1,000 was used running against the 25-GB tablespace created within the database. Load on the database was used to both test the storage environment and to make changes to the database before performing a recovery operation.

Installation and configuration As this is not a detailed step-by-step installation and configuration guide, only the major steps required to bring up the preceding environment are listed, noting specific steps as needed. For a list of documents referenced for this project, see the References section.

Configure the operating system 1. Starting from an HP-UX 11.23 operating system install, verify the patch levels required for an

Oracle 10g installation are installed.

2. Install and configure HP AutoPath to support the required storage path failover. In this example,

AutoPath is configured to support an Active/Passive path failover. 3. Assign alias names to the shared XP storage, being sure to match the alias to the same XP volume

as seen on both servers.

Install the Oracle Clusterware environment The following steps are detailed in the “Oracle Clusterware and Oracle Real Application Clusters Installation Guide.” 1. Create the required users and directories, paying attention to user privileges, directory, and file

permissions. The installation process uses Secure Shell (SSH) and Secure Copy (SCP) operations to push the installation onto the other node, so verify all steps using the Oracle-provided cluster verification utility, “cluvfy.” Small omissions or mistakes made at this step will cause significant problems later.

2. Download all Oracle-provided files from the Oracle website, and uncompress them before the

installation process. These files are placed on local storage on the first node, orclux1.

3. Export the ORACLE_HOME and ORACLE_BASE environment variables and install Oracle

Clusterware onto a local volume on the primary node using the “runInstaller” application. 4. During the Clusterware installation, specify the path for both the OCR and voting disk. In this

example, they are located in /dev/rdsk/.

7

5. After completing the installation, verify that the CRS environment is running using the crs_stat

command: The following formatted data was created by using the Oracle-provided CRS resource status query script, detailed in the Oracle document “CRS and 10g Real Applications Clusters.” CRS STATUS QUERY RESULTS

HA Resource ----------ora.orclux1.gsd ora.orclux1.ons ora.orclux1.vip ora.orclux3.gsd ora.orclux3.ons ora.orclux3.vip

Target -----ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE

State ----ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE

on on on on on on

orclux1 orclux1 orclux1 orclux3 orclux3 orclux3

Install the Oracle Database files The following steps are detailed in the document “Oracle Database Installation Guide for hp-ux Itanium.” 1. Export the ORACLE_HOME and ORACLE_BASE environment variables and install Oracle

Database files onto a local volume on the primary node using the “runInstaller” application.

2. During the installation, choose to install the Oracle files only. You will install and create the ASM

instance and database later. 3. Install the Oracle Database Patch Set for 10.2.0.2.0. For pre- and post-installation steps, refer to

the Oracle release notes provided with the Patch Set.

Configure and start a listener on both nodes 1. Use “netca” to configure and start a listener on both nodes. For this environment, default values

were used.

2. Verify that the CRS environment is running using the crs_stat command. As can be seen in the

following screen capture, the listeners for both nodes are installed and online. HA Resource ----------ora.orclux1.LISTENER_ORCLUX1.lsnr ora.orclux1.gsd ora.orclux1.ons ora.orclux1.vip ora.orclux3.LISTENER_ORCLUX3.lsnr ora.orclux3.gsd ora.orclux3.ons ora.orclux3.vip

Target -----ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE

State ----ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE

on on on on on on on on

orclux1 orclux1 orclux1 orclux1 orclux3 orclux3 orclux3 orclux3

Configure ASM 1. Use the Oracle application “dbca” to install and configure an ASM instance. 2. As outlined in the Storage configuration section, create two disk groups and assign two LUNs to

each disk group.

dbca will install the ASM instance and start ASM on both nodes.

8

3. Verify that the ASM instances are online on both nodes using the crs_stat command. HA Resource ----------ora.orclux1.ASM1.asm ora.orclux1.LISTENER_ORCLUX1.lsnr ora.orclux1.gsd ora.orclux1.ons ora.orclux1.vip ora.orclux3.ASM2.asm ora.orclux3.LISTENER_ORCLUX3.lsnr ora.orclux3.gsd ora.orclux3.ons ora.orclux3.vip

Target -----ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE

State ----ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE

on on on on on on on on on on

orclux1 orclux1 orclux1 orclux1 orclux1 orclux3 orclux3 orclux3 orclux3 orclux3

4. On Node1, enter “export ORACLE_SID=+ASM1” from an Oracle user terminal session (or the

name given the SID when installing ASM) and start sqlplus. 5. Review the v$asm_diskgroup and v$asm_disk views and verify that ASM is configured properly.

Configure a database 1. Use the Oracle application “dbca” to configure a Clustered Database. 2. Use ASM to manage the storage. a.

Assign the database to ASMDB.

b.

Assign the Flash Recovery area to ASMBK.

This example installation is designed to support up to 1,000 users so the Shared Server parameter is set to 120, and the Processes value is set to 1100. 3. Finish the dbca installation. 4. Verify that the database instances and services are online on both nodes using the crs_stat

command.

HA Resource ----------ora.orclux1.ASM1.asm ora.orclux1.LISTENER_ORCLUX1.lsnr ora.orclux1.gsd ora.orclux1.ons ora.orclux1.vip ora.orclux3.ASM2.asm ora.orclux3.LISTENER_ORCLUX3.lsnr ora.orclux3.gsd ora.orclux3.ons ora.orclux3.vip ora.snxp.db ora.snxp.orclux1.cs ora.snxp.orclux1.bcxp1.srv ora.snxp.orclux3.cs ora.snxp.orclux3.bcxp2.srv ora.snxp.bcxp1.inst ora.snxp.bcxp2.inst

Target -----ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE

State ----ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE

on on on on on on on on on on on on on on on on on

orclux1 orclux1 orclux1 orclux1 orclux1 orclux3 orclux3 orclux3 orclux3 orclux3 orclux1 orclux1 orclux1 orclux3 orclux3 orclux1 orclux3

Note File and process names reflect the XP Business Copy installation, and were only changed for ease of operation when testing XP Snapshot functionality.

9

Verify that the new database is operating properly by starting a sqlplus session on both nodes and reviewing various v$ views like: • v$active_instances; SQL> select * from v$active_instances; INST_NUMBER ----------1 2

INST_NAME -----------------orclux1:bcxp1 orclux3:bcxp2

• v$instance SQL> select instance_name,instance_number,host_name,active_state,status from v$instance; INSTANCE_NAME ------------bcxp2

INSTANCE_NUMBER --------------2

HOST_NAME --------orclux3

ACTIVE_ST ---------NORMAL

STATUS ---------OPEN

To review the overall database, individual instance, and ASM instance status, use Enterprise Manager. At this point, a 2-node Oracle RAC environment using ASM for storage management should be installed and operational.

HP StorageWorks XP RAID Manager The XP Business Copy and XP Snapshot operations were performed on the XP disk array using the HP StorageWorks XP RAID Manager application. To allow XP RAID Manager to operate on the assigned volumes, several configuration variables must be set and the XP RAID Manager services started. The XP RAID Manager service gets its configuration data from the configuration file “horcm.conf.” The file is edited before the initial XP RAID Manager startup. The actual location and horcm.conf file is set within the /usr/bin/horcmstart.sh shell script.

XP RAID Manager Instance Configuration Files For XP RAID Manager to control the XP Business Copy or XP Snapshot operation, there must be a minimum of two instances started—either one instance on each server, or two instances on one server. In this example, two instances are run on Node1, orclux1. Each XP RAID Manager instance references one side of the XP Business Copy/XP Snapshot pair of volumes. This document does not detail the configuration of the storage array for either XP Business Copy or XP Snapshot operations as that process is outside the scope of this paper. For assistance in the configuration of the XP array, consult your HP field representative. Note Between testing XP Business Copy and XP Snapshot operations, the storage volumes on the XP array were reconfigured as the configuration is different between the two operations so as to reuse the same storage. Both XP Business Copy and XP Snapshot can be used on the same XP array simultaneously, allowing advanced operations such as snapshots of the XP Business Copy volumes. The same horcm.conf file was used for both tests.

10

For a basic XP Business Copy/XP Snapshot configuration, four sections of the horcm.conf file must be edited with the proper values that reflect the environment to be used. • HORCM_MON—This parameter is used for monitoring and communicating within the environment. It describes the IP address, port number, and paired volume error monitoring interval for the local host. • HORCM_CMD—This parameter is used to describe the raw (UNIX) special files used to access the command devices. The command devices must be mapped to a SCSI/Fibre port (path) using the RAID SVP. • HORCM_DEV/HORCM_LDEV—These parameters are used to describe the relationship between physical volumes (which represent half of a pair) and their logical names. Each group name and volume name is unique within this file and should give a hint as to its use (for example, VG01). The group and paired logical volume names described in this file must be the same in the remote host version of this file. • HORCM_INST—This parameter specifies the IP address of the second server and the port number used by the process. In this example, the IP address is the same for each instance. The following details the four sections as modified for this test environment. horcm1.conf—Example #/************************** For HORCM_CMD ************************************/ HORCM_MON #ip_address service pool(10ms) timeout(10ms) NONE 40001 1000 3000 #/****************************************************************************/ HORCM_CMD #dev_name /dev/rdsk/c36t0d6

dev_name

dev_name

#/************************* For HORCM_LDEV ************************************/ HORCM_LDEV #dev_group oracle oracle oracle oracle

dev_name asmdb-1 asmdb-2 asmbk-1 asmbk-2

Serial# 30250 30250 30250 30250

CU:LDEV(LDEV#) 05:02 05:03 05:22 05:23

MU# 0 0 0 0

#/************************* For HORCM_INST ************************************/ # HORCM_INST #dev_group ip_address service oracle 192.168.1.30 40002

11

horch2.conf—Example #/************************** For HORCM_CMD ************************************/ HORCM_MON #ip_address service pool(10ms) timeout(10ms) NONE 40002 1000 3000 #/****************************************************************************/ HORCM_CMD #dev_name /dev/rdsk/c36t0d6

dev_name

dev_name

#/************************* For HORCM_LDEV ************************************/ HORCM_LDEV #dev_group dev_name oracle asmdb-1 oracle asmdb-2 oracle asmbk-1 oracle asmbk-2 #/************************* For # HORCM_INST #dev_group ip_address oracle 192.168.1.30

Serial# CU:LDEV(LDEV#) MU# 30250 05:01 0 30250 05:21 0 30250 05:41 0 30250 05:61 0 HORCM_INST ************************************/ service 40001

Each volume that the host server sees is actually a volume made up from many physical disk drives within the XP array. These primary volumes (P-Vols) are matched with secondary volumes (S-Vols) for the XP Business Copy function. When the array is configured for XP Business Copy operations, XP RAID Manager commands are used to establish the pair and split the pairs, and copy data from the S-Vol back to the P-Vol for recovery. In XP Snapshot operation, the P-Vols are matched with a different type of volume, virtual volumes (V-Vols). These V-Vols are a type of pointer within the XP array that reference a storage pool. In this example the storage pool is set at 80 Gb but can be set as large as the storage within the array has available. The difference between XP Business Copy and XP Snapshot on the XP disk array is that with XP Business Copy, the copy is a byte-for-byte mirror copy, and XP Snapshot is a copy of changed data only, allowing for point-in-time data recovery. When the array is configured for XP Snapshot operations, XP RAID Manager commands are used to establish the pair relationships, initiate the snapshot, and recover back to the point-in-time of the snapshot.

XP RAID Manager commands used during testing XP RAID Manager’s command line interface supports several commands that are used for the XP Business Copy/XP Snapshot functions of the array. For this example, four commands were used. Note The XP RAID Manager commands make reference to storage volumes as local and remote. In this example, when executing XP RAID Manager commands from the user session that started the instance using horch1.conf, the P-Vols are referred to as local, and the S-Vols/V-Vols are referred to as remote. The second instance started using the horch2.conf file, which referenced the volumes in a reverse order of P-Vols and S-Vols/V-Vols as the first instance. All commands in this example were executed from the user session that is running the first instance.

12

• paircreate—Used to establish a primary to secondary pair relationship between volumes. The relationship is enabled, and the data in the P-Vols are copied to the S-Vols. Normal operation to the array is supported during this operation. Options used: – -vx specifies the direction of the P-Vol to S-Vol/V-Vol relationship x=l copy from local P-Vol to remote S-Vol/V-Vol x=r copy from remote S-Vol/V-Vol to local P-Vol – -g defines the group name – -d defines the device name for a single volume within a group • pairdisplay—Displays the pairing status of a volume or group of volumes Options used: – -g defines the group name – -d defines the device name for a single volume within a group – -l displays the local paired volume status – -fxc displays the copy operation rate and completion percentage, displaying data in hex format • pairsplit—Used to change the status of a paired volume. This command suspends or breaks the pair into pair in suspend mode (PSUS) or Simple state (SMPL). This command is used to stop the copy operation within XP Business Copy or to start the point-in-time data copy operation for XP Snapshot. – -g defines the group name – -d defines the device name for a single volume within a group – -S is used to bring the primary and secondary volumes into SMPL mode • pairresync—Used to recover the data back to the point in time that the snapshot was taken. To do this, the -restore option must be used. – -g defines the group name – -restore causes differential data from the S-Vols to be copied to the P-Vols. The P-Vols must not be mounted or in use during this operation.

Oracle recovery using XP Business Copy The actual use of XP Business Copy is straightforward. The XP Business Copy pair’s relationship is created, which causes the XP array to copy the data in the P-Vols to the S-Vols (or S-Vols to P-Vols depending on command usage). When this process is complete, there exists an exact copy of the data held in the S-Vols (or P-Vols depending on command usage). As data changes in the P-Vols, the same changes are made to the S-Vols, always keeping the volumes consistent. The pair can be split at any time, thus creating an exact copy of the data at the time of the split. The S-Vols can now be used as a backup to the P-Vols, allowing the data to be copied back to the P-Vols if the primary data is damaged or lost. The S-Vols can also be presented to a second system where the data can be used for data mining, backup to offline storage, or other operations. When working with applications that have open files on the P-Vols, care must be taken to guarantee that when a pairsplit operation is initiated, write activity is idle or other steps have been taken to allow for consistency within the paired volumes. The following examples demonstrate the simplest methods of using XP Business Copy technology with Oracle 10gR2 by performing user-managed backup and recovery. For details on performing Oracle user-managed backups, see the Oracle document “Backup and Recovery Advanced User’s Guide,” 10g Release2 (10.2).

13

Before performing the XP Business Copy operation, Benchmark Factory was used to load a tablespace named “BMF” (as reference to the Benchmark Factory load generating application) with approximately 25 GB of test data. The v$asm_diskgroup view is a quick and easy way to monitor the total usage of an ASM disk group. You can see the basic storage information after loading the database but before running load on the database environment. Group#

Name

BlkSz

Total

Free

STATE

TYPE

ASMBK ASMDB

4096 4096

51,412 51,412

51,045 17,091

MOUNTED MOUNTED

EXTERN EXTERN

----- ---------- ----- ------- ------- --------- -----1 2

Local/remote cold backup of database This type of backup can be a simple copy of the database files to remote storage while the database is shut down. What makes it unique while using XP Business Copy and/or XP Snapshot technology is that it takes minimal time measured in seconds not hours to shut down the database, split the paired volumes, and start up the Oracle environment. After the pairs are split, restart the Oracle environment and continue normal operation. In the event that you need to recover back to the point in time of the pair split to recover from a catastrophic failure in the database or correct a critical user error, the process is simple and reliable. To create the XP Business Copy: 1. Start with the database in normal operational mode. 2. Use the XP RAID Manager command paircreate to form the XP Business Copy relationship. 3. Shut down the Oracle environment (ASM and database instances on all nodes). 4. Split the pairs, creating the copied data set. 5. Start the Oracle environment, and resume normal activity.

A mirror image copy of the data files now exist, residing in the S-Vols on the XP array. These volumes can now be used to move the database back in time to perform recovery of the database. To recover the database: 1. Shut down the Oracle environment. 2. Use the XP RAID Manager command paircreate remote (S-Vol) to local (P-Vol) to copy the data

back.

3. Split the pairs, restoring the P-Vol/S-Vol order. 4. Start the Oracle environment.

Create the XP Business Copy First the storage volume pair relationship must be created. The paircreate command is used to associate the P-Vol to the XP Business Copy S-Vol. It is important to note that all XP RAID Manager commands are performed on the user session that started the HORCM Instance 1. If you run on the session that started HORCM Instance 2, then the reference for P-Vol and S-Vol are reversed. paircreate -g oracle –vl

14

After creating the pair, the pairdisplay command can be used to verify the operation and monitor the percentage complete. Depending on the size of the database and the load on the database, this operation will take from minutes to hours to complete. Remember, the database is online and available for use during this operation. The following data demonstrates the results of a pairdisplay command, showing that the P-Vol to S-Vol copy operation is in process: pairdisplay -g oracle -fxc Group PairVol(L/R) (Port#,TID, LU-M) ,Seq#,LDEV#.P/S,Status, oracle asmdb-1(L) (CL1-E-3, 0, 0-0 )30250 502.P-VOL COPY, oracle asmdb-1(R) (CL1-E-3, 2, 4-0 )30250 501.S-VOL COPY, oracle asmdb-2(L) (CL1-E-3, 0, 1-0 )30250 503.P-VOL COPY, oracle asmdb-2(R) (CL1-E-3, 2, 5-0 )30250 521.S-vOL COPY, oracle asmbk-1(L) (CL1-E-3, 0, 2-0 )30250 522.P-VOL COPY, oracle asmbk-1(R) (CL1-E-3, 3, 3-0 )30250 541.S-VOL COPY, oracle asmbk-2(L) (CL1-E-3, 0, 3-0 )30250 523.P-VOL COPY, oracle asmbk-2(R) (CL1-E-3, 3, 4-0 )30250 561.S-VOL COPY,

% ,P-LDEV# M 12 501 12 502 16 521 16 503 20 541 20 522 18 561 18 523 -

Splitting the pairs When you are ready to make the backup and the paircreate operation is finished, the pairsplit command is used. At this point all changes made to the database moving forward are stored in the P-Vols and the backup data remains unchanged in the S-Vols. Before issuing this command, shut down the Oracle environment on all nodes, treating this process as a cold backup. After issuing the pairsplit command, start the Oracle environment. pairsplit –g oracle -S Use the pairdisplay command to verify the operation: pairdisplay -g oracle -fxc Group PairVol(L/R) (Port#,TID, LU-M) ,Seq#,LDEV#.P/S,Status, % ,P-LDEV# M oracle asmdb-1(L) (CL1-E-3, 0, 0-0 )30250 502.SMPL ----,--------- oracle asmdb-1(R) (CL1-E-3, 2, 4-0 )30250 501.SMPL ----,--------- oracle asmdb-2(L) (CL1-E-3, 0, 1-0 )30250 503.SMPL ----,--------- oracle asmdb-2(R) (CL1-E-3, 2, 5-0 )30250 521.SMPL ----,--------- oracle asmbk-1(L) (CL1-E-3, 0, 2-0 )30250 522.SMPL ----,--------- oracle asmbk-1(R) (CL1-E-3, 3, 3-0 )30250 541.SMPL ----,--------- oracle asmbk-2(L) (CL1-E-3, 0, 3-0 )30250 523.SMPL ----,--------- oracle asmbk-2(R) (CL1-E-3, 3, 4-0 )30250 561.SMPL ----,--------- -

Restoring with the XP Business Copy If required to recover from a loss of data, the database (everything stored in the ASM disk groups) can be recovered to the point in time that the pairsplit command was issued. To recover the database, shut down the Oracle environment on all nodes and execute the paircreate command with the –vr option. This will instruct the XP array to copy the data in the S-Vols back into the P-Vols, recovering the database from the mirror image copy. paircreate –g oracle -vr Use the pairdisplay command as before to monitor the copy status. When the Status field on all the volumes has changed from COPY to PAIR, the copy operation is complete. The XP Business Copy pairs must be split before starting Oracle. In the process of moving the data from the S-Vols to the P-Vols, the volume rolls reversed. By examining the pairdisplay data, you can see that the P-Vol and S-Vol information has changed. The S-Vol in the configuration is not writable, so we need to split the pair, causing the P-Vol and S-Vol to return to the original configuration.

15

pairdisplay -g oracle -fxc Group PairVol(L/R) (Port#,TID, LU-M) ,Seq#,LDEV#.P/S,Status, oracle asmdb-1(L) (CL1-E-3, 0, 0-0 )30250 502.S-VOL PAIR, oracle asmdb-1(R) (CL1-E-3, 2, 4-0 )30250 501.P-VOL PAIR, oracle asmdb-2(L) (CL1-E-3, 0, 1-0 )30250 503.S-VOL PAIR, oracle asmdb-2(R) (CL1-E-3, 2, 5-0 )30250 521.P-VOL PAIR, oracle asmbk-1(L) (CL1-E-3, 0, 2-0 )30250 522.S-VOL PAIR, oracle asmbk-1(R) (CL1-E-3, 3, 3-0 )30250 541.P-VOL PAIR, oracle asmbk-2(L) (CL1-E-3, 0, 3-0 )30250 523.S-VOL PAIR, oracle asmbk-2(R) (CL1-E-3, 3, 4-0 )30250 561.P-VOL PAIR,

% ,P-LDEV# M 100 501 100 502 100 521 100 503 100 541 100 522 100 561 100 523 -

pairsplit –g oracle –S Execute a pairdisplay command again and notice that the volumes have returned to the original order. pairdisplay -g oracle -fxc Group PairVol(L/R) (Port#,TID, LU-M) ,Seq#,LDEV#.P/S,Status, % ,P-LDEV# M oracle asmdb-1(L) (CL1-E-3, 2, 4-0 )30250 501.SMPL ----,--------- oracle asmdb-1(R) (CL1-E-3, 0, 0-0 )30250 502.SMPL ----,--------- oracle asmdb-2(L) (CL1-E-3, 2, 5-0 )30250 521.SMPL ----,--------- oracle asmdb-2(R) (CL1-E-3, 0, 1-0 )30250 503.SMPL ----,--------- oracle asmbk-1(L) (CL1-E-3, 3, 3-0 )30250 541.SMPL ----,--------- oracle asmbk-1(R) (CL1-E-3, 0, 2-0 )30250 522.SMPL ----,--------- oracle asmbk-2(L) (CL1-E-3, 3, 4-0 )30250 561.SMPL ----,--------- oracle asmbk-2(R) (CL1-E-3, 0, 3-0 )30250 523.SMPL ----,--------- -

Restart the Oracle environment. All files stored on the four volumes used by ASM have been returned to the original state at the time of the pairsplit operation. This can be verified by reviewing the files managed by ASM. They will be the same as before the first pairsplit operation.

Testing results The intent of this example was to prove that XP Business Copy can be used to do a point-in-time backup and recovery of an Oracle Database in a complex Oracle environment. The test sequence was: 1. Install Oracle RAC on a 2-node cluster, using ASM for the created database. 2. Configure XP RAID Manager for XP Business Copy operations. 3. Create a general purpose database and fill a 25-GB tablespace. 4. Perform XP Business Copy operation, create and split volume pairs. 5. Use Benchmark Factory to effect change to the database. 6. Recover using the XP Business Copy data. 7. Verify that the database was returned to the original state. 8. Repeat multiple times to prove repeatability.

No issues or problems were encountered during the testing, and the operation was found to be easy and straightforward. Benchmark Factory was used to compare overall system performance. Two tests were performed: one with the XP Business Copy pairs created and the second with the XP Business Copy pairs split. The test consisted of a 500 to 1,000 user operation against a 25-GB tablespace.

16

The Benchmark Factory results did not show any noticeable performance difference with XP Business Copy active in a paired state, or not active in a non-pared state as can be seen in the Figure 2.

Figure 2.

Business Copy - SMPL / PAIR STATE 28 26 24

TPS

22 20 18 16 14 1000

980

960

940

920

900

880

860

840

820

800

780

760

740

720

700

680

660

640

620

600

580

560

540

520

500

12

Users Node 2 PAIR

Node 1 PAIR

Node 2 SMPL

Node 1 SMPL

Oracle recovery using XP Snapshot The actual use of XP Snapshot is straightforward. The XP Snapshot V-Vols, which contain all differential or changed data since the snapshot was taken, can be used by mounting them to a different server, or in the case of this paper, moved back into the P-Vols, to do a point-in-time recovery. When working with applications that have open files on the P-Vols, care must be taken to guarantee that when a restore operation is initiated, write activity is idle or other steps have been taken to allow for consistency within the copied volumes. The following examples demonstrate the simplest methods of using XP Snapshot technology with Oracle 10gR2 by performing user-managed backup and recovery. For details on performing Oracle user-managed backups, see the Oracle document “Backup and Recovery Advanced User’s Guide,” 10g Release2 (10.2). Before performing the XP Snapshot operation, Benchmark Factory was used to load a tablespace called BMF with approximately 28 GB of test data. The v$asm_diskgroup view is a quick and easy way to monitor the total usage of an ASM disk group. You can see the basic storage information after loading the database but before the XP Snapshot activities. Group#

Name

BlkSz

Total

Free

STATE

TYPE

ASMBK ASMDB

4096 4096

51,412 51,412

51,045 17,091

MOUNTED MOUNTED

EXTERN EXTERN

----- ---------- ----- ------- ------- --------- -----1 2

17

Local/remote cold backup of database When the snapshot has been created, restart the Oracle environment and continue normal operation. In the event that you need to recover back to the point in time of the snapshot to recover from a catastrophic failure in the database or user error, the process is simple and reliable. To create the snapshot: 1. Use the XP RAID Manager command paircreate to form the XP Snapshot relationship. 2. Shut down the Oracle environment (ASM and database instances on all nodes). 3. Create the snapshot using the pairsplit command. 4. Start the Oracle environment, and resume normal activity.

A snapshot of all changes made to the data files moving forward now exists, residing in the storage pool referenced by the V-Vols. These volumes can now be used to move the database back in time to perform recovery of the database. To move back in time: 1. Shut down the Oracle environment. 2. Perform the Snap-Back operation using the pairresync command. 3. Start the Oracle environment.

Create the snapshot pairs First the storage volume pair relationship must be created. The paircreate command is used to associate the P-Vol to the Snapshot S-Vol. paircreate

-g oracle –vl

After creating the pair, the pairdisplay command can be used to verify the operation: pairdisplay -g oracle -fx Group PairVol(L/R) (Port#,TID, LU-M) ,Seq#,LDEV#.P/S,Status, Seq#,P-LDEV# oracle asmdb-1(L) (CL1-E-3, 0, 0-0 )30250 502.P-VOL PAIR,30250 510 oracle asmdb-1(R) (CL1-E-3, 2, 0-0 )30250 510.S-VOL PAIR,----502 oracle asmdb-2(L) (CL1-E-3, 0, 1-0 )30250 503.P-VOL PAIR,30250 511 oracle asmdb-2(R) (CL1-E-3, 2, 1-0 )30250 511.S-VOL PAIR,----503 oracle asmbk-1(L) (CL1-E-3, 0, 2-0 )30250 522.P-VOL PAIR,30250 512 oracle asmbk-1(R) (CL1-E-3, 2, 2-0 )30250 512.S-VOL PAIR,----522 oracle asmbk-2(L) (CL1-E-3, 0, 3-0 )30250 523.P-VOL PAIR,30250 513 oracle asmbk-2(R) (CL1-E-3, 2, 3-0 )30250 513.S-VOL PAIR,----523

M -

Splitting the pairs—Taking the snapshot When you are ready to make the snapshot, the pairsplit command is used. At this point all changes made to the database are stored in the P-Vols and the original data is stored in V-Vols. Before issuing this command, shut down the database and ASM instances on all nodes, treating this process as a cold backup. After issuing the pairsplit command, start ASM on all nodes, and then start the database instance. pairsplit –g oracle

18

Use the pairdisplay command to verify the operation. Notice the changes in the pairdisplay information—the status field has changed from PAIR to PSUS or SSUS. The M=W value on the four P-Vols indicates that the associated S-Vols are in a read/write mode. pairdisplay -g oracle -fx Group PairVol(L/R) (Port#,TID, oracle asmdb-1(L) (CL1-E-3, 0, oracle asmdb-1(R) (CL1-E-3, 2, oracle asmdb-2(L) (CL1-E-3, 0, oracle asmdb-2(R) (CL1-E-3, 2, oracle asmbk-1(L) (CL1-E-3, 0, oracle asmbk-1(R) (CL1-E-3, 2, oracle asmbk-2(L) (CL1-E-3, 0, oracle asmbk-2(R) (CL1-E-3, 2,

LU-M),Seq#,LDEV#.P/S,Status, Seq#,P-LDEV# 0-0 )30250 502.P-VOL PSUS,30250 510 0-0 )30250 510.S-VOL SSUS,----502 1-0 )30250 503.P-VOL PSUS,30250 511 1-0 )30250 511.S-VOL SSUS,----503 2-0 )30250 522.P-VOL PSUS,30250 512 2-0 )30250 512.S-VOL SSUS,----522 3-0 )30250 523.P-VOL PSUS,30250 513 3-0 )30250 513.S-VOL SSUS,----523

M W W W W –

Monitoring the volumes To monitor the status of the snapshot storage area, use the raidvchkscan command. With this command you can monitor several different parameters relating to the storage used in the XP Snapshot operations. The following command sequence is reporting that the storage pool is in normal state (POLN) with 26% of the pool used (57050 MB of 77118 MB available) and there are four snapshot instances in the pool (one per volume in the “oracle” storage group). The following data represents approximately 2 hours of 1,000 users against the database. raidvchkscan -v pid PID

POLS

U(%)

001

POLN

32

-fx

SSCNT

Available(MB)

4

52283

Capacity(MB) 77118

Seq# 30250

Num LDEV# 3

504

Restoring with the snapshot If required to recover from a loss of data, the database (everything stored in the ASM disk groups) can be recovered to the point in time that the pairsplit command was issued. To do this, shut down the Oracle environment on all nodes and execute the pairresync command with the –restore option. This will instruct the XP array to copy back the saved change data stored in the storage pool to the point in time that the snapshot was taken. pairresync -restore -g oracle Use the pairdisplay command to monitor the recovery status. The following shows volumes asmdb-1 and asmdb-2 are in the recovery process and volumes asmbk-1 and asmbk-2 are finished. pairdisplay -g oracle -fx Group PairVol(L/R) (Port#,TID, LU-M) ,Seq#,LDEV#.P/S,Status, Seq#,P-LDEV# M oracle asmdb-1(L) (CL1-E-3, 0, 0-0 )30250 502.P-VOL RCPY,30250 510 oracle asmdb-1(R) (CL1-E-3, 2, 0-0 )30250 510.S-VOL RCPY,----502 oracle asmdb-2(L) (CL1-E-3, 0, 1-0 )30250 503.P-VOL RCPY,30250 511 oracle asmdb-2(R) (CL1-E-3, 2, 1-0 )30250 511.S-VOL RCPY,----503 oracle asmbk-1(L) (CL1-E-3, 0, 2-0 )30250 522.P-VOL PAIR,30250 512 oracle asmbk-1(R) (CL1-E-3, 2, 2-0 )30250 512.S-VOL PAIR,----522 oracle asmbk-2(L) (CL1-E-3, 0, 3-0 )30250 523.P-VOL PAIR,30250 513 oracle asmbk-2(R) (CL1-E-3, 2, 3-0 )30250 513.S-VOL PAIR,----523 –

When all pair volumes show the status of PAIR, the operation is complete. The data has been restored to that point in time when the pairsplit command was issued. Restart the Oracle environment. All files stored on the four volumes used by ASM have been returned to the original state at the time of the pairsplit operation. This can be verified by reviewing the files managed by ASM. They will be the same as before the pairsplit operation.

19

Testing results The intent of this example was to prove that XP Snapshot can be used to do a point-in-time recovery of an Oracle Database in a complex Oracle environment. The test sequence was: 1. Install Oracle RAC on a 2-node cluster, using ASM for the created database. 2. Configure XP RAID Manager for XP Snapshot operations. 3. Create a general purpose database and fill a 25-GB tablespace. 4. Perform XP Snapshot. 5. Use Benchmark Factory to effect change to the database. 6. Recover to XP Snapshot point-in-time. 7. Verify that the database was returned to the original state. 8. Repeat multiple times to prove repeatability.

No issues or problems were encountered during the testing, and the operation was found to be easy and straightforward. Benchmark Factory was used to compare overall system performance. Two tests were performed: one with XP Snapshot active and one without XP Snapshot. The test consisted of a 500 to 1,000 user operation against a 25-GB tablespace. The Benchmark Factory results did not show any noticeable performance issues between when XP Snapshot was active (pairs split) and when XP Snapshot was not active (pairs created but not split). There is published knowledge that there is a possible performance penalty on the XP disk array when using XP Snapshot, as any change to data on the P-Vol (user writes) must be copied to V-Vol (storage pool) before writing the new data to the P-Vol. The preceding test did not expose this performance issue.

20

Figure 3.

98 0

94 0

90 0

86 0

82 0

78 0

74 0

70 0

66 0

62 0

58 0

54 0

28 26 24 22 20 18 16 14 12 50 0

TPS

Database Performance With/Without Snapshot

Users Node1 Snapshot

Note2 Snapshot

Node1 NoSnapshot

Node2 NoSnapshot

Conclusion There are many ways to back up or copy an Oracle 10g database. This paper documented two such methods. The HP StorageWorks XP Business Copy and XP Snapshot products performed the task of providing an efficient and timely backup method for an Oracle Database while not affecting the operation of the primary database. These solutions are not server based, but storage based and thus do not require processing power from the servers. The XP Business Copy and XP Snapshot products provide additional flexibility in that multiple databases can be backed up simultaneously. While XP Business Copy offers the security of having a mirror image copy of your data, XP Snapshot offers the ability of moving back in time to recover from a loss. Both technologies offer a usable solution for Oracle Database recovery.

References • “Oracle Database Backup and Recovery Advanced Users Guide,” 10g Release 2(10.2), Oracle Corporation • “Oracle Database, Quick Installation Guide, 10g Release 2 (10.2) for hp-ux Itanium,” B25294-01, January 2006, Oracle Corporation • “Oracle Database Administrator’s Guide 10g Release 2 (10.2),” B14231-01, June 2005 • “Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide,” 10g Release 2 (10.2), August 2005 • “Oracle Clusterware and Oracle Real Application Clusters Installation Guide,” 10g Release 2 (10.2) for HP-UX, January 2006 • “HP StorageWorks RAID Manager XP User’s Guide,” T1610-96004, November 2005, HewlettPackard

21

For more information • www.hp.com/go/oracle • www.hp.com/go/storageworks

© 2007 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Itanium and Intel are trademarks of Intel Corporation in the U.S. and other countries. Oracle is a registered US trademark of Oracle Corporation, Redwood City, California. Microsoft and Windows are U.S. registered trademarks of Microsoft Corporation. UNIX is a registered trademark of The Open Group. 4AA1-4116ENW, July 2007