White Paper

EMC SCALING PERFORMANCE FOR ORACLE VIRTUAL MACHINE EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC • Automate performance • Scale OLTP workloads • Rapid provisioning of Oracle databases

EMC Solutions Group Abstract This solution illustrates the benefits of deploying EMC® FAST™ Suite for Oracle OLTP databases in an optimized, scalable virtual environment. The Oracle Real Application Cluster (RAC) 11g database accesses an EMC VNX®7500 storage using the Oracle direct NFS (dNFS) Client enabling simplified configuration, improved performance, and enhanced availability. EMC’s SnapSure™ technology and the Oracle dNFS clonedb feature create rapid provisioning of Oracle databases. Oracle VM provides the virtualization platform. September 2012

Copyright © 2012 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided “as is.” EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All trademarks used herein are the property of their respective owners. Part Number H10739

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM,

2

Table of contents Executive summary............................................................................................................................... 6 Business case .................................................................................................................................. 6 Solution overview ............................................................................................................................ 6 Key results ....................................................................................................................................... 7 Introduction.......................................................................................................................................... 8 Purpose ........................................................................................................................................... 8 Scope .............................................................................................................................................. 8 Audience ......................................................................................................................................... 8 Terminology ..................................................................................................................................... 8 Technology overview .......................................................................................................................... 10 Introduction ................................................................................................................................... 10 EMC VNX7500 ................................................................................................................................ 10 EMC FAST Suite (FAST VP, FAST Cache) ........................................................................................... 10 EMC FAST VP ............................................................................................................................. 10 EMC FAST Cache ........................................................................................................................ 10 EMC SnapSure ............................................................................................................................... 11 Oracle VM ...................................................................................................................................... 11 Oracle RAC ..................................................................................................................................... 11 Oracle dNFS ................................................................................................................................... 11 Solution architecture .......................................................................................................................... 12 Introduction ................................................................................................................................... 12 Solution architecture...................................................................................................................... 12 Hardware resources ....................................................................................................................... 13 Software resources ........................................................................................................................ 13 Oracle storage layout ..................................................................................................................... 14 Oracle file system allocation on VNX7500 ...................................................................................... 16 Oracle dNFS Client configuration .................................................................................................... 16 Configuring Oracle databases ............................................................................................................ 18 Database and workload profile ...................................................................................................... 18 Oracle database schema................................................................................................................ 18 Configuring FAST Cache on EMC VNX7500.......................................................................................... 19 Overview ........................................................................................................................................ 19 Analyze the application workload .................................................................................................. 19 Create and enable/disable FAST Cache .......................................................................................... 19 FAST Cache best practices for Oracle.............................................................................................. 19

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

3

Configuring FAST VP on EMC VNX7500 ............................................................................................... 21 Overview ........................................................................................................................................ 21 Tiering policies .......................................................................................................................... 21 Configure FAST VP .......................................................................................................................... 22 Configuring Oracle virtual machines................................................................................................... 23 Overview ........................................................................................................................................ 23 Step 1: Prepare Oracle VM Server ................................................................................................... 23 Step 2: Install Oracle VM Manager ................................................................................................. 24 Step3: Configure server pool .......................................................................................................... 24 Step 4: Configure network .............................................................................................................. 25 Step 5: Configure Oracle virtual machines...................................................................................... 26 Node scalability test........................................................................................................................... 29 Test objective................................................................................................................................. 29 Test procedure ............................................................................................................................... 29 Test result ...................................................................................................................................... 30 FAST Suite test ................................................................................................................................... 31 FAST Suite and manual tiering comparison .................................................................................... 31 FAST Cache test ............................................................................................................................. 31 FAST Cache warm-up ................................................................................................................. 31 FAST Cache test procedure ........................................................................................................ 32 FAST VP test ................................................................................................................................... 33 FAST VP moving data across tiers .............................................................................................. 33 FAST VP test procedure .............................................................................................................. 33 FAST Suite test ............................................................................................................................... 34 FAST Suite test procedure .......................................................................................................... 34 Test result ...................................................................................................................................... 35 FAST Suite effects on database transactions per minute ............................................................ 35 FAST Suite performance on read response time ......................................................................... 37 Wait statistics from Oracle AWR reports ..................................................................................... 37 Statistics from Unisphere for VNX .............................................................................................. 39 dNFS clonedb test............................................................................................................................... 40 Test objectives ............................................................................................................................... 40 Test procedures ............................................................................................................................. 40 Test results .................................................................................................................................... 41 Resilience test .................................................................................................................................... 42 Test objectives ............................................................................................................................... 42 Test procedures ............................................................................................................................. 42

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM,

4

Physical NIC failure.................................................................................................................... 42 Data mover panic ...................................................................................................................... 43 Test result ...................................................................................................................................... 44 Physical NIC failure.................................................................................................................... 44 Data mover panic ...................................................................................................................... 44 Conclusion ......................................................................................................................................... 45 Summary ....................................................................................................................................... 45 Findings ......................................................................................................................................... 45 References.......................................................................................................................................... 47 White papers ................................................................................................................................. 47 Product documentation.................................................................................................................. 47 Other documentation ..................................................................................................................... 47

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

5

Executive summary Business case

Oracle databases and their dependant applications are mission-critical for businesses. They require high performance with low I/O latency. As a result, Oracle environments must address an increasingly broad range of business demands, including the ability to: •

Scale Oracle online transaction processing (OLTP) workloads for performance.

•

Maximize performance while reducing the cost of ownership of the system.

•



Traditional methods of using Flash drive technology are manual, complex, repetitive, and disruptive



EMC FAST Suite automatically and nondisruptively tunes an application, based on the access patterns.



FAST Cache services active data with less Flash drives, while FAST VP optimizes disk utilization and efficiency with Serial Attached SCSI (SAS) and Near-Line SAS (NL-SAS) drives.

Meet on-demand Oracle requirements to create, deploy, and manage numerous production, development, and testing environments.

This solution addresses all these challenges for a scalable virtualized Oracle RAC 11g database deployment. Solution overview

This solution uses the following technologies to support the demands of the growing enterprise infrastructure: •

EMC® VNX®7500 series

•

EMC Unisphere®

•

EMC Fully Automated Storage Tiering for Virtual Pools (FAST VP)

•

EMC FAST Cache

•

EMC SnapSure™ checkpoint

•

Oracle VM

•

Oracle direct NFS (dNFS)

•

Oracle dNFS clonedb

Technologies such as simplified storage management and fully automated storage tiering (FAST™) provide an infrastructure foundation that meets the following business needs: •

Efficient—Automate Oracle performance tuning With FAST VP and FAST Cache enabled, the storage array continuously tunes an application, based on the access patterns.

•

Cost saving—Improve the total cost of ownership (TCO)

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM,

6

FAST Cache can manage active data with fewer Flash drives, while FAST VP optimizes disk utilization and efficiency across SAS and NL-SAS drives.

Key results

•

Scale—Support growing Oracle workloads that requires increasing high I/Os per second (IOPS) enabled by scaling out a virtual Oracle Real Application Clusters (RAC) node with Oracle dNFS Client and the latest 10 GbE data center technology.

•

Agile—Rapid cloning of Oracle environments such as testing, development, and patch databases by using Oracle dNFS clonedb technology.

This solution demonstrates the following key results: •

Performance improvement with FAST Suite: 

2 times improvement in transactions per minute (TPM)



3.5 times improvement in IOPS



30 times decrease in average read response times



91 percent hit ratio after a warm-up period of FAST Cache

•

Simple management—Only a few steps are required to configure FAST VP and FAST Cache. Customers can enable or disable FAST Cache and FAST VP without affecting the system.

•

Nondisruptive—FAST VP and FAST Cache can identify hot data automatically and nondisruptively. This frees Oracle database administrators (DBAs) from the complex, repetitive, and manual processes of tuning the back-end storage.

•

Scale—Customers can easily and nondisruptively scale out Oracle virtual RAC nodes as application needs evolve, enabling customers to take an incremental approach to address growing workload needs.

•

Agile—The EMC SnapSure checkpoint and the Oracle dNFS clonedb feature enable Oracle DBAs to rapidly deploy additional database copies from a production database for testing, development, or other purposes, while minimizing the storage capacity requirements for those additional database instances.

•

Resilient—The EMC VNX standby data mover and the Oracle dNFS Client enable high availability for Oracle RAC database. The database is still up during the physical NIC failure and the data mover panic, enabling a resilience database with automatic failover.

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

7

Introduction Purpose

This white paper describes the design, testing, and validation of an Oracle OLTP system using the EMC VNX7500 storage platform, EMC FAST Suite, EMC SnapSure, Oracle VM, Oracle dNFS, and the Oracle dNFS clonedb technologies. This solution demonstrates the performance, scalability, and resiliency of the Oracle RAC database on EMC infrastructure.

Scope

The scope of the white paper is to: •

Introduce the key solution technologies.

•

Describe the solution architecture and design.

•

Describe the solution scenarios and present the results of validation testing.

•

Identify the key business benefits of the solution.

Audience

This white paper is intended for chief information officers (CIOs), data center directors, Oracle DBAs, storage administrators, system administrators and virtualization administrators, technical managers, and any others involved in evaluating, acquiring, managing, operating, or designing Oracle database environments.

Terminology

This paper includes the following terminology. Table 1.

Terminology

Acronym

Term

AWR

Automatic Workload Repository

dNFS

Direct NFS

FAST VP

Fully Automated Storage Tiering for Virtual Pools

FC

Fibre Channel

IOPS

I/Os per second

LUN

Logical unit number

NIC

Network interface card

NFS

Network File System

ODM

Oracle Disk Manager

OLTP

Online transaction processing

OVM

Oracle virtual machine

OVS

Oracle VM Server

RAC

Real Application Clusters

SAS

Serial Attached SCSI

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM,

8

Acronym

Term

SCSI

Small Computer System Interface

SGA

System global area

TCO

Total cost of ownership

TPM

Transactions per minute

VNX OE

VNX Operating Environment

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

9

Technology overview Introduction

EMC VNX7500

The solution uses the following hardware and software components: •

EMC VNX7500

•

EMC FAST Suite

•

EMC SnapSure

•

Oracle VM

•

Oracle Database 11g Release 2 Enterprise Edition with Oracle Clusterware

•

Oracle dNFS Client

VNX7500 is a member of the VNX series next-generation storage platform, which is powered by Intel quad-core Xeon 5600 series processors and delivers five 9s availability. The VNX series is designed to deliver maximum performance and scalability for enterprises, enabling them to dramatically grow, share, and costeffectively manage multiprotocol file and block systems.

The VNX Operating Environment (VNX OE) allows Microsoft Windows and Linux/UNIX clients to share files in multiprotocol NFS and Common Internet File System (CIFS) environments. VNX OE also supports iSCSI, FC, and Fibre Channel over Ethernet (FCoE) access for high-bandwidth and latency-sensitive block applications. EMC FAST Suite (FAST VP, FAST Cache)

The FAST Suite for VNX arrays includes FAST VP and FAST Cache. EMC FAST VP VNX FAST VP is a policy-based, auto-tiering solution for enterprise applications. FAST VP operates at a granularity of 1 GB, referred to as a "slice". The goal of FAST VP is to efficiently use storage tiers to lower a customer's TCO by tiering colder slices of data to high-capacity drives, such as NL-SAS, and to increase performance by keeping hotter slices of data on performance drives, such as Flash drives. This process occurs automatically and transparently to the host environment. High locality of data is important to realize the benefits of FAST VP. When FAST VP relocates data, it moves the entire slice to the new storage tier. To successfully identify and move the correct slices, FAST VP automatically collects and analyzes statistics before relocating data. Customers can initiate the relocation of slices manually or automatically by using a configurable, automated scheduler that can be accessed from the Unisphere management tool. The multitiered storage pool allows FAST VP to fully use all three storage tiers: Flash, SAS, and NL-SAS. EMC FAST Cache FAST Cache uses Enterprise Flash drives to add an extra layer of cache between the dynamic random access memory (DRAM) cache and rotating disk drives, thereby creating a faster medium for storing frequently accessed data. FAST Cache is an extendable, read/write cache. It boosts application performance by ensuring that the

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM,

10

most active data is served from high-performing Flash drives and can reside on this faster medium for as long as is needed. FAST Cache tracks data activity at a granularity of 64 KB and promotes hot data into FAST Cache by copying it from the hard disk drives (HDDs) to the Flash drives assigned to FAST Cache. Subsequent I/O access to that data is handled by the Flash drives and is serviced at Flash drive response times—this ensures very low latency for the data. As data ages and becomes less active, it is flushed from FAST Cache to be replaced by more active data. A small number of Flash drives implemented as FAST Cache provides a greater performance increase than a large number of short-stroked HDDs. FAST Cache is particularly suited to applications that randomly access storage with high frequency, such as Oracle OLTP databases. In addition, OLTP databases have inherent locality of reference with varied I/O patterns. Applications with these characteristics benefit most from deploying FAST Cache. The optimal use of FAST Cache is achieved when the working dataset can fit within the FAST Cache. EMC SnapSure

SnapSure enables you to create point-in-time, logical images of a Production File System (PFS). SnapSure uses a "copy on first modify" principle. A PFS consists of blocks. When a block within the PFS is modified, a copy containing the block’s original contents is saved to a separate volume called the SavVol. Subsequent changes made to the same block in the PFS are not copied into the SavVol. SnapSure reads the original blocks from the PFS in the SavVol and the unchanged PFS blocks remaining in the PFS according to a bitmap and blockmap data-tracking structure. These blocks combine to provide a complete point-in-time image called a checkpoint.

Oracle VM

Oracle VM is a server virtualization solution composed of Oracle VM Server for x86 and Oracle VM Manager. Oracle VM Server is composed of a hypervisor and privileged domain (Dom0) that allows multiple domains or virtual machines (such as Linux, Solaris, Windows, and so on.) to run on one physical machine. Oracle VM Manager controls the virtualization environment, and creates and monitors Oracle VM servers and the virtual machines.

Oracle RAC

Oracle RAC extends Oracle Database so that you can store, update, and efficiently retrieve data using multiple database instances on different servers at the same time. Oracle RAC provides the software that manages multiple servers and instances as a single group.

Oracle dNFS

Oracle dNFS is an alternative to using kernel-managed NFS. With Oracle Database 11g release 2 (11.2), instead of using the operating system kernel NFS client, you can configure Oracle Database to access NFS V3 servers direct using an Oracle internal dNFS Client. This native capability enables direct I/O with the storage devices, bypassing the operating system file cache and reducing the need to copy data between the operating system and database memory. The dNFS Client also enables asynchronous I/O access to NFS appliances.

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

11

Solution architecture Introduction

The solution is designed to test the following: •

Node scalability

•

Performance of FAST Suite

•

Provisioning test/development environments

•

System resilience of an Oracle OLTP RAC database configured using dNFS

We carried out the testing on an Oracle RAC 11g database using an EMC VNX7500 array as the underlying storage. Oracle VM was used as the virtualization platform. The VNX array was configured as an NFS server and the Oracle RAC nodes were configured to access the NFS server directly using the Oracle internal dNFS Client. Solution architecture

Figure 1 depicts the architecture of the solution. With Oracle VM version 3.0.3 installed, the Oracle VM server farm consists of two Oracle VM servers; four Oracle virtual machines (two on each VM server) were deployed as a four-node RAC database. As Oracle Support's suggestion, we deployed Oracle RAC 11.2.0.3 for this virtualized solution. The storage and cluster interconnect networks used 10 Gigabit Ethernet (GbE).

Figure 1.

Architecture overview

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM,

12

Hardware resources

Table 2 details the hardware resources for the solution. Table 2.

Hardware resources

Purpose

Quantity

Configuration

Storage array

1

EMC VNX7500 with: • 2 storage processors, each with24 GB cache • 75 x 300 GB 10k 2.5 inch SAS drives • 4 x 300 GB 15k 3.5 inch SAS drives (vault disk) • 11 x 200 GB 3.5 inch Flash drives • 4 x data movers( 2 primary and 2 standby) • Dual-port 10 GbE for each data dover

Oracle VM Server

2

• 4 x 8-core CPUs, • 256 GB RAM, • 2 x dual-port 1 Gb/s Ethernet NICs •

Ethernet switches

Software resources

2 x dual-port 10 Gb/s CNA NICs

2

10 Gb/s Ethernet switches

2

1 Gb/s Ethernet switches

Table 3 details the software resources for the solution. Table 3.

Software resources

Software

Version

Purpose

EMC VNX OE for block

05.32.000.5.006

VNX operating environment

EMC VNX OE for file

7.1.47-5

VNX operating environment

Unisphere

1.2.0.1.0556

VNX management software

Oracle Grid Infrastructure

11.2.0.3

Oracle ASM, Oracle Clusterware, and Oracle Restart

Oracle Database

11.2.0.3

Oracle Database and Oracle RAC

Oracle Enterprise Linux 6.2

6.2

Database server OS

Oracle VM 3.0.3

3.0.3

Oracle Virtual Machine

Swingbench

2.4

TPC-C like benchmark tool

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

13

Oracle storage layout

The disk configuration uses four back-end 6 Gb SAS ports within the VNX7500 storage array. Figure 2 illustrates the disk layout of the environment.

Figure 2.

Note

Disk layout

A Cluster Ready Services (CRS) pool was deployed on the data vault disks due to low I/O activities.

Figure 3 shows a logical representation of the layout of the file system used for the Oracle datafiles. We used four data movers in a 2+2 active/standby configuration. Two active data movers were used to access the file systems, which were distributed evenly across the four SAS ports. The back-end configuration was based on the I/O requirements.

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM,

14

Figure 3.

Datafile system logical view

Unisphere provides a simple GUI to create and manage the file systems. Figure 4 shows the usage of each file system and its serving data mover. It is well balanced for the workload.

Figure 4.

The file system information panel in Unisphere

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

15

Oracle file system allocation on VNX7500

Table 4 details the Oracle file system storage allocation on VNX7500. All the storage pools were created on 300 GB 10k SAS drives. Table 4.

Oracle file system allocation on VNX7500

File type

Raid type

No. of LUNs

Disk volumes (dVols)

Size

Data Mover

FRA files

4+1 RAID 5

10

D11 to D20

4 TB

Server2

CRS files

2+2 RAID 10

1

D21

5 GB

Server2

Datafiles, control files

4+1 RAID 5

10

D1 to D10

2.5 TB

Server2

2.5 TB

Server3

Temp files

4+1 RAID 5

2

D22

200 GB

Server2

D23

200 GB

Server3

Redo logs

2+2 RAID 10

2

D24

100 GB

Server2

D25

100 GB

Server3

Oracle dNFS Client configuration

The Oracle 11g Database needs to be configured to use the Oracle 11g dNFS Client ODM disk libraries. This is a one-time operation and once set, the database will be configured to use the Oracle-optimized, native Oracle dNFS client, rather than the operating system’s hosted NFS client. The standard ODM library was replaced with one that supports dNFS Client. Figure 5 shows the commands that enable the dNFS Client ODM library.

Figure 5.

Enable the dNFS Client ODM library

We configured the Oracle dNFS Client for the virtual environments. We mounted the Oracle file systems and made them available over regular NFS mounts. Oracle dNFS Client used the oranfstab configuration file to determine the mount point settings for the NFS storage devices. Figure 6 shows an extract from the oranfstab file used for this solution.

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM,

16

Figure 6.

Extract from oranfstab configuration file

Once configured, the management of dNFS mount points and load balancing is controlled from the oranfstab and not by the OS.

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

17

Configuring Oracle databases Database and workload profile

Table 5 details the database and workload profile for this solution. Table 5.

Database and workload profile

Profile characteristic

Details

Database type

OLTP

Database size

2 TB

Oracle RAC

4 nodes

Oracle SGA for each node

12 GB

Database performance metric

TPM

Database read/write ratio

60/40

User scaling

• 50 to 250, in increments of 50 (node scalability testing) • 250 to 750, in increments of 100 (FAST Suite testing)

Oracle database schema

This solution applied a simulated OLTP workload by scaling users using Swingbench. We populated a 2 TB database. One 1 TB data database was accessed by different

sessions that were running on the four nodes: ovm-01, ovm-02, ovm-03, and ovm-04. Another 1 TB schema data was left idle to simulate a more realistic skew in the dataset.

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM,

18

Configuring FAST Cache on EMC VNX7500 Overview

FAST Cache uses Enterprise Flash drives to add an extra layer of high-speed cache between DRAM cache and rotating disk drives, thereby creating a faster medium for storing frequently accessed data. FAST Cache is an extendable, read/write cache. It boosts application performance by ensuring that the most active data is served from high-performing Flash drives and can reside on this faster medium for as long as is needed. FAST Cache is most effective when application workloads exhibit high data activity skew. This is where a small subset of data is responsible for most of the dataset activities. Fast Cache is more effective when the primarily block reads and writes are small, fits within the 64K FAST Cache track. The storage system is able to take advantage of such data skew by dynamically placing data according to its activity. For those applications whose datasets exhibit a high degree of skewing, FAST Cache can be assigned to concentrate a high percentage of application IOPs on Flash capacity. This section discusses using FAST Cache and outlines the main steps we carried out to configure and enable FAST Cache for this solution. You can perform the configuration steps using either the Unisphere GUI or the Unisphere command line interface (CLI). For further information about configuring FAST Cache, see Unisphere Help in the Unisphere GUI.

Analyze the application workload

Before you decide to implement FAST Cache, you must analyze the application workload characteristics. Array-level tools are available to EMC field and support personnel for determining both the suitability of FAST Cache for a particular environment and the right size for FAST Cache to configure. Contact your EMC sales teams for guidance. Whether a particular application can benefit from using FAST Cache, and what the optimal cache size should be, depends on the size of the application’s active working set, the access pattern, the IOPS requirement, the RAID type, and the read/write ratio. As indicated in the Technology overview > EMC FAST Cache section of this white paper, the workload characteristics of OLTP databases make them especially suitable for using FAST Cache. For further information, see the white papers: EMC FAST Cache—A Detailed Review and Deploying Oracle Database 11g Release 2 on EMC

Unified Storage.

For this solution, we performed an analysis using the EMC array-level tools, which recommended using FAST Cache and four 200 GB Flash drives as the optimal configuration. Create and enable/disable FAST Cache

For details about creating, enabling, or disabling FAST Cache for a pool, see the white paper EMC FAST Cache—A Detailed Review.

FAST Cache best practices for Oracle

The following are recommended practices: •

Disable FAST Cache on pool/LUNs that do not require it.

•

Size FAST Cache appropriately, depending on the application’s active dataset.

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

19

•

Disable FAST Cache on pool/LUNs where Oracle online redo logs reside.

•

Never enable FAST Cache on archive logs, because these files are never overwritten and are rarely read back (unless you need to recover the database).

EMC recommends that you enable FAST Cache for the Oracle datafiles only. Oracle archive files and redo log files have a predictable workload composed mainly of sequential writes. The array’s write cache and assigned HDDs can efficiently handle these archive files and redo log files. Enabling FAST Cache on these files is neither beneficial nor cost effective.

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM,

20

Configuring FAST VP on EMC VNX7500 Overview

FAST VP is a “game-changing” technology that provides compelling advantages over traditional tiering options. It combines the advantages of automated storage tiering with Virtual Provisioning to optimize performance and cost while radically simplifying management and increasing storage efficiency. Like FAST Cache, FAST VP works best on datasets that exhibit a high degree of skew. FAST VP is very flexible and supports several tiered configurations, such as single tiered, multitiered, with or without a Flash tier, and FAST Cache support. Adding a Flash tier can locate “hot data” on Flash storage in 1 GB slices. FAST VP can be used to aggressively reduce TCO and/or to increase performance. A target workload that requires a large number of performance tier drives can be serviced with a mix of tiers, and a much lower drive count. In some cases, you can achieve an almost two-thirds reduction in drive count. In other cases, performance throughput can double by adding less than 10 percent of a pool’s total capacity in Flash drives. You can use FAST VP in combination with other performance optimization software, such as FAST Cache. A common strategy is to use FAST VP to gain TCO benefits while using FAST Cache to boost overall system performance. There are other scenarios where it makes sense to use FAST VP for both purposes. This paper discusses considerations for an optimal deployment of these technologies. For further information on Fast VP algorithm and policies, see EMC FAST VP for Unified Storage Systems.

Tiering policies

FAST VP includes the following tiering policies: •

Start high then auto-tier (new default policy)

•

Auto-tier

•

Highest available tier

•

Lowest available tier

•

No data movement

Start high then auto-tier (new default policy) Start high then auto-tier is the default setting for all pool LUNs on their creation. Initial data placement is on the highest available tier and then data movement is subsequently based on the activity level of the data. This tiering policy maximizes the initial performance and takes full advantage of the most expensive and fastest drives first, while providing subsequent TCO by allowing less active data to be tiered down, making room for more active data in the highest tier. When a pool has multiple tiers, the start high then auto-tier design is capable of relocating data to the highest available tier regardless of the drive type combination. Also, when adding a new tier to a pool, the tiering policy remains the same and there is no need to manually change it.

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

21

Auto-tier FAST VP relocates slices of these LUNs based on their activity level. Slices belonging to LUNs with the auto-tier policy have second priority for capacity in the highest tier in the pool after the LUNs are set to the highest tier.

Highest available tier Select the highest available tier setting for those LUNs which, although not always the most active, require high levels of performance whenever they are accessed. FAST VP prioritizes slices of a LUN with the highest available tier selected above all other settings. Slices of LUNs set to the highest tier are rank ordered with each other according to activity. Therefore, in cases where the sum total of LUN capacity set to the highest tier is greater than the capacity of the pool’s highest tier, the busiest slices occupy that capacity.

Lowest available tier Select the lowest available tier for LUNs that are not performance-sensitive or response time-sensitive. FAST VP maintains slices of these LUNs on the lowest storage tier available, regardless of activity level.

No data movement The no data movement policy may only be selected after a LUN has been created. FAST VP will not move slices from their current positions once the no data movement selection has been made. Statistics are still collected on these slices for use if and when the tiering policy is changed. Configure FAST VP

In this solution, we set the Auto-Tiering policy to Scheduled. We configured the Data Relocation Schedule setting as Monday to Sunday, starting from 00:00 to 23:45, which determines the time window when FAST VP moves data between tiers. Note

The Data Relocation Rate and Data Relocation Schedule are highly dependent on the real workload in a customer environment. Usually, setting the Data Relocation Rate to Low has less impact on the current running workload.

Set the Tiering Policy for all LUNs containing datafiles to Auto-Tier, so that FAST VP can automatically move the most active data to Flash drive devices. For the details of FAST VP configuration, refer to EMC FAST VP for Unified Storage

Systems — A Detailed Review.

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM,

22

Configuring Oracle virtual machines Overview

Oracle VM Server for x86 is a Oracle server virtualization platform that makes enterprise applications easier to deploy and manage. As shown in Figure 7, the Oracle virtual machine environment consists of one or more Oracle VM servers that are grouped into a server pool and managed by Oracle VM Manager. Oracle VM Server allows hardware resources (CPUs, memory, networks, and storage) to become flexible resource pools from which virtual machines are built.

Figure 7.

Step 1: Prepare Oracle VM Server

Oracle VM environment overview

Oracle VM Server is a central component of Oracle VM. Oracle VM Server includes a virtual machine agent that enables the communication channel with the Oracle VM Manager for management purposes. After successful installation of the Oracle VM Server, the agent is running as shown in Figure 8. Refer to Oracle VM installation and upgrade Guide for detailed installation guidelines.

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

23

Figure 8.

Step 2: Install Oracle VM Manager

Oracle VM agent is running after installation

Oracle VM Manager provides a graphical interface to create and manage the virtual infrastructure. After successful installation of the Oracle VM Manager, a login window to the web-based console is available as shown in Figure 9.

Figure 9.

Oracle VM Manager web console

Refer to Oracle VM installation and upgrade Guide for detailed installation guidelines. Step3: Configure server pool

The Oracle VM Manager connects with one or more Oracle VM servers to create, configure, and manage virtual resources. You can configure one or more Oracle VM servers in a server pool. Within a server pool, you can live-migrate virtual machines for load-balancing purposes or for scheduled maintenance. If a pool member disappears, you can recover its virtual machines and bring them back up on another Oracle VM server, because all necessary resources are available on the shared storage.

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM,

24

In this solution, we configured two Oracle VM server pools consisting of three Oracle VM servers providing database and application services. Each server pool was assigned with virtual IPs, as shown in Figure 10.

Figure 10.

Step 4: Configure network

Oracle VM server pool configuration

The networking infrastructure in the Oracle VM environment is composed of a 1 Gb public network for accessing virtual machine, a 10 Gb IP storage network for virtual machines to access data movers, and a 10Gb network among virtual machines for RAC nodes interconnect. Creating an Oracle VM network involves mapping available network ports to a set of logical Ethernet networks. An Oracle VM network can perform one or more of the following network functions: •

Server management

•

Live migration

•

Virtual machine

•

Storage

In this solution, we used two network functions: •

The network function “server management” was used to manage the virtual machine’s public network.

•

The network function “virtual machine” was used to manage the network traffic between different virtual machines in a server pool and to manage the connections from the virtual machine to external data movers.

As shown in Figure 11, a total of six Oracle VM networks were created on the Oracle VM Server (OVS). •

Two networks were 1 Gb public networks which were created automatically for server management, managing the physical Oracle VM servers in two server pools.

•

The other four Oracle VM networks were configured with next-generation, 10 GbE connectivity. 

“vInterconnect” was the private network dedicated to the Oracle cluster interconnects.

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

25



“vStorage1” and “vStorage2” were private networks serving the NFS storage array to two Oracle virtual machines deployed on it individually.



“vStorageResil” was created for storage redundancy purpose, in order to demonstrate the multipath function of Oracle dNFS.

Port (5) of the two OVS was used for RAC interconnect.

Figure 11.

10 Gb network was used for RAC interconnect and IP storage connection.

Oracle VM network configuration

Figure 12 and Figure 13 show the detailed network configurations of the two Oracle VM servers where the four-node RAC is running.

Step 5: Configure Oracle virtual machines

Figure 12.

Ethernet port configuration of OVS 1

Figure 13.

Ethernet port configuration of OVS 2

You can create Oracle VM guest operating systems in paravirtualized or hardware virtualized mode. In the paravirtualized mode, you can recompile the kernel of the guest operating system to make it aware of the virtual environment. This allows the paravirtualized guest to run at near native speed, because most memory, disk, and network accesses are optimized for maximum performance.

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM,

26

Note

11.2.0.3 RAC for Linux x86 and Linux x86_64 are certified on Oracle VM, but in the Paravirtualized (PV) mode only (Guest OS and drivers). In this solution, Xen PVM was chosen as shown in Figure 14. See Oracle Support Doc ID 464754.1 for details.

Figure 14.

Oracle VM type

For this solution, we first created one paravirtualized virtual machine and configured with the prerequisites for the Oracle software (see Table 6), including: •

Virtualization mode

•

Operating system RPM packages

•

Kernel configuration

•

OS users

•

Supporting software

We then cloned the virtual machines to create the four-node RAC environment. Table 6.

Oracle virtual machine configuration

Part

Description

CPU

14 vCPUs

Memory

32 GB

Operating system

Oracle Linux Server release 6.2

Kernel

2.6.32-300.3.1.el6uek.x86_64

Virtualization mode

paravirtualized

Network interfaces

Eth0: public/management IP network Eth1 (10 Gb): dedicated to cluster interconnect Eth2 (10 Gb): dedicated to NFS connection to Data Mover 2 Eth3 (10 Gb): dedicated to NFS connection to Data Mover 3 Eth4 (10 Gb): dedicated to NFS connection to Data Mover 2 as redundancy Eth5 (10 Gb): dedicated to NFS connection to Data Mover 3 as redundancy

OS user (user created and password set)

Username: oracle UserID:1101

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

27

Part

Description

OS groups

Group: oinstall GroupID:1000 Group: dba GroupID:1031

RPM packages installed (as Oracle prerequisites)

See the relevant Oracle installation guide.

Disk configuration

30 GB – virtual disk for root, /tmp, and the swap space 15 GB – virtual disk for Oracle 11g R2 Grid and RAC Database binaries Note

System configuration (as Oracle prerequisites)

As of Oracle Grid Infrastructure 11.2.0.2, allow for an additional 1 GB of disk space per node for the Cluster Health Monitor (CHM) Repository. By default, this resides within the Grid Infrastructure home.

See the following Oracle Installation Guides: • Oracle Real Application Clusters Installation Guide 11g

Release 2 (11.2) for Linux

• Oracle Grid Infrastructure Installation Guide 11g Release 2

(11.2) for Linux

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM,

28

Node scalability test Test objective

The goal of this test is to demonstrate the performance scalability, with both nodes and users scaled out on Oracle RACs with dNFS and 10 GbE in a virtualized environment. We ran an OLTP-like workload against a single node. We then added users and nodes to show the scalability of both node and user.

Test procedure

We used Swingbench to generate the OLTP-like workload. The testing included the following steps: 1.

Ran the workload on the first node by gradually increasing the number of concurrent users from 50 to 250 in increments of 50.

2.

Added the second node into the workload, and ran the same workload as the previous step on each node separately, which means the total users scaled from 100 (50 on each node) to 500 (250 on each node).

3.

Repeated the previous two steps after adding the third and fourth nodes separately.

4.

For each user iteration, we recorded the front-end IOPS from Unisphere for VNX, the TPM from Swingbench, and the performance statistics from Oracle AWR reports.

Notes • Benchmark results are highly dependent on workload, specific application requirements, and system design and implementation. Relative system performance varies based on many factors. Therefore, you cannot use this workload as a substitute for a specific environment’s application benchmark when making critical capacity planning or product evaluation decisions. • The testing team obtained all performance data in a rigorously controlled environment. Results of other operating environments can vary significantly. • EMC Corporation does not guarantee that a user can achieve similar performance demonstrated in TPM.

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

29

Test result

The Cache Fusion architecture of Oracle RAC immediately uses the CPU and memory resources of the new node(s). Thus, we can easily scale out the system CPU and memory resource without affecting the online users. It provides a scalable computing environment that supports the application workload. Figure 15 shows the TPM Swingbench recorded during the node scalability testing, scaling from both nodes and concurrent users. We scaled the RAC database nodes from one to four. In each RAC configuration, we ran the Swingbench workload with 50, 100, 150, 200, and 250 users on each node. We observed a near-linear scaling TPM from Swingbench as the concurrent user load increased along with the scale of nodes. The chart illustrates the benefits of using EMC VNX7500 storage with Oracle RAC and dNFS for achieving a scalable OLTP environment. Oracle RAC provides not only horizontal scaling, but also guaranteed continuous availability.

Figure 15.

Node scalability test

EMC FAST Suite can automatically optimize the storage to ensure the highest system performance at all times, which helps to improve the system efficiency. FAST Cache working with FAST VP not only boosts the application performance but also provides improved TCO of the whole system. See the FAST Suite test section for test results of enabling FAST Suite on the OLTP workload.

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM,

30

FAST Suite test FAST Suite and manual tiering comparison

Manual tiering involves a repeated process that can take up to nine hours or more to complete each time. In contrast, both FAST VP and FAST Cache operate automatically, eliminating the need to manually identify and move or cache the hot data. As shown in Figure 16, configuring FAST Cache is a one-off process taking 50 minutes or less; hot and cold data is then cached in and out of FAST Cache continuously and automatically.

Figure 16.

Note

FAST Cache test

FAST Suite and manual tiering comparison

The time stated for configuring FAST VP is a conservative estimate. For details about configuring FAST VP, see the Configuring FAST VP on EMC VNX section of this white paper.

FAST Cache boosts the overall performance of the I/O subsystem. Fast Cache enables applications to deliver consistent performance by absorbing heavy read/write loads at Flash drive speeds. We configured four 200 GB Flash drives for FAST Cache. This provided a FAST Cache of 400 GB. We enabled FAST Cache for the storage pool that contains the database datafiles. FAST Cache warm-up FAST Cache requires some warm-up time before the I/O subsystem can achieve a high performance. Figure 17 tracks the FAST Cache read/write hit ratio of the storage pool that stores the datafiles.

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

31

Figure 17.

FAST Cache warm-up period

FAST Cache was empty when it was initially created. During the warm-up period, as more hot data was cached, the FAST Cache hit rate increased gradually. In this test, the hit ratio increased to 91 percent after a warm-up period of approximately six hours. When the locality of the active data changes, it is required to warm up the new data. This process is a normal, expected behavior and is fully automatic. FAST Cache test procedure To test the performance enhancement provided by FAST Cache, we ran the Swingbench workload on all of the four RAC nodes concurrently, with and without FAST Cache enabled. The test procedure includes the following steps: 1.

2.

Baseline testing: a.

Ran the workload against the database from four RAC nodes at the same time without FAST Cache, and scaled it from 250 concurrent users to 750 on each node. The active data size was 1 TB, which was deployed on SAS drives only.

b.

Monitored the performance statistics, including average front-end IOPS and database TPM for each user iteration from Oracle AWR reports and Unisphere for VNX.

FAST Cache testing: a.

Enabled FAST Cache on the storage array after the baseline testing, then ran the same workload and collected performance statistics as the baseline. Along with the number of users, the running workload increased. FAST Cache cached more hot data, which improved the performance.

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM,

32

b.

FAST VP test

After all the FAST Cache testing finished, compared the performance data with the baseline to determine how much the performance enhancement FAST Cache can offer.

We created a two-tier FAST VP with a mixed storage pool, consisting of five Flash drives and 40 SAS drives on VNX7500. FAST VP automatically relocated the LUN data from one disk tier to another within a pool. FAST VP moving data across tiers Initially, all datafiles were placed on SAS devices. With the workload running against the database from four RAC nodes at the same time for a few hours, FAST VP monitored and moved data, as shown in Figure 18.

Figure 18.

FAST VP in progress of moving data

As long as the storage pool followed the applied FAST VP policy, the load continued to ensure the sustainability of the performance levels observed, as shown in Figure 19.

Figure 19.

FAST VP in a steady state

FAST VP test procedure In order to test the performance enhancement of FAST VP, we ran the same workload as that used in FAST Cache testing against the database with FAST VP enabled. The test procedure included the following steps: 1.

Ran the same workload and collected performance statistics as the baseline.

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

33

2.

After all the FAST VP testing finished, compared the performance data with the baseline to determine how much the performance enhancement FAST VP can offer.

The results of these tests are detailed in the Test result section. Note

FAST Suite test

The synthetic benchmark used during testing is more uniformly random than real-world applications, which tends to have greater locality of reference. Customer environments have more inactive data. As a result, we believe that most organizations are able to use less Flash drives capacity to achieve similar, if not better, performance benefits with FAST VP. Customers can achieve additional cost savings if introducing the NL-SAS drives to create the third layer as inactive data is down tiered to NL-SAS. In this solution, the 1 TB of inactive data is automatically moved to the third layer if the layer has been configured.

FAST Suite is the combination of FAST Cache and FAST VP. To demonstrate the advantages of FAST Cache in absorbing random I/O bursts and the benefits of FAST VP’s auto-tiering feature in performance improvement and cost saving, we designed the following two test scenarios: •

Five Flash drives in FAST VP and four Flash drives for FAST Cache.

•

Five Flash drives in FAST VP and two Flash drives for FAST Cache.

Note

Refer to the Analyze the application workload section to appropriately size the Flash drives for FAST Cache. To understand why we used five Flash drives for FAST VP, refer to EMC VNX Unified Best Practice For Performance - Applied Best Practices Guide. The rule of thumb for tier construction on extreme performance Flash tier is 4+1 RAID 5. It yields the best performance versus capacity balance.

In order to test the performance of FAST Suite, we ran the same workload as that used in the FAST Cache test scenario and FAST VP test scenario. FAST Suite test procedure Initially, we placed all datafiles on SAS devices and tested the FAST Suite with four Flash drives for Fast Cache and five Flash drives for FAST VP. 1.

Enabled FAST Cache using four Flash drives and enabled FAST VP using five Flash drives.

2.

Generated the workload against the database to warm up the FAST Cache so that it can reach a stable read/write hit ratio and FAST VP’s moving data was also stable.

3.

Generated the workload against the database to ensure that FAST VP was monitoring and moving data.

4.

Increased the number of users running transactions at intervals to determine how the database performed.

5.

Monitored the performance of the database, and recorded the average frontend IOPS and database TPM for each user iteration.

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM,

34

Then we tested the FAST Suite with two Flash drives for Fast Cache and five Flash drives for FAST VP:

Test result

6.

Destroyed FAST Cache and enabled FAST Cache again using two Flash drives.

7.

Restored the database.

8.

Repeated step 2 to step 5.

FAST Suite effects on database transactions per minute This section compares the database TPM for each test scenario mentioned previously, which includes the following: •

Baseline testing

•

FAST Cache-only testing using four 200 GB Flash drives, configured with RAID 1.

•

FAST VP-only testing using five 200 GB Flash drives, configured with RAID 5.

•

FAST Suite combination testing using four 200 GB Flash drives for FAST Cache and five 200 GB Flash drives for FAST VP.

•

FAST Suite combination testing using two 200 GB Flash drives for FAST Cache and five 200 GB Flash drives for FAST VP.

Figure 20 shows the TPM recorded during the period that the Swingbench workload scaled from 250 to 750 users on each node. This chart shows that the number of transactions processed was much higher when introducing EMC FAST Suite.

Figure 20.

TPM with and without FAST Suite

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

35

When enabling FAST VP, five Flash drives were added to the data pool as RAID 5, the TPM increased by about 20 percent and stabilized at around 270,000, and the read response time was reduced by 50 percent. When enabling FAST Cache, we used four Flash drives. The TPM surged to around 500,000 and stabilized at that level. The response time was dramatically decreased to less than 10 ms. See the Wait statistics from Oracle AWR reports section for detailed analysis from the database side. The other two test results in Figure 20 show the performance of combining FAST Cache and FAST VP, which are two complementary technologies. When configuring four Flash drives for FAST Cache and five Flash drives for FAST VP, the TPM is slightly higher than that of using four Flash drives for FAST Cache only, and the read response time was reduced by 36 percent accordingly. When configuring two Flash drives for FAST Cache and five Flash drives for FAST VP, the TPM is slightly lower than when using four Flash drives for FAST Cache only, and the read response time was tripled. Notes •

When using FAST VP, customers can achieve additional cost savings if NL-SAS drives are added to create the third layer. The higher tiers fill to 90 percent of their available capacity, and cooler data is automatically migrated down to the lower tier as a result of hotter data displacing it. • In this solution, we had 1 TB inactive data, which would automatically be moved to the third layer if the layer was configured, thus making space available on the SAS tier for other applications.

Figure 21 shows a different view of TPM comparison. The performance improvement of using FAST Suite is clear.

Figure 21.

TPM comparison

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM,

36

FAST Suite performance on read response time Figure 22 depicts the significant improvement of read response time when enabling EMC Fast Suite compared with the baseline. •

When using FAST VP, the response time decreased from 120.32 ms to 61.1 ms. In addition, if you have a number of NL-SAS drives, you can use them as a capacity layer, which reduces your TCO by moving cold data to the capacity layer.

•

When enabling FAST Cache, the response time decreased from 120.32 ms to 6.28 ms.

•

When using both FAST Cache and FAST VP, we reduced the read response time from 120.32 ms to 18.55 ms using seven Flash drives, and reduced the read response time from 120.32 ms to 4.28 ms using nine Flash drives.

Figure 22.

Read response time comparison

Wait statistics from Oracle AWR reports Oracle foreground wait statistics highlight potential bottlenecks in Oracle RAC environments. Figure 23 and Figure 24 show the excerpt of top wait events from the RAC Automatic Workload Repository (AWR) reports and compare the waits for the baseline and FAST-only test as well as the FAST Suite combination test respectively. The figures show that the I/O performance was greatly improved when using FAST Cache or FAST Suite combination - the average wait time for db file sequential read decreased dramatically. Because of the increase of supported concurrent user transactions, the commit operations grew rapidly. However, the average wait time for log file sync decreased by almost 50 percent.

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

37

Figure 23.

AWR reports comparison between baseline and FAST Cache-only tests

Figure 24.

AWR reports comparison between baseline and FAST Suite combination tests

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM,

38

Statistics from Unisphere for VNX Figure 25 shows the increase in average IOPS for the datafile systems. The IOPS increased over 200 percent when enabling FAST Suite.

Figure 25.

IOPS comparison

The I/O statistics generated from Unisphere, the TPM from Swingbench (Figure 21), and the read response time (Figure 22) demonstrated the advantages of enabling FAST Suite from different perspectives.

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

39

dNFS clonedb test Test objectives

Customers often need to clone a production database to develop and test new application patches. A new feature called dNFS clonedb enables users to clone test databases instantaneously.

Test procedures

To quickly provision a test database based on the snapshot of database file systems created by EMC VNX SnapSure using the dNFS clonedb feature, the testing team performed the following steps: 1.

Installed Oracle 11.2.0.3 database software in the test environment.

2.

Ran the command as shown in Figure 26 to enable dNFS in the test/development environment and create a dNFS configuration file, as shown in the Oracle dNFS Client configuration section.

Figure 26.

3.

Enable dNFS

To take a hot backup, a.

Put the database in hot backup mode with the following command in SQL*PLUS: alter database begin backup;

b.

Created the VNX SnapSure checkpoint against the database file systems with the following commands.

fs_ckpt data1 -name ck_data1 -Create pool=Save_pool fs_ckpt data2 -name ck_data2 -Create pool=Save_pool

Note

If using SnapSure to create user checkpoints of the primary file system, place SavVol on separate disks when possible and avoid enabling FAST Cache on SavVol.

For details, see Applied Best Practices Guide: EMC VNX Unified Best Practices for

Performance. c.

Take the database out of hot backup mode with the following command in SQL*PLUS.

alter database end backup;

4.

Mounted the VNX SnapSure checkpoint to the target virtual database server.

5.

Generated the backup control file script from the production database with the following command in SQL*PLUS. alter database backup controlfile to trace;

6.

Copied the spfile and the backup control file from the production database to the test environment and made the necessary changes.

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM,

40

Note

To avoid failure of the dbms_dnfs.clonedb_renamefile, you must set clonedb=true in the initialization parameter file for the cloned database.

7.

Started up the cloned database instance with the nomount option and ran the modified backup control file script.

8.

Ran the dbms_dnfs.clonedb_renamefile procedure for each datafile in the cloned database. For example: declare begin dbms_dnfs.clonedb_renamefile('/u02/oradata/racdb784/ soe3_13.dbf.dbf', '/clonedb/uc784/dnfs784/soe3_13.dbf.dbf .dbf'); end;

9.

Recovered database with the following command in SQL*PLUS. recover database using backup controlfile until cancel;

This command asks the archivelogs for the period when the backup was taken and input those file names. 10. Opened the cloned database with the resetlogs option. Test results

When the cloned database is up and running, we can perform read and write activities on the test database. When the workload running is started, storage consumption of the cloned database grows with the speed at which the data is modified. To verify the function of the dNFS clonedb database, we generated the workload using Swingbench against the cloned database as shown in Figure 27.

Figure 27.

Workload against the dNFS clonedb database

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

41

Resilience test Test objectives

The goal is to outline the availability and resilience of the dNFS architecture by demonstrating the database availability during the physical NIC failure and a data mover panic. Up to four network paths defined in the oranfstab file for an NFS server can be used with Oracle dNFS features. The dNFS Client performs load balancing across all specified paths. If one of the paths fails, dNFS reissues I/O commands over any other remaining paths.

Test procedures

Physical NIC failure We manually shut down the NIC to simulate physical NIC failure. The test procedure included: 1.

Configured the database with two paths to two data movers separately.

2.

Ran the Swingbench workload against the first node with 100 users.

3.

Shut down the physical NIC on the virtual machine server to disconnect the resiliency path route to the two data movers as shown in Figure 28.

Figure 28.

4.

Monitored the alert log for warnings as shown in Figure 29.

Figure 29.

5.

Physical NIC startup

Monitored the alert log for warnings as shown in Figure 31.

Figure 31.

7.

dNFS path down messages

After a few seconds, started up the physical NIC as shown in Figure 30.

Figure 30.

6.

Physical NIC shutdown

dNFS path up messages

Waited for the Swingbench workload to complete.

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM,

42

Data mover panic We manually failed over one data mover to the standby one to simulate a data mover panic. 1.

Deployed the database datafiles on two file systems across two data movers: server_2 and server_3.

2.

Ran the Swingbench workload against the first node with 100 users.

3.

Failed the data mover server_2 over to server_5, as shown in Figure 32.

Figure 32.

Data mover failover

This command activates the server_5 (standby data mover) to take over server_2 (the primary data mover). 4.

Verified that the standby data mover server_5 replaced the primary data mover server_2 as shown in Figure 33.

Figure 33.

5.

Failed back the data mover server_2 as shown in Figure 34.

Figure 34.

6.

Data mover status check after failover

Data mover failback

Verified that the data mover server_2 is being restored to the primary data mover successfully as shown in Figure 35.

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

43

Figure 35.

7. Test result

Data mover status check after failback

Waited for the Swingbench workload to complete.

Physical NIC failure When simulating a physical NIC failure, we observed no database outages because Oracle dNFS provided proactive failover operations when using multiple paths. In this solution, we configured two paths to each data mover. When one path was down, the other path was still available. When shutting down one of the physical NICs, Oracle dNFS completed the failover operation in two minutes automatically. When starting up the physical NIC, the second path reconnected automatically and rebalanced the workload among available paths within one minute. Data mover panic The data mover failover and failback completed in one minute or less and no database outage was observed. We checked the database status as well as the Swingbench status, and no error was found in the database log and the Swingbench log.

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM,

44

Conclusion Summary

This paper demonstrates a highly scalable system in a virtualized environment using Oracle 11g RAC deployed on the Oracle VM virtualization platform with EMC Fast Suite technology, SnapSure checkpoint, and dNFS deployed on an EMC VNX 7500. It also shows the high availability of dNFS multipath and demonstrates the ease and quick provisioning of a production database by using EMC VNX SnapSure with Oracle dNFS clonedb. Oracle RAC 11g can easily scale out the nodes to increase the server resources (CPU, memory) of the database server as application needs grow, enabling customers to take an incremental approach to address increases in the Oracle workload. EMC FAST Suite, including FAST Cache and FAST VP, is ideal for the Oracle database environment. FAST Cache and FAST VP complement each other, can boost storage performance, and can lower TCO if used together. FAST Cache can improve performance immediately for burst-prone Oracle data workloads, while FAST VP optimizes TCO by moving Oracle data to the appropriate storage tier, based on sustained data access and demands over time. Additionally, deploying NAS with a 10 Gb Ethernet fabric on the VNX7500 (NFS, CIFS, and pNFS) delivers both infrastructure cost and people and process cost efficiencies versus a block-deployed storage solution. The VNX 7500 platform provides consistent, optimal performance scalability for the Oracle workload. By deploying an Oracle RAC database on a VNX 7500 array, the performance is near-linear scalable when involving additional storage network and RAC nodes, providing higher throughput based on the configuration in this solution. With the combination of the EMC SnapSure checkpoint and the Oracle 11g dNFS clonedb feature, Oracle DBAs can replicate their production environments for test/development purposes in less than 30 minutes, offering near immediate access to the newly provisioned database.

Findings

The key findings of the testing performed for the solution demonstrate: •

Efficiency 

Automate Oracle performance tuning—Compared with the baseline, the performance enhancements offered by FAST Suite include: −

FAST Cache—Creating a FAST Cache with four Flash drives, the performance of the TPM improves by 129 percent and the average read response time reduces from 120.32 ms to 6.78 ms. Using FAST Cache as a secondary cache delivers a 249 percent improvement in IOPS.

−

FAST VP—Enabling FAST VP by using only five Flash drives can improve the performance by 24 percent while reducing the average read response time from 120.32 ms to 61.1 ms. It delivers a 37 percent improvement in IOPS.

−

FAST VP and FAST Cache (configuration 1)—Combining FAST Cache and FAST VP using seven Flash drives increases the performance by 108

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

45

percent and decreases the average response time from 120.32 ms to 18.55 ms. It delivers a 207 percent improvement in IOPS. −

•

•

Performance 

Scale OLTP workloads—The TPM increased almost linearly when adding additional RAC nodes. Customers can take this solution as a baseline or foundation and scale it in a flexible, predicable, and near-linear way, by adding additional storage network, front-end ports, and RAC nodes, to provide higher throughput, based on the configuration in this solution.



Performance improvement with FAST Suite: −

2 times improvement in transactions per minute (TPM)

−

3.5 times improvement in IOPS

−

30 times decrease in average read response times

−

91 percent hit ratio after a warm-up period of FAST Cache

Agility 

•

FAST VP and FAST Cache (configuration 2)—Combining FAST Cache and FAST VP when using nine Flash drives increases the performance by 136 percent and decreases the average response time from 120.32 ms to 4.28 ms. It delivers a 252 percent improvement in IOPS.

Rapid provisioning of Oracle databases—In comparison with the traditional way of database cloning, using EMC SnapSure checkpoint, the Oracle dNFS clonedb feature can quickly and simply provision database clones for test/development purposes, minimizing the impact on the performance of the production database. In addition, it saves DBA time and reduces the storage requirement.

Resilience 

Automatic failover—dNFS Client optimizes multiple network paths to not only load balance I/O across all available storage paths but also provide high availability. EMC VNX 7500 integrates with the Oracle dNFS feature seamlessly to provide the high database availability. The database is still alive during the physical NIC failure and the data mover panic.

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

EMC VNX7500, FAST Suite, Oracle RAC, Oracle VM,

46

References White papers

Product documentation

Other documentation

For additional information, see the EMC white papers listed below. •

Deploying Oracle Database Applications on EMC VNX Unified Storage

•

EMC CLARiiON, Celerra Unified, and FAST Cache—A Detailed Review

•

EMC FAST Cache—A Detailed Review

•

Leveraging EMC FAST Cache with Oracle OLTP Database Applications—Applied Technology

•

Optimizing EMC Celerra IP Storage on Oracle 11g Direct NFS—Applied Technology

•

EMC FAST VP For Unified Storage System—A Detailed Review

For additional information, see the EMC product documents listed below. •

Unisphere Help in the Unisphere GUI

•

Configuring Standbys on VNX

•

EMC VNX Series Release 7.0—Configuring and Managing Network High Availability on VNX

•

EMC VNX Series Release 7.0—Command Line Interface Reference for File

•

EMC VNX Series Release 7.0—Command Line Interface Reference for Block

For additional information, see the documents listed below. •

Oracle Real Application Clusters Installation Guide 11g Release 2 (11.2) for Linux

•

Oracle Real Application Clusters Administration and Deployment Guide 11g Release 2 (11.2)

•

Oracle Grid Infrastructure Installation Guide 11g Release 2 (11.2) for Linux

•

Oracle Clusterware Administration and Deployment Guide 11g Release 2 (11.2)

•

Oracle VM User’s Guide for Release 3.0.3

•

Certified Software on Oracle VM [ID 464754.1]

EMC Scaling Performance for Oracle Virtual Machine EMC VNX7500, EMC FAST Suite, EMC SnapSure, and Oracle RAC

47