Oracle GoldenGate for MS SQL Server 2005

Oracle GoldenGate for MS SQL Server 2005 Objective Upon completion of this lesson, you will be able to configure GoldenGate to provide SQL Server to SQL Server transactional data management using log-based extraction. During this lesson, you will learn how to:  Prepare your user environment  Prepare the database  Configure and start the change capture process of database operations  Configure and execute the initial data load process  Configure and start the change delivery process of database operations

SQL Server configuration The following diagram illustrates a SQL Server to SQL Server configuration. Windows server

Windows server

MS SQL Server Manager

Source

Log

Manager

Network

Extract

Collector

Trails

MS SQL Server Replicat

Target

                                         Page 1

Oracle GoldenGate for MS SQL Server 2005

Overview of Tasks Prepare the GoldenGate Environment The GoldenGate application must be installed on both the source and target systems. A checkpoint table and Manager Service will be implemented. In order to execute this lesson, the SQL Server source database must be configured to support log-based extraction, and an ODBC data source must be created and configured correctly. The source and target tables will be created with scripts provided in the GoldenGate installation. For log-based SQL Server extraction, additional logging must be enabled to provide enough information to reconstruct update operations.

Configure Initial Data Load Initial database synchronization is almost always required. To initially load data across actively changing databases, you have limited choices. GoldenGate provides the ability to perform initial data synchronization while your application remains active. There are several techniques available, including sending data to files to be loaded by Replicat or by Microsoft BCP/DTS bulk-load utility, or sending data directly to Replicat. This lesson demonstrates extracting the data with Extract and sending the data directly to the Replicat process. This is known as the Direct Load method.

server

server

MS SQL Server

MS SQL Server Network

Source

Initial Extract

Initial Replicat

Target

Configure Change Capture The Extract process is configured to capture change data directly from the SQL Server transaction logs and distribute the changes to a series of files known as a GoldenGate trails.

Configure Change Delivery Once the tables have been initially loaded with data, the Replicat process is configured to deliver the captured change data into the target database.

                                         Page 2

Prepare the Environment

Exercise 1.

Prepare the Environment               

Objective The goals of this exercise are to:  Create the database  Create SQL Server logins and users  Create ODBC data source.  Set up transaction logging.  Create tables and populate source data.

Prerequisites This lab assumes that the GLOBALS parameters have been created with a name and that the Manager service has been added. This should have been done when installing GoldenGate for Windows.

Prepare the SQL Server 2005 source environment Install Manager 1. Configure Manager process on the source Execute the following commands on the system. Note! The port numbers must be unique if you are installing GoldenGate more than once on

the same server. Shell> ggsci GGSCI> EDIT PARAMS MGR

 In the parameter file, enter the following parameter, then save and close the file. PORT

 Start Manager. GGSCI> START MANAGER

 Verify the results:                                          Page 3

Prepare the Environment

GGSCI> INFO MANAGER

Create the database 2. Create the database From Start>Programs, run SQL Server Management Studio.  Enter the name of the local SQL Server instance, then click connect.

Note: These instructions assume SQL Server Authentication which requires the more complex, double layer, of authentication. SQL Server Authentication is an option only if it is permitted at the server level, and only Windows Authentication is enabled by default. If you are using Windows Authentication, you can leave out most of the login and password entries.  This displays the main SQL Server Management Studio dialog box. Right, right-click Database and select the New Database option.

                                         Page 4

Prepare the Environment

 The New Database dialog box appears. In the Name box, type the from the Lab Preparation worksheet.

 Click OK to add the database.  The main SQL Server Management Studio dialog box reappears.

3. Create SQL Server logins and users  Right click Security and select the New > Login.

 The New Login dialog box appears.

                                         Page 5

Prepare the Environment

 For Name, type the .  Select Authentication,  If you selected SQL Server Authentication, type the in the two Password fields.  Under Defaults, select your for Database, and leave Language set to the default.  Do not click OK. Instead, from Select a page in the upper left-hand box, double click Server Roles.

                                         Page 6

Prepare the Environment

 Under Server Role, check the box for the System Administrators role.  Click OK to exit. The next step creates a schema that you will associate with the new user.  While you have the database selected, click on the New Query button in the main menu.  The right side of the dialog displays the SQL Query interface. Type in: create schema ;

And press the Execute button from the toolbar.

                                         Page 7

Prepare the Environment

Now you need to associate the new user and the schema that you have created with the new database.  Right click on the Security option that is under the database and select New > User.  The New Database User dialog box appears. Enter the name of your user and press the browse button labeled with three dots (…).

                                         Page 8

Prepare the Environment

 The Select Login dialog displays. Press the Browse button.

 Select your new user from the list. Then press OK.  Your selected user now displays in the object names box. Press OK to exit from Select Login.  Back in the New Database User dialog, enter the you added as the default schema and press OK.

4. Create the ODBC system data source name The GoldenGate Extract process connects to a SQL Server database through an ODBC (Open Database Connectivity) connection. Both the Extract and Replicat components of GoldenGate require that a system data source name (DSN) be established, which stores the information about how to connect to the SQL Server.  Click Start > Settings > Control Panel.  Double-click Administrative Tools.  Double-click Data Sources (ODBC) to open the ODBC Data Source Administrator dialog box.  Click the System DSN tab, and then click the Add button. The Create New Data Source dialog box appears.

                                         Page 9

Prepare the Environment

 Select the SQL Server driver and then click Finish. The Create a New Data Source to SQL Server dialog box appears.

 For Name, type .  Optionally type in a description, type for this source DSN.  For Which SQL Server do you want to connect to, select local if SQL Server is on the local system, or enter the server name at your SQL Server location.  Click Next.  Answer the question about how SQL Server should verify the login by selecting the option that you entered when creating the . Then type in Login ID and in Password.

                                         Page 10

Prepare the Environment

 Click Next.  Make certain the default database is set to the database that you created; otherwise select Change the default database to: and select it. Leave the other settings to their defaults to use ANSI.

 Click Next.  Leave the next dialog box set to the defaults, and click Finish.  In the confirmation, click Test Data Source to test the connection.  Close the confirmation dialog box and the Create a New Data Source box.                                          Page 11

Prepare the Environment

 You can leave the SQL Server Management Studio running on the source system, because you will be using it later.

5. Create the practice tables and insert source data Execute the following commands on the system only.  Go to SQL Server Management Studio, select your database and select New Query from the toolbar.

 If the correct database is not displayed in the drop down box (showing ggsdata above), click on the arrow and select it from the list.  Click the File > Open and navigate to the demo_mss_create.sql script.  Click Open to open the script in the New Query window.  Execute the script by clicking the Execute Query button on the toolbar.  Verify the results using the following commands in the input window: sp_help tcustmer go sp_help tcustord go

                                         Page 12

Prepare the Environment

 Click the X button to close the input tab. Click New Query whenever you need to run another SQL command.  Following the steps you used to execute the table create script, run the demo_mss_insert.sql script to insert source data into the source tcustmer and tcustord tables. Note! Run this on the source only.  Verify the results as you did before.  Leave the SQL Management Studio running on the source system.

Install replication components for SQL Server 2005 Installing replication components sets up the Microsoft SQL Server 2005 Distributor database and Publisher. This is not needed if you meet the following conditions:  You have installed MicroSoft’s Cumulative Update 6 (CU6) for SQL Server 2005.  You are running GoldenGate version 9.5.1.15 or higher. If you do not meet both of these conditions, replication components must be installed on your system in order to capture changes generated by SQL Server 2005. Unless your instructor indicates that these components have already been installed and configured, perform the workshop SQL Server 2005 Replication Components before proceeding to the next section.

                                         Page 13

Prepare the Environment

Prepare for transaction logging To support GoldenGate extraction, the following are required. Execute these steps on the system only.  Log truncation and non-logged bulk copy must be turned off.  The SQL Server database must be set to the full recovery model (this is the default for SQL Server 2005).  At least one full database backup must be done before GoldenGate processes are started for the first time.  Additional log data must be enabled so GoldenGate can reconstruct update operations.  Microsoft SQL Server 2005 replication components must be installed (previous step).

6. Turn off log truncation and bulk copy In SQL Management Studio on the source system perform the following steps to turn off log truncation and bulk copy.  Clear the input window and then use the following command to check the status of the log truncation option. Type the command in the input window (upper window), then click the F5 key to execute it. exec sp_dboption '', 'trunc. log on chkpt.'

 The following response appears in the results (lower) window with a status of either ON or OFF.

If log truncation is ON, turn it off by executing the following command. exec sp_dboption '', 'trunc. log on chkpt.', false

The output should now show a CurrentSetting of OFF’.  Check the status of the non-logged bulk copy option by executing the following command: exec sp_dboption '', 'select into/bulkcopy'

If non-logged bulk copy is on, disable it with the following command: exec sp_dboption '', ' select into/bulkcopy ', false

                                         Page 14

Prepare the Environment

7. Execute backup Still in SQL Server Management Studio:  In the console tree, right-click the database name and select Tasks > Backup

.  The Back Up Database dialog box displays. If the back up destination is correct, skip the next two bulleted steps and initiate the backup as explained in the last step. If it is not correct, change the backup destination and click ADD.

                                         Page 15

Prepare the Environment

 The Select Backup Destination appears, where you can select File Name and then click the navigation (...) button to navigate to a directory and specify the backup file name.

 Click OK to close the Select Backup Destination dialog box. The backup file is added to the Destination list box in the SQL Server Backup dialog box.  Click OK to start the backup. This procedure ensures that no transaction log information is lost when GoldenGate starts up.

                                         Page 16

Prepare the Environment

8. Set up capture of additional log data Perform the following steps on the source system to configure the database to log full before and after images for each update operation.  From the source operating system's command shell, run GGSCI.  Log into the database with the following command: GGSCI > DBLOGIN SOURCEDB , USERID , PASSWORD

 Issue the following commands to force the logging of the full before and after image for updates. GGSCI > ADD TRANDATA .tcustmer GGSCI > ADD TRANDATA .tcustord

 Verify that supplemental logging has been turned on for these tables. GGSCI> INFO TRANDATA .tcust*

Note: With SQL Server 2005 you cannot drop tables that have TRANDATA set. First you must delete the TRANDATA setting.

Disable SQL Server Log Reader Agent job 9. Stop and disable the job The Log Reader Agent job cannot run concurrently with the GoldenGate Extract process. Perform the following steps to stop and disable this job.  In SQL Server Management Studio, connect to the SQL Server 2005 instance.  Start SQL Server Agent, if not running.  Expand the SQL Server Agent folder.  Expand the Jobs folder.  Find the job that was created by the ADD TRANDATA command. The name is based

on the server, instance, and database name, plus the iteration of the publication.It will look similar to the following: KTANCO\SQL2005-SQLLBE_SRC-1  Right click the job and select Stop Job.  Right click the job again and select Disable.

                                         Page 17

Prepare the Environment

Prepare the SQL Server 2005 target system Install Manager 1. Configure Manager process on the target Execute the following commands on the system. Note! The port numbers must be unique if you are installing GoldenGate more than once on

the same server. Shell> cd Shell> ggsci GGSCI> EDIT PARAMS MGR

 In the parameter file, enter the following parameter, then save and close the file. -- GoldenGate Manager Parameter file PORT

 Start Manager. GGSCI> START MANAGER

 Verify the results: GGSCI> INFO MANAGER

Steps to create the database 2. Create the database From Start>Programs, run SQL Server Management Studio.  Enter the name of the local SQL Server instance, then click connect. Note: These instructions assume SQL Server Authentication which requires the more complex, double layer, of authentication. For only Windows Authentication, you can leave out most of the login and password entries.

                                         Page 18

Prepare the Environment

 This displays the main SQL Server Management Studio dialog box. Right, right-click Database and select the New Database option.

 The New Database dialog box appears. In the Name box, type the from the Lab Preparation worksheet.

                                         Page 19

Prepare the Environment

 Click OK to add the database.  The main SQL Server Management Studio dialog box reappears.

3. Create SQL Server logins and users  Right-click Security and select the New > Login.

 The New Login dialog box appears.

                                         Page 20

Prepare the Environment

 For Name, type the .  Select Authentication,  If you selected SQL Server Authentication, type the in the two Password fields.  Under Defaults, select your for Database, and leave Language set to the default.  Do not click OK. Instead, from Select a page in the upper left-hand box, double click User Mapping.

                                         Page 21

Prepare the Environment

 In the upper Users mapped to this login area, activate the checkmark next to your database.  In the lower Database role membership for area,, check the box for the db_owner role.  Click OK to exit. The next step creates a schema that you will associate with the new user.  While you have the database selected, click on the New Query button in the main menu.  The right side of the dialog displays the SQL Query interface. Type in: create schema ;

And press the Execute button from the toolbar.

                                         Page 22

Prepare the Environment

Now you need to associate the new user and the schema that you have created with the new database.  Right click on the Security option that is under the database and select New > User.  The New Database User dialog box appears. Enter the name of your user and press the browse button labeled with three dots (…).

                                         Page 23

Prepare the Environment

 The Select Login dialog displays. Press the Browse button.

 Select your new user from the list. Then press OK.  Your selected user now displays in the object names box. Press OK to exit from Select Login.  Back in the New Database User dialog, enter the you added as the default schema and press OK.

4. Create the ODBC system data source name The GoldenGate Extract process connects to a SQL Server database through an ODBC (Open Database Connectivity) connection. Both the Extract and Replicat components of GoldenGate require that a system data source name (DSN) be established, which stores the information about how to connect to the SQL Server.  Click Start > Settings > Control Panel.  Double-click Administrative Tools.  Double-click Data Sources (ODBC) to open the ODBC Data Source Administrator dialog box.  Click the System DSN tab, and then click the Add button. The Create New Data Source dialog box appears. Note: For SQL Server 2000, you may use any ODBC driver supported by the database. For SQL Server 2005, however, you must use the ODBC driver that is included with Microsoft Data Access Components (MDAC). Both Extract and Replicat will abend if the SQL Native Client ODBC driver that is provided with SQL Server 2005 is used.

                                         Page 24

Prepare the Environment

 Select the SQL Server driver and then click Finish. The Create a New Data Source to SQL Server dialog box appears.

 For Name, type .  For a description, type "GG Training Data source" when creating this source DSN. This field is optional.  For Which SQL Server do you want to connect to, select the server name.  Click Next.  Answer the question about how SQL Server should verify the login by selecting the option that you entered when creating the . Then type in Login ID and in Password.

                                         Page 25

Prepare the Environment

 Click Next.  Make certain the default database is set to the database that you created; otherwise select Change the default database to: and select it. Leave the other settings to their defaults to use ANSI.

 Click Next.  Leave the next dialog box set to the defaults, and click Finish.  In the confirmation, click Test Data Source to test the connection.  Close the confirmation dialog box and the Create a New Data Source box.                                          Page 26

Prepare the Environment

5. Create the practice tables Execute the following commands on the system.  Go to SQL Server Management Studio, select your database and select New Query from the toolbar.

 If the correct database is not displayed in the drop down box (showing ggsdata above), click on the arrow and select it from the list.  Click the File > Open and navigate to the demo_mss_create.sql script.  Click Open to open the script in the New Query window.  Execute the script by clicking the Execute Query button on the toolbar.  Verify the results using the following commands in the input window: sp_help tcustmer go sp_help tcustord go

                                         Page 27

Initial Data Load using Direct Load Method

Exercise 2.

Initial Data Load using Direct Load Method Objective These steps configure initial load groups that copy source data and apply it to the target tables.

Steps to configure initial load 1. Add the initial data load extract batch task group Execute the following commands on the system to add an Extract process called EINI1.  Execute the following command in GGSCI to create the batch task. Shell> cd Shell> ggsci GGSCI> ADD EXTRACT EINI, SOURCEISTABLE

 Verify the results: GGSCI> INFO EXTRACT *, TASKS

2. Configure the initial data load Extract parameter file Execute the following commands on the system.  Execute the following command in GGSCI to open the editor. GGSCI> EDIT PARAMS EINI

 Add the following lines to the parameter file: --- GoldenGate Initial Data Capture -- for TCUSTMER and TCUSTORD -EXTRACT EINI SOURCEDB , USERID , PASSWORD RMTHOST , MGRPORT RMTTASK REPLICAT, GROUP RINI TABLE .TCUSTMER; TABLE .TCUSTORD; The process names used in lab exercises, for example EINIBD, are made up of 1) one character for the GoldenGate process (E for Extract, R for Replicat); 2) three or four to describe the process type (INI for initial data load, ORA for capture from or delivery to an Oracle database, etc.) and 3) two characters to create a unique identifier (usually your initials). 1

                                         Page 28

Initial Data Load using Direct Load Method

3. Add the initial data load Replicat batch task group Execute the following commands on the system.  Execute the following command in GGSCI to create the batch task. GGSCI> ADD REPLICAT RINI, SPECIALRUN

 Verify the results: GGSCI> INFO RINI*, TASKS

4. Configure the initial data load Replicat parameter file Execute the following commands on the system.  Execute the following command in GGSCI to open the editor. GGSCI> EDIT PARAMS RINI

 Add the following lines to the parameter file -- GoldenGate Initial Data Load Delivery -REPLICAT RINI ASSUMETARGETDEFS TARGETDB , USERID , PASSWORD DISCARDFILE ./dirrpt/RINI.txt, PURGE MAP .TCUSTMER, TARGET .TCUSTMER; MAP .TCUSTORD, TARGET .TCUSTORD;

5. Execute the initial data load process Execute the following commands on the system.  Execute the following command to start the initial data load process. GGSCI> START EXTRACT EINI

 Verify the results: GGSCI> VIEW REPORT EINI

Execute the following commands on the system.  Verify the results: GGSCI> VIEW REPORT RINI

                                         Page 29

Initial Data Load using Direct Load Method In your SQL query facility, enter the following commands to verify the content of the tables: SELECT * FROM „.TCUSTMER‟ SELECT * FROM „.TCUSTORD‟

                                         Page 30

Configure Change Capture

Exercise 3.

Configure Change Capture Objective The goals of this exercise are to:  Add the Extract process that will capture changes.  Add the trail that will store the changes.  Start the Extract process.

Steps to configure change 1. Add the Extract checkpoint group Execute the following commands on the system to create the Extract group. GGSCI> ADD EXTRACT EMSS, TRANLOG, BEGIN NOW

Verify the results: GGSCI> INFO EXTRACT EMSS

2. Create the Extract parameter file Execute the following commands on the system.  Execute the following command in GGSCI to open the editor. GGSCI> EDIT PARAM EMSS

 Add the following lines to the parameter file. Note: MANAGESECONDARYTRUNCATIONPOINT is valid only for SQL Server 2005, so should not be included for SQL Server 2000. --- Parameter file to capture -- TCUSTMER and TCUSTORD Changes -EXTRACT EMSS --Include the following only for SQL Server 2005 TRANLOGOPTIONS MANAGESECONDARYTRUNCATIONPOINT -- If SQL Server Replication is also running, choose -- the NOMANAGESECONDARYTRUNCATIONPOINT option SOURCEDB , USERID , PASSWORD RMTHOST , MGRPORT RMTTRAIL ./dirdat/ TABLE .TCUSTMER;                                          Page 31

Configure Change Capture

TABLE .TCUSTORD;

Note: Record the two characters selected for your : ______. You will need this in the next step and when you set up the Replicat.

3. Define the GoldenGate extract trails Execute the following commands on the system to create a trail. GGSCI> ADD RMTTRAIL ./dirdat/, EXTRACT EMSS, MEGABYTES 50

Verify the results: GGSCI> INFO RMTTRAIL *

4. Start the Extract process On the system, issue the following command in GGSCI to start the Extract process. GGSCI> START EXTRACT EMSS

Verify the results: GGSCI> INFO EXTRACT EMSS, DETAIL

                                         Page 32

Configure Change Delivery

Exercise 4.

Configure Change Delivery Objective The goals of this exercise are to:  Set up the checkpoint table on the target system.  Create a named group that includes the Replicat process and the checkpoint tables.  Configure the Replicat group by adding parameters.  Start the Replicat group.

Set up checkpoints 1. Edit the GLOBALS file on the target system Execute the following commands on the system.  Edit the GLOBALS parameter file to add the checkpoint table. Shell> cd Shell> ggsci GGSCI> EDIT PARAMS ./GLOBALS

In the text editor, add the following line: CHECKPOINTTABLE .ggschkpt

 Record the checkpoint table owner and name, then save and close the file. Table owner ____________________ name ___________________ Note: You could name the table anything you want, but for training purposes we are using ggschkpt.

2. Activate the GLOBALS parameters For the changes to the GLOBALS configuration to take effect, you must exit the session in which the changes were made. Execute the following command to exit GGSCI. GGSCI> EXIT

                                         Page 33

Configure Change Delivery

3. Add a Replicat checkpoint table Execute the following commands on the system. This step adds the checkpoint table that you specified when you created the GLOBALS parameter file. It will be created in the database.  Run GGSCI on the target. Shell> cd Shell> ggsci

 Execute the following commands in GGSCI. GGSCI> DBLOGIN SOURCEDB , USERID , PASSWORD GGSCI> ADD CHECKPOINTTABLE

Configure delivery 4. Add the Replicat checkpoint group Execute the following commands on the system.to create the Replicat group named RMSS. GGSCI> ADD REPLICAT RMSS, EXTTRAIL ./DIRDAT/

Note: Refer to your Extract set up for the correct two-character .

5. Create Replicat parameter file  Execute the following command on the system to edit the Replicat parameter file. GGSCI> EDIT PARAM RMSS

 Add the following lines to the parameter file. REPLICAT RMSS TARGETDB , USERID , PASSWORD “” HANDLECOLLISIONS ASSUMETARGETDEFS DISCARDFILE ./DIRRPT/RMSS.DSC, PURGE MAP .TCUSTMER, TARGET .TCUSTMER; MAP .TCUSTORD, TARGET .TCUSTORD;

4. Start the Replicat process Execute the following commands on the system.

                                         Page 34

Configure Change Delivery

 Execute the following command in GGSCI to start the Replicat process. GGSCI> START REPLICAT RMSS

 Verify the results: GGSCI> INFO REPLICAT RMSS

Discussion points Search in the Windows/UNIX Reference Guide for the information on the following questions.

1. When to use HANDLECOLLISIONS When would you use HANDLECOLLISIONS? What does it do? _________________________________________________________________________ _________________________________________________________________________

2. When should you use ASSUMETARGETDEFS? _________________________________________________________________________ _________________________________________________________________________

3. What is the purpose of the DISCARDFILE? _________________________________________________________________________ ________________________________________________________________________

                                         Page 35

Generate Activity and Verify Results

Exercise 5.

Generate Activity & Verify Results The goals of this exercise are to:  Execute miscellaneous update, insert, and delete operations on the source system.  Verify the delivery of the changes to the target  Turn off the error handling used for initial load.

Verify your source SQL Server 2005 results Steps to generate database operations 1. Generate inserts, updates, and deletes Execute the following commands on the system.  Go to SQL Server Management Studio, select your database and press New Query.

 If the correct database is not displayed in the drop down box (showing ggsdata above), click on the arrow and select it from the list.  Click the File > Open and navigate to the demo_mss_misc.sql script and click Open to open it in the input window.  Execute the script by clicking the Execute Query button on the toolbar.  Verify and record processing statistics with the following command in GGSCI. GGSCI> SEND EXTRACT EMSS, REPORT GGSCI> VIEW REPORT EMSS

___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________                                          Page 36

Generate Activity and Verify Results

 Verify the data by executing the following commands in the New Query window of SQL Server Management Studio on the source: select * from TCUSTMER; select * from TCUSTORD;

Verify your target SQL Server 2005 results 3. Verify your results on the SQL Server 2005 target  Verify and record processing statistics on the with the following command in GGSCI. GGSCI> SEND REPLICAT RMSS, REPORT GGSCI> VIEW REPORT RMSS

 Verify the data by executing the following commands in the New Query window of SQL Server Management Studio on the source: select * from TCUSTMER; select * from TCUSTORD;

Turn off error handling 4. Turn off initial load error handling for the running delivery process Execute the following commands on the system. GGSCI> SEND REPLICAT RMSS, NOHANDLECOLLISIONS

5. Remove initial load error handling from the parameter file GGSCI> EDIT PARAMS RMSS

Remove the HANDLECOLLISIONS parameter.

                                         Page 37

Exercise Name

                                       Page 38