IBM eServerJ iSeriesJ
Session:
Webcast
Remote Journaling = Better Plumbing
Larry Youngren Adam Stallman 8 Copyright IBM Corporation, 2003. All Rights Reserved. This publication may refer to products that are not currently available in your country. IBM makes no commitment to make available any products referred to herein.
1
IBM eServer iSeries
Remote Journal SQL Table Applications
Journal
Receiver
HSL
Remote Receiver
Source System Target System © 2003 IBM Corporation 2
IBM eServer iSeries
What can we do with such technology ?
© 2003 IBM Corporation 3
IBM eServer iSeries
Continuous HA recovery SQL Table
HA Vendor Replay Software
Applications
Replicated SQL Table
Harv est
Journal
Receiver
Remote Journal Source System
Remote Receiver
Target System
© 2003 IBM Corporation 4
IBM eServer iSeries
Role Swap Applications
SQL Table HA Vendor Software
SQL Table
Harv est
Journal
Receiver
Remote Receiver Remote Journal
Source System
Target System
© 2003 IBM Corporation 5
IBM eServer iSeries
Role Swap Why the extra Journal on the Target ?
SQL Table
Applications
SQL Table
Harv est
Journal
Receiver
Remote Receiver Remote Journal
Source System
Target System Delta
© 2003 IBM Corporation 6
IBM eServer iSeries
Better Plumbing ?
© 2003 IBM Corporation 7
IBM eServer iSeries
Pre-RJ use of HA BP software for Harvesting on Source SQL Table Applications
BP Software Replica SQL Table
Journal
Send
Harv es
t
MI Boundary Log Space
Receiver
Source System
W/o Remote Journaling
Target System
© 2003 IBM Corporation 8
IBM eServer iSeries
Vintage approach Primary System Apps
RCVJRNE job
Backup System
Exit program
Target job
Apply job
MI Send entry
DB files
JRN
Communications transport
Receive entry
Log space
Replicated DB files
JRN Receiver © 2003 IBM Corporation 9
IBM eServer iSeries
RJ-enabled approach Primary System
Backup System RCVJRNE job
Apps
Apply job
MI Receive entry
DB files
Remote JRN
JRN JRN Receiver
Communications transport
Replicated DB files
JRN Receiver
© 2003 IBM Corporation 10
IBM eServer iSeries
How Does Remote Jrn Differ From Prior HA Support? Less CPU overhead than prior support Shifts work to the backup/target machine Frees up CPU cycles on production/source machine
Puts packets on the wire faster More efficient transport mechanism Microcode to microcode transport (below MI) Memory to memory transfer
DB images can be sent to target machine in real time
© 2003 IBM Corporation 11
IBM eServer iSeries
Net: Better Plumbing !
© 2003 IBM Corporation 12
IBM eServer iSeries
Data Replication players Lakeview Technologies (MIMIX) Vision Solutions (OMS)
Traditional HA BP's (offer both RJ and non-RJ versions)
Data Mirror Corp. (iCluster) Maximum Availability (*NoMax) Employ Pure Remote Journal
iTERA (Echo2) IBM's Data_Propagator product (for Data Warehousing) Others pursing independent solutions for smaller shops e.g. Sloan's TAA tools from Barsa Consulting
© 2003 IBM Corporation 13
IBM eServer iSeries
What is Remote Journal? An enabler New plumbing for HA Vendor products NOT a replacement for HA Vendor products ! Someone still needs to replay the resulting remote Journal entries
© 2003 IBM Corporation 14
IBM eServer iSeries
Setting it up
Adam Stallman
© 2003 IBM Corporation 15
IBM eServer iSeries
How Do I Set It Up? No application changes required The building blocks: Can code to APIs Can use CL Commands HA Vendor products will often orchestrate these steps for you
© 2003 IBM Corporation 16
IBM eServer iSeries
CL Commands
Three CL Commands Add Rmt Jrn Rmv Rmt Jrn Chg Rmt Jrn
© 2003 IBM Corporation 17
IBM eServer iSeries
Examples
ADDRMTJRN
RDB(JOSNA)
SRCJRN(TESTRJ/J1) TGTJRN(*SRCJRN) RMTRCVLIB(*SRCRCVLIB) RMTJRNTYPE(*TYPE1) MSGQ(QSYSOPR) DLTRCV(*NO)
Defining the Target
CHGRMTJRN RDB(JOSNA) SRCJRN(TESTRJ/J1) TGTJRN(*SRCJRN) JRNSTATE(*ACTIVE) DELIVERY(*SYNC) STRJRNRCV(*ATTACHED)
The Handshake
Notice that you drive the whole shebang from the production sys
© 2003 IBM Corporation 18
IBM eServer iSeries
So.. I've got two separate Journals on my Target machine ? (both a Rmt Jrn and a regular/local Jrn). How can I keep from getting confused ?
© 2003 IBM Corporation 19
IBM eServer iSeries
Redirection: Using an Alt Lib
RJ: Redirected Library
New
ALT_Lib
SQL Table
SQL Table Replica
Journal
Comm Line
Remote Receiver
REG_Lib
REG_Lib Receiver Source System
Local Receiver
Target System © 2003 IBM Corporation 20
IBM eServer iSeries
Redirecting a Remote Journal
How do you redirect a remote journal ?
Ahh... Name conflict resolution through redirection !
This will be your redirected (alternative) Library
1. On target: CRTLIB LIB(ALT_LIB) 2. ADDRDBDIRE RDB(RDBNAME) RMTLOCNAME(SYSTEM *IP) where RDBNAME matches the *LOCAL RDB Entry on the Target and SYSTEM matches the Target's System Name or IP address 3. ADDRMTJRN RDB(RDBNAME) SRCJRN(REG_LIB/J1) TGTJRN(ALT_LIB/J1) RMTRCVLIB(ALT_LIB) This command will create the remote journal in the Alternative (Redirected) library ALT_LIB on the target system. 4. CHGRMTJRN RDB(RDBNAME) SRCJRN(REG_LIB/J1) TGTJRN(ALT_LIB/J1) JRNSTATE(*ACTIVE) This command will activate remote journal and create the journal receiver in library ALT_LIB on the target system. © 2003 IBM Corporation 21
IBM eServer iSeries
API Interfaces
Three primary APIs QJO Add Remote Journal
Any coding examples ?
QJO Remove Remote Journal QJO Change Journal State
Coding examples at: http://www.as400.ibm.com/db2/db2code.htm © 2003 IBM Corporation 22
IBM eServer iSeries
How can I see the status of my Rmt Jrn connection ?
© 2003 IBM Corporation 23
IBM eServer iSeries
WRKJRNA - F16 Work with Remote Journal Information Journal
. . . . . . :
J1
Library
. . . . . . :
Journal type . . . . : *LOCAL Journal state Remote journal type : Delivery mode Local journal . . . : Source journal Library . . . . . : Library . . ASP group . . . . : ASP group . System . . . . . . : System . . . Redirected receiver library . . . . . . . . . . . . . . Number of remote journals . . . . . . . . . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
: : : : : : : :
TESTRJ *ACTIVE
*NONE 1
Type options, press Enter. 13=Activate 14=Inactivate ... --------------------Remote-------------------Relational Journal Delivery Opt Database Journal Library State Mode SYSTEM1 J1 TESTRJ *INACTIVE Bottom ===> F3=Exit
F4=Prompt
F5=Refresh
F9=Retrieve
F12=Cancel
F23=More options
F6=Work with remote journal list
© 2003 IBM Corporation 24
IBM eServer iSeries
Identifying the target machine
So that's how Rmt Jrn figures out which remote machine to use !
Rmt Jrn uses RDB (Relational Database) entries to locate the remote systems ADDRDBDIRE RDB(RDBNAME) RMTLOCNAME(SYSTEM *IP) Where RDBNAME matches the *LOCAL RDB Entry on the Target and SYSTEM matches the Target System's Name or IP address
© 2003 IBM Corporation 25
IBM eServer iSeries
What Communication connections can I use ?
Six transport mechanisms: Opticonnect (Optical bus) Virtual (LPAR to LPAR) opticonnect TCP/IP SNA (APPC) ATM HSL
Everything but a tin can and a string
Chapter 6 of Journal Performance Redbook documents the most recent Remote Journal speeds over various communication lines © 2003 IBM Corporation 26
IBM eServer iSeries
Resources Two Redbooks: Original Remote Journal Redbook: AS/400 Remote Journal Function for High Availability and Data Replication Found at www.ibm.com/redbooks
Striving For Optimal Journal Performance Remote Journal speeds in Chapter 6
Recent Magazine Articles: Eight "Tips" for Optimal Journal Performance Jan and Feb '03 issues of iSeries News
ISeries Information Center: Found at www.ibm.com/eserver/iseries/infocenter © 2003 IBM Corporation 27
IBM eServer iSeries
Remote Journal Topologies
Larry © 2003 IBM Corporation 28
IBM eServer iSeries
Broadcast mode of Remote Journal Target 1
Source Sys
Target 2
Target 255
Can broadcast to up to 255 secondaries/targets © 2003 IBM Corporation 29
IBM eServer iSeries
Employing a centralized Regional Hub Source Sys 1
Hub
Source Sys 2
Source Sys 3
© 2003 IBM Corporation 30
IBM eServer iSeries
Cascade mode of Remote Jrn Source Sys
Syn c First Target
Asy nc Second Target
Can cascade without limit © 2003 IBM Corporation 31
IBM eServer iSeries
Let's compare Transport modes
Sync
Async
© 2003 IBM Corporation 32
IBM eServer iSeries
Async Remote Journal Your application doesn't wait SQL Table Applications
Journal
SLIC Async Sending Task
Receiver
Source System
Remote Receiver
Async Remote Journaling
Target System
© 2003 IBM Corporation 33
IBM eServer iSeries
Sync Remote Journal
Your application pauses until the response is received
SQL Table Applications
Journal
Acknowledgment response Receiver
Remote Receiver Remote Journal
Source System
Target System
© 2003 IBM Corporation 34
IBM eServer iSeries
Pros and Cons of Sync vs Async mode Sync mode Big advantage: no trapped transactions All Journal entries reach the target system in real time
JRN image reaches target machine BEFORE reaching disk of source Especially practical with Virtual OptiConnect, HSL, and high-speed ATM Most practical over short distances
Bene: No commit transactions are viewed as complete by your customer, then rolled back by your Take-over (target) machine !
Sync
Async
© 2003 IBM Corporation 35
IBM eServer iSeries
Pros and Cons of Sync vs Async mode
Async mode
Sync
Async
Advantage: slightly less performance overhead Probably good choice for heavy batch jobs or slow communication lines Customary choice when source and target machines are separated by substantial distances
Drawback: some risk of lost/trapped recent transactions not sent across Same kinds of risks as vintage async Business Partner solutions In-house for slow lines we've seen up to 1.5 sec latency But only 5 Msec latency for virtual Opti-connect and HSL
Drawback: Probably tougher to recalibrate after a fail-over
Traffic Jam possible
© 2003 IBM Corporation 36
IBM eServer iSeries
Ok, I see the HA benefit of Rmt Jrn. But... what else can I do with this technology ?
© 2003 IBM Corporation 37
IBM eServer iSeries
Alternative uses of Remote Journal
Adam © 2003 IBM Corporation 38
IBM eServer iSeries
Alternative Rmt Jrn uses:
Alternative #1 A remote Vault
SQL Table Applications
Journal
Remote system: Less expensive. Head to Hot Site w/ tapes Receiver
Remote Receiver Remote Journal
Source System
Target System
© 2003 IBM Corporation 39
IBM eServer iSeries
Off-Site Vaulting Example Production System
Alternate System
(large system $$$$$)
(smaller system $)
Remote Journal Connection
PF's
...
...
Saving from an alternate system reduces the impact on the production system. This also allows you to always have access to the latest changes to apply even if you haven't saved your Journal Receivers lately! (just the delta) © 2003 IBM Corporation 40
IBM eServer iSeries
Alternative Rmt Jrn uses:
Alternative #2 Refresh a Data Warehouse
SQL Table
Data Propagator
Applications
Data Warehouse
Harv est
Journal
Receiver
Remote Receiver Remote Journal
Source System
Target System
© 2003 IBM Corporation 41
IBM eServer iSeries
Replaying from a Rmt Jrn
Replicated critical files
SQL Table
Replay Step ApyJ rnCh gX
Applications
Alternative #3 Warm Site Hourly refresh
Journal
Local Receiver
Receiver
Remote Journal Remote Receiver
Source System
How are you going to make the Rmt Jrn look like a "Local" Jrn ?
Target System
© 2003 IBM Corporation 42
IBM eServer iSeries
Replaying from a Rmt Jrn
Fooling Journal into thinking the Rmt Jrn is a "Local"
ApyJ rnCh gX
Replicated critical files
Local Receiver
Remote Receiver
Rest ore
Save_File
Target System
© 2003 IBM Corporation 43
IBM eServer iSeries
Can I APYJRNCHG from a Remote Journal Receiver ? Yes !
See online magazine article: MC Press online 1/21/03 Remote Journaling and Data Recovery by Chris Hird
© 2003 IBM Corporation 44
IBM eServer iSeries
Got any suggestions regarding how to make Rmt Jrn have minimal overhead ?
© 2003 IBM Corporation 45
IBM eServer iSeries
Remote Journal Performance Considerations
Larry Youngren
© 2003 IBM Corporation 46
IBM eServer iSeries
What happens AFTER your source machine is fixed ?
© 2003 IBM Corporation 47
IBM eServer iSeries
Swap Back to Production SQL Table HA Vendor Software
SQL Table
Harv est
Journal
Receiver
Remote Receiver
Source System
Target System
Delta
Catch-up phase
Delta
© 2003 IBM Corporation 48
IBM eServer iSeries
The secrets of optimal Rmt Jrn performance Watch that pesky extra journal on the target machine
© 2003 IBM Corporation 49
IBM eServer iSeries
Performance tuning on the Target side
Source System
Target System
© 2003 IBM Corporation 50
IBM eServer iSeries
Secret #1
Speeding up Target side Remote Journal via Journal Caching
© 2003 IBM Corporation 51
IBM eServer iSeries
Journal Bundling HA Replay Job #1
HA Replay Job #3
HA Replay Job #2
PF 3 PF 2
PF 1
Main Memory Buffer
128K Buffer Journal
Main Memory Journal Caching
Delta
Journal Receiver
IOA
Known as a journal "Bundle"
User ASP
Write Cache
10
© 2003 IBM Corporation 52
IBM eServer iSeries
Qty transactions unapplied on Target Thousands
How far "behind" are the replay jobs on the Target ? Retail store Target backlog Teraplex model 840 w/ 11-CPU partition HA BP software harvesting on Source vs. Rmt Jrn 900 793,601 800 700 600 500 400 300 200 100 0 Trgt Jrn/ BP
The Caching approach
565,518
290
Async RJ
Best case for Replay Only 0.1 Sec behind.
Jrn Cache
Shows worst case backlog unapplied on Target
Conclusion: Use of the Journal Caching PRPQ (5799-BJC) on the target machine helps HA Vendor replay software from falling behind during especially aggressive journal-intensive processing such as batch jobs. © 2003 IBM Corporation 53
IBM eServer iSeries
Potential Performance Benefits of Journal Caching (For Target Machine's Keep-up Mode)
JrnCache(*Yes) Apply rate on Target machine
Before Caching
600,000 transactions/Hr
With Caching on target
2,400,000 transactions/Hr
Source System Target System © 2003 IBM Corporation 54
IBM eServer iSeries
What Else Can I Do To Speed It Up?
Target machine suggestions:
Software (Journal) Caching Enable Journal Caching on the Target machine
Elbow Room Assure base pool on Target machine has enough elbow room At least 10 Meg for catch-up/reconnect phase per RJ connection
Sufficient Disk arms Assure ASP housing target receiver has at least as many disk arms as matching receiver from the source system
Write cache Configure lots of IOA write cache - - especially on the target machine Choices: 26 Meg 104 Meg 757 Meg
Newest
© 2003 IBM Corporation 55
IBM eServer iSeries
Don't skimp on the write cache
Model (older) NEW !!
Journal
Write Cache 2763 10 Meg 2782 40 Meg
(older) 2778 26 (104) Meg NEW !! 2757 235 (757) Meg Receiver
IOP
104 Meg of Write Cache (older vintage IOPs have less)
Feb '03
Write Cache
Up to 1 Million Jrn Ents/Minute
. . . Need enough Arms © 2003 IBM Corporation 56
IBM eServer iSeries
Have you got sufficient IOA Write cache ?
See Appendix "A" Shows algorithm/Query to analyze write cache overflow
Found at www.ibm.com/redbooks © 2003 IBM Corporation 57
IBM eServer iSeries
V5R2!
Secret #2
Speeding up Remote Journal via Standby mode
© 2003 IBM Corporation 58
IBM eServer iSeries
Standby mode CHGJRN...
JRNSTATE(*STANDBY)
V5R2!
SQL Table HA vendor Software
SQL Table
Harv est
Journal
Receiver
Remote Receiver Remote Journal
Source System
Standby mode
Target System
© 2003 IBM Corporation 59
IBM eServer iSeries
Switching out of Standby mode
Role Swap:
Applications
CHGJRN ... JRNSTATE(*ACTIVE) HA Vendor Software
Replicated SQL Table Harv est
Journal
Receiver
Source System
Remote Receiver
Active mode
Target System
Becomes Active
Delta
© 2003 IBM Corporation 60
IBM eServer iSeries
How far "behind" are the replay jobs on the Target ?
Qty transactions unapplied on Target Thousands
Retail store backlog on Target Teraplex model 840 w/ 11-CPU partition HA BP software harvesting on Source vs. Rmt Jrn 900 793,601 800 700 600 500 400 300 200 100 0 BP/Trgt Jrn
565,518
Standby Mode
588
Async RJ
Standby
Shows worst case backlog unapplied on Target © 2003 IBM Corporation 61
IBM eServer iSeries
Potential Standby mode Benefit: Very little performance overhead on the target machine while in standby mode Much more rapid failover in the event of a failure OS/400 merely flips one flag in the Journal rather than writing to disk thousands of Database file headers
Mode transition rates during Role-swap as fast as... How long does it take to re-enable Journaling on the Target during a Role-swap ?
© 2003 IBM Corporation 62
IBM eServer iSeries
V5R2!
Role-Swap on the Target
Start Journal vs. Standby to Active Transition 2500
Seconds
2000 1500
The V5R2 advantage
1000 500 0 10,000 files Start Journaling (STRJRNPF)
25,000 files Standby to Active Transition (CHGJRN)
© 2003 IBM Corporation 63
IBM eServer iSeries
Summary
Net: V5R2!
Order & Install Option 42 Option 42 encompasses both: JrnCache(*Yes) JrnState(*Standby)
© 2003 IBM Corporation 64
IBM eServer iSeries
Performance tuning on the Source side
Source System
Target System
© 2003 IBM Corporation 65
IBM eServer iSeries
Secret #3
Sending fewer bytes down the communication line (withholding the hidden entries)
© 2003 IBM Corporation 66
IBM eServer iSeries
Overlapped writes with CHGJRN ... RcvSizOpt(*RMVINTENT) Main Memory AP Journal Entry
Hidden Jrn entries
PF
IOP1
IOP2
Journal Entry
Write Cache
PF JOE
Write Cache
.. .
Only these must be sent via RJ
AP
JOE
User ASP
© 2003 IBM Corporation 67
IBM eServer iSeries
Secret #4
Reduce the fixed overhead per Journal Entry (Sending only the data)
© 2003 IBM Corporation 68
IBM eServer iSeries
Customizing what we collect Change Journal (CHGJRN) Type choices, press Enter. Journal . . . . . . . . . . . . Library . . . . . . . . . . . Journal receiver: Journal receiver . . . . . . . Library . . . . . . . . . . Receiver size options . . . . . + for more values Minimize entry specific data . . Journal caching . . . . . . . . Fixed length data . . . . . . .
> JRN1 > JRNLIB > *GEN
> > > > + for more values > Text 'description' . . . . . . .
F3=Exit F4=Prompt F24=More keys
F5=Refresh
*SAME *SAME *SAME *JOB *USR *PGM _ *SYSSEQ *THD *SAME
F12=Cancel
Name Name, *LIBL, *CURLIB Name, *SAME, *GEN Name, *LIBL, *CURLIB *SAME, *NONE, *RMVINTENT... *SAME, *NONE, *FILE, *DTAARA *SAME, *NO, *YES *JOBUSRPGM, *JOB, *USR...
V5R2!
More... F13=How to use this display
CHGJRN command with new FIXLENDTA parm © 2003 IBM Corporation 69
IBM eServer iSeries
Summary
Be Selective: Use the Goldilocks principle... Advise us to collect only what you truly need, no more and no less
© 2003 IBM Corporation 70
IBM eServer iSeries
Secret #5
Sending fewer bytes down the communication line (Sending only the delta)
© 2003 IBM Corporation 71
IBM eServer iSeries
Journaling Minimal Data A new parameter (MINENTDTA) is provided to allow specifying which object types are allowed to minimize their journal entries. Specified on: CRTJRN - Create Journal
CHGJRN - Change Journal
© 2003 IBM Corporation 72
IBM eServer iSeries
Journaling Minimal Data Journal Minimal Data is provided for Database Updates With Journal Minimal Data enabled only the essential changed bytes are written to the journal (not the whole record image). This can lead to: Reduction in the size of journal receivers Sending minimized journal entries remotely for object replication will also reduce comm line traffic.
The Remote Jrn advantage
© 2003 IBM Corporation 73
IBM eServer iSeries
In-house measurement
Interactive Job Reduction of Journal Receiver size Performance consequences
Batch Job
40% fewer bytes generated
60% fewer bytes generated
4.6% more thruput, 3.5% less CPU
3% reduction in batch run time
ESP Customer in Netherlands reported similar results
© 2003 IBM Corporation 74
IBM eServer iSeries
Summary: Secrets to Good Rmt Jrn performance On the Target System: Use Journal Caching CHGJRN ... JrnCache(*Yes)
Consider use of Journal Standby mode CHGJRN ... JrnState(*Standby)
On the Source System: Employ CHGJRN ... RcvSizOpt(*RmvIntEnt) Puts a "fork" in the road Inhibits sending Access Path journal images to target system Do this only if you have more than one disk arm in your User ASP
Consider use of Journal-Minimal Data CHGJRN ... MinEntDta(*File) Reduces quantity of traffic across the communication line Speeds up Re-play processing by HA vendor software
Consider use of MINFIXLEN or FIXLENDTA settings © 2003 IBM Corporation 75
IBM eServerJ iSeriesJ
Session:
Webcast
Remote Journaling = Better Plumbing
Larry Youngren Adam Stallman 8 Copyright IBM Corporation, 2003. All Rights Reserved. This publication may refer to products that are not currently available in your country. IBM makes no commitment to make available any products referred to herein.
76
IBM eServer iSeries
Trademarks and Disclaimers 8 IBM Corporation 1994-2003. All rights reserved. References in this document to IBM products or services do not imply that IBM intends to make them available in every country. The following terms are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both:
AS/400 AS/400e eServer
IBM IBM (logo) iSeries OS/400
Other company, product or service names may be trademarks or service marks of others. Information is provided "AS IS" without warranty of any kind. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Information in this presentation concerning non-IBM products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-IBM products. Questions on the capability of non-IBM products should be addressed to the supplier of those products. All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Contact your local IBM office or IBM authorized reseller for the full text of the specific Statement of Direction. Some information in this presentation addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in IBM product announcements. The information is presented here to communicate IBM's current investment and development activities as a good faith effort to help with our customers' future planning. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here.
© 2003 IBM Corporation 77
