Backup and Recovery-Exchange 2010 DAG
Backup and Recovery – Exchange 2010 DAG By Francis Sabarinath [frasa03]
1
Backup and Recovery-Exchange 2010 DAG
Table of Contents BACKUP AND RECOVERY – EXCHANGE 2010 DAG ........................................................................................ 3 1. Exchange Backup................................................................................................................................... 3 2. Backup Sequence Diagram – using Store Writer .................................................................................. 3 3. Backup Sequence Diagram – using Replica Writer ............................................................................... 4 4. Difference between VSS backup of Exchange 2010 using Windows Server Backup and that done by CA ARCserve Agent for Microsoft Exchange ............................................................................................. 4 5. Important Tool for Troubleshooting Exchange Backup issues – MS Event Viewer .............................. 5 6. User Problems related to Backup ......................................................................................................... 8 6.1 User Problem 1 -Exchange Database Level Backup fails ................................................................. 8 6.2 User Problem 2 -Exchange Database Level Backup fails - Client halt error .................................... 8 6.3 User Problem 3 - Transaction Logs not truncated .......................................................................... 9 6.4 User Problem 4 – Unable to find Exchange Database .................................................................. 10 7. Exchange Restore ................................................................................................................................ 10 8. Restore Sequence Diagram – only Store Writer is used and not Replica Writer ................................ 11 9. User Problems related to Restore....................................................................................................... 11 9.1 User Problem 1-Restore using Recovery Database – How to extract the Mailbox ...................... 11 9.2 User Problem 2 –Unable to find the Recovery Database ............................................................. 12
2
Backup and Recovery-Exchange 2010 DAG
BACKUP AND RECOVERY – EXCHANGE 2010 DAG 1. Exchange Backup Microsoft provides 2 ways of backup – legacy streaming and VSS. Legacy streaming backup is not supported with Exchange 2010. The VSS communicates with the Windows Server File system and with the mass storage device driver through a third-party (or customer) provider. The backup/restore application is also called the VSS Requestor. The requestor communicates with the VSS to obtain information about the Exchange 2010, to command the creation of shadow copies and to gain access to the data for backup. To support the VSS, Exchange 2010 includes an Exchange writer that it built into the Exchange Store. The Exchange Writer coordinates with the Exchange Store, to freeze and dismount the database before backing it up, and then to unfreeze and mount the database after the backup is complete. Backup from the active copy uses the Store Writer, and backup from the replica (copy database) uses the Replica Writer. It is good to understand the Backup sequence using the Store and the Replica writer which is diagrammatically provided in the following section.
2. Backup Sequence Diagram – using Store Writer
3
Backup and Recovery-Exchange 2010 DAG
3. Backup Sequence Diagram – using Replica Writer
Ref: http://msdn.microsoft.com/en-us/library/exchange/aa579076(v=exchg.140).aspx
4. Difference between VSS backup of Exchange 2010 using Windows Server Backup and that done by CA ARCserve Agent for Microsoft Exchange The sequence of backup will be the same except that a physical consistencies check that will be initiated by ARCserve Exchange Agent. Our agent uses the exported functions from chksgfiles.dll to check the consistency. The CHKSGFILES API ships as part of Exchange 2010. The API can be used on a computer that has Mailbox server role installed, or a computer that has the Administrator-only install. CHKSGFILES is available only as a 64-bit DLL in Exchange 2010. By default, the CHKSGFILES DLL is installed in the C:\Program Files\Exchange 2010\bin directory. The backup/restore application may choose to validate the integrity of a shadow copy backup set. During a backup operation managed by the Volume Shadow Copy Service, there is no opportunity for Microsoft Exchange Server 2010 to read each database file in its entirety and verify its checksum integrity. Therefore, database and transaction log file integrity should be verified by the backup application. Microsoft recommends that the backup application verify the physical consistency of the
4
Backup and Recovery-Exchange 2010 DAG shadow copy set prior to informing the Exchange Writer that the backup is complete. Upon successful backup, Exchange store updates the headers of the backed-up databases to reflect the last successful backup times and removes transaction logs from the server that are no longer required to roll forward from the last successful backup. In order to provide better granularity and control during the consistency check process, Exchange 2010 includes a set of unmanaged public APIs packaged into a DLL (chksgfiles.dll). The CHKSGFILES DLL is installed with all Exchange server roles as well as part of the Admin-only option to enable running consistency checks off of an Exchange server on a proxy backup machine. More Reference: http://msdn.microsoft.com/en-us/library/bb204049.aspx http://msdn.microsoft.com/en-us/library/bb204219.aspx
5. Important Tool for Troubleshooting Exchange Backup issues – MS Event Viewer Microsoft Event Viewer (application events) is an important tool to be used for troubleshooting an Exchange Backup. All the events will be recorded in sequence – from the Backup initiation till the end. Sequence of the Exchange Backup can be understood by looking at the application event during an Exchange VSS backup. Provided below is a sample set of application events, from a backup taken from the replica, and it is important to understand this sequence for troubleshooting purpose: Replica Events: SNo
TIME
SOURCE
1
3:27:34 AM
MSExchangeRepl
EVENT ID 2021
TASK Category Exchange VSS Writer
DESCRIPTION
2
3:27:38 AM
MSExchangeRepl
2110
Exchange VSS Writer
The Microsoft Exchange VSS Writer instance cfda266f-d071-47fd-92958ec31213f378 has successfully prepared for a full or a copy backup of database 'FRANCISTEST'. The following database will be backed up: FRANCISTEST.
3
3:27:38 AM
MSExchangeRepl
2023
Exchange VSS Writer
The Microsoft Exchange Replication service VSS Writer (Instance cfda266fd071-47fd-9295-8ec31213f378) successfully prepared for backup.
4
3:27:49 AM
MSExchangeIS Public Store
9539
General
General "The Microsoft Exchange Information Store database """" was stopped. “
5
3:27:49 AM
ESE BACKUP
960
General
6
3:27:49 AM
MSExchangeRepl
2027
Exchange VSS Writer
msexchangerepl (2444) This computer is performing a surrogate backup. The master server is USSDWIN2K8R1. The Microsoft Exchange VSS Writer instance cfda266f-d071-47fd-92958ec31213f378 has successfully frozen the databases.
7
3:27:49 AM
MSExchangeRepl
2025
Exchange VSS Writer
The Microsoft Exchange Replication service VSS Writer (Instance cfda266fd071-47fd-9295-8ec31213f378) successfully prepared for a snapshot.
8
3:27:49 AM
MSExchangeRepl
2027
Exchange VSS Writer
The Microsoft Exchange VSS Writer instance cfda266f-d071-47fd-92958ec31213f378 has successfully frozen the databases.
The Microsoft Exchange VSS Writer has successfully collected the metadata document in preparation for backup.
5
Backup and Recovery-Exchange 2010 DAG 9
3:27:49 AM
MSExchangeRepl
2029
Exchange VSS Writer
The Microsoft Exchange VSS Writer instance cfda266f-d071-47fd-92958ec31213f378 has successfully thawed the databases.
10
3:27:50 AM
MSExchangeRepl
2035
Exchange VSS Writer
The Microsoft Exchange Replication service VSS Writer (Instance cfda266fd071-47fd-9295-8ec31213f378) has successfully processed the postsnapshot event.
11
3:27:54 AM
Storage Group Consistency Check
100
Initialization
Instance 1: A physical consistency check has been successfully initiated. Database '\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2\Program Files\Microsoft\Exchange Server\V14\Mailbox\FRANCISTEST\FRANCISTEST.edb' and the transaction log files in '\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2\Program Files\Microsoft\Exchange Server\V14\Mailbox\FRANCISTEST\' with a base name of 'E05' will be validated.
12
3:27:54 AM
Storage Group Consistency Check
200
Database Header Validation
Instance 1: Database headers have been successfully validated. All databases are in a dirty shutdown state. To bring these databases to a clean shutdown state, log generations 3 (0x00000003) to 3 (0x00000003) will be required.
13
3:27:57 AM
Storage Group Consistency Check
300
Log File Validation
Instance 1: Transaction log files in '\\? \GLOBALROOT\Device\HarddiskVolumeShadowCopy2\Program Files\Microsoft\Exchange Server\V14\Mailbox\FRANCISTEST\' with a base name of 'E05' have been successfully validated. No errors were detected.
14
3:27:57 AM
ESE
103
General
"caagstart (788) Instance 1: Log File Integrity Check (\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2\Program Files\Microsoft\Exchange Server\V14\Mailbox\FRANCISTEST\E05): The database engine stopped the instance (0). Internal Timing Sequence: [1] 0.000, [2] 0.000, [3] 0.000, [4] 0.000, [5] 0.000, [6] 0.000, [7] 0.000, [8] 0.000, [9] 0.000, [10] 0.000."
15
3:27:57 AM
ESE
105
General
"caagstart (788) Instance 1: Log File Integrity Check (\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2\Program Files\Microsoft\Exchange Server\V14\Mailbox\FRANCISTEST\E05): The database engine started a new instance (0). (Time=0 seconds) Internal Timing Sequence: [1] 0.000, [2] 0.000, [3] 0.000, [4] 0.000, [5] 0.000, [6] 0.000, [7] 0.000, [8] 0.000, [9] 0.000, [10] 0.000, [11] 0.000.“
16
3:27:57 AM
ESE
102
General
"caagstart (788) Instance 1: Log File Integrity Check (\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2\Program Files\Microsoft\Exchange Server\V14\Mailbox\FRANCISTEST\E05): The database engine (14.00.0692.0000) is starting a new instance (0).
17
3:27:57 AM
400
Termination
Instance 1: The physical consistency check has completed successfully. All files were validated and no errors were detected.
18
3:28:47 AM
Storage Group Consistency Check MSExchangeRepl
2021
19
3:29:03 AM
ESE BACKUP
963
Exchange VSS Writer General
The Microsoft Exchange VSS Writer has successfully collected the metadata document in preparation for backup. msexchangerepl (2444) The surrogate backup procedure to USSDWIN2K8R1 has been successfully completed.
20
3:29:03 AM
MSExchangeRepl
2046
Exchange VSS Writer
"The Microsoft Exchange Replication service VSS Writer instance cfda266f-d071-47fd-9295-8ec31213f378 has successfully completed the backup of database 'FRANCISTEST'. Database log truncation has been requested for this database. Log truncation will occur on the active copy after the next log generation is created. Log truncation will occur automatically on the passive copies after that log file is copied."
21
3:29:03 AM
MSExchangeRepl
2033
Exchange
The Microsoft Exchange Replication service VSS Writer (Instance cfda266f-
6
Backup and Recovery-Exchange 2010 DAG
22
3:29:03 AM
MSExchangeRepl
2037
VSS Writer
d071-47fd-9295-8ec31213f378) has successfully processed the backup completion event.
Exchange VSS Writer
The Microsoft Exchange Replication service VSS Writer (Instance cfda266fd071-47fd-9295-8ec31213f378) backup has been successfully shut down.
Active Events: SNo 1
TIME 3:27:51 AM
SOURCE ESE
EVENT ID 210
TASK Category Logging/Recovery
DESCRIPTION "Information Store (3400) FRANCISTEST: A full backup is starting.
2
3:27:51 AM
ESE BACKUP
910
General
3
3:29:04 AM
MSExchangeRepl
3156
Service
Information Store (3400) A full surrogate backup is starting. This server is the master. The requestor is USSDWIN2K8R1. Active Manager successfully mounted database FRANCISTEST on server ussdwin2k8r1.gotir.ca.com.
4
3:29:04 AM
ESE BACKUP
913
General
5
3:29:04 AM
ESE
213
Logging/Recovery
Information Store (3400) The surrogate backup procedure has been successfully completed by USSDWIN2K8R1. "Information Store (3400) FRANCISTEST: The backup procedure has been successfully completed.
Will the transaction logs be purged from the Active or the Replica – when an Exchange backup is performed via the Replica? The log truncation will happen on the Active Database only and not from the Copy databases. Once the backup is completed from the Replica, it will signal the Active database. The logs will be truncated from the Active database and sync will take place with the copies, which will affect the transaction logs on the Copy databases. Backup initiated transaction log file truncation will be triggered based on the type of backup chosen. In non-DAG configurations, the Store Writer will truncate the transaction log files at the completion of successful Full or Incremental backups. In DAG replicated configurations, log truncation will be delayed by the Replication service until all necessary log files are replayed into all other copies. The Replication service will delete the backed up log files both from the active and the copy log file paths after it verifies that the to-be-deleted log files have successfully been applied to the copy database and both active database and the database copies checkpoint has passed the log files to be deleted. Log truncation doesn't occur on the active mailbox database copy when one or more passive copies are suspended. If your planned maintenance activities are going to take an extended period of time (for example, several days), you may have considerable log file buildup. To prevent the log drive from filling
7
Backup and Recovery-Exchange 2010 DAG up with transaction logs, you can remove the affected passive database copy instead of suspending it. When the planned maintenance is completed, you can re-add the passive database copy.
6. User Problems related to Backup 6.1 User Problem 1 -Exchange Database Level Backup fails From the Application event, check at which step the backup failed. Run a Windows Server Backup on that mailbox database to narrow down the case. Refer the link below on how to run a VSS backup of Exchange using Windows Server Backup: http://technet.microsoft.com/en-us/library/dd876854(v=exchg.141).aspx
6.2 User Problem 2 -Exchange Database Level Backup fails - Client halt error The following errors will be observed from the Application event and it will occur during the backup (after sequence 18 from the Application Events in section 5) Active Events: Event ID: 215 Source: ESE Category: Logging/Recovery Level: Error Message: "Information Store (3084) EXCHDB1_08: The backup has been stopped because it was halted by the client or the connection with the client failed. Replica Events: Source: ESE BACKUP Event ID: 914 Category: General Level: WarningMessage: General Information Store (3084) The surrogate backup by VMEXCHANGEMBS1A has stopped with error 0xFFFFFFFF.
In this case as well, run a Windows Server Backup of Exchange, as explained above to narrow down the case. If the Windows Server backup is successful, then do the steps below to disable TCP Offloading TCP Offloading has to be disabled on the NIC from the Backup Server and the Exchange Nodes. On the Backup servers: IPv4 Checksum Offload Off IPv4 Large Send Offload Disabled IPv6 Checksum Offload Off IPv6 Large Send Offload Disabled
8
Backup and Recovery-Exchange 2010 DAG Receive Side Scaling Disabled On the Exchange Nodes: IPv4 Checksum Offload Disabled IPv4 Giant TSO Offload Disabled IPv4 TSO Offload Disabled IPv4 TCP Segmentation Offload Disabled Offload IP Options Disabled TCP Checksum Offload (IPv4) Disabled TCP Checksum Offload (IPv6) Disabled
6.3 User Problem 3 - Transaction Logs not truncated Truncation of Transaction logs takes place after a full or an incremental backup of the Exchange Database. If there is an issue where the transaction logs are not getting truncated, then the step is to run a Windows Server backup of the Exchange Database to check if the logs are getting truncated. If the transaction log truncation does not take place from the Windows Server Backup of the Exchange, then check the status of the copy databases, in case of an Exchange 2010 DAG. Log truncation doesn't occur on the active mailbox database copy when one or more passive copies are suspended. If your planned maintenance activities are going to take an extended period of time (for example, several days), you may have considerable log file buildup. To prevent the log drive from filling up with transaction logs, you can remove the affected passive database copy instead of suspending it. When the planned maintenance is completed, you can re-add the passive database copy.
9
Backup and Recovery-Exchange 2010 DAG 6.4 User Problem 4 – Unable to find Exchange Database
When the backup is initiated, you will get an error as “E3577 Cannot find DB on Exchange server” In such cases, run the adsiedit.msc and use the same domain controller and credentials as provided in the ARCserve Backup UI. Mostly, there will be ldap connectivity issues, or the Exchange containers cannot be seen. The solution is to use a different domain controller or credentials, so that the Exchange objects can be viewed correctly. A sample screenshot of the adsiedit.msc is provided below:
7. Exchange Restore During a restore, the backup/restore application instructs the Exchange Writer to coordinate with the Exchange Store to dismount the database, replace the database files, mount the database and then replay the transaction logs as needed. When restoring, the requestor also communicates with the VSS to prepare the system for the restore, and then to put the data back on to the mass storage device. The backup/restore application is also responsible for working with the Windows Server to read data from and write data to the backup storage media, whether a tape archive, a storage area network, or the other backup medium. The restored database can be mounted either as a regular, active database, or as the Exchange 2010 Recovery Database. Only one mounted database can be designated as a recovery database on each Exchange Server.
10
Backup and Recovery-Exchange 2010 DAG
8. Restore Sequence Diagram – only Store Writer is used and not Replica Writer
Ref: http://msdn.microsoft.com/en-us/library/exchange/bb204079(v=exchg.140).aspx
9. User Problems related to Restore 9.1 User Problem 1-Restore using Recovery Database – How to extract the Mailbox After the mailbox database is restored to the Recovery Database, ensure that it is mounted. Mount the Recovery Database either from Exchange Management Console or from Exchange Management Shell by running the command Mount-Database "Recovery Database"
Make sure the destination mailbox is enabled and existing in the Exchange Server. Make sure the database corresponding to the destination mailbox is mounted. Before proceeding to recover the mailbox contents from Recovery Database, run the following command to get the details of mailboxes available in the Recovery Database. Get-MailboxStatistics –Database "Recovery Database"
11
Backup and Recovery-Exchange 2010 DAG The above command can be run to make sure that the mailbox that we are trying to restore is existing in the Recovery Database.
Restore a single mailbox 1.
Run the following command to restore the content of mailbox 'production mailbox' in the Recovery Database to the production mailbox: Restore-Mailbox –Identity "production mailbox" –RecoveryDatabase "Recovery Database"
2. 3.
Say 'Yes' for the confirmation asked. Restore process will start immediately. When completed, you will get statistics about the restore mailbox job. You can see the information about the source and target mailbox database etc.
Restore a single mailbox to a different mailbox 1.
Run the following command to restore the content of 'production mailbox' in the Recovery Database to 'test mailbox' under the folder 'Recovered Contents Folder' within 'spare mailbox': Restore-Mailbox -Identity "test mailbox" -RecoveryDatabase "Recovery Database" RecoveryMailbox "production mailbox" -TargetFolder "Recovered Contents Folder" NOTE : The above command will create a new folder "Recovered Contents Folder" inside "test mailbox" and will restore the content of "production mailbox" into the new folder "Recovered Contents Folder"
2. 3.
Say 'Yes' for the confirmation asked. Restore process will start immediately. When completed, you will get statistics about the restore mailbox job. You can see the information about the source and target mailbox database etc.
http://msexchangeguru.com/2010/05/23/exchange2010-recovery-database/ http://technet.microsoft.com/en-us/library/ee332351(v=exchg.141).aspx
9.2 User Problem 2 –Unable to find the Recovery Database When a restore to a Recovery Database is attempted, it will fail with “unable to find the Recovery Database”. In order to narrow down the problem do the following: Have the adsiedit.msc installed. Then connect to the same Domain Controller and check if the containers can be displayed, and check if the “Recovery Database” created can be viewed from the Active Directory (Refer 6.4 User Problem 4 for a screenshot of the adsiedit.msc)
12
