ABSTRACT This Solution Guide provides an introduction to the concepts and architectural options available within the Federation Enterprise Hybrid Cloud solution. It should be used as an aid to deciding on the most suitable configuration for the initial deployment of a Federation Enterprise Hybrid Cloud solution. February 2016
Copyright © 2016 EMC Corporation. All rights reserved. Published in the USA. Published February 2016 EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. EMC2, EMC, Avamar, Data Domain, Data Protection Advisor, Enginuity, GeoSynchrony, Hybrid Cloud, PowerPath/VE, RecoverPoint, SMI-S Provider, Solutions Enabler, VMAX, Syncplicity, Unisphere, ViPR, EMC ViPR Storage Resource Management, Virtual Storage Integrator, VNX, VPLEX, VPLEX, Geo, VPLEX Metro, and the EMC logo are registered trademarks or trademarks of EMC Corporation in the United States and other countries. All other trademarks used herein are the property of their respective owners. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. Federation Enterprise Hybrid Cloud 3.5 Concepts and Architecture Guide Solution Guide Part Number H14719
2
Contents
Chapter 1
Executive Summary ............................................................. 5
Federation solutions ............................................................................................ 6 Document purpose .............................................................................................. 6 Audience ............................................................................................................ 6 Essential reading ................................................................................................ 6 Solution purpose ................................................................................................. 6 Business challenge .............................................................................................. 7 Technology solution............................................................................................. 7 Terminology ....................................................................................................... 8 We value your feedback! ...................................................................................... 9
Chapter 2
Cloud Management Platform Options .................................... 10
Overview ..........................................................................................................11 Cloud management platform components .............................................................11 Cloud management platform models ....................................................................14
Chapter 3
Network Topologies ............................................................ 21
Overview ..........................................................................................................22 Implications of virtual networking technology options .............................................22 Logical network topologies ..................................................................................24
Chapter 4
Single-Site/Single vCenter Topology..................................... 31
Overview ..........................................................................................................32 Single-site networking considerations ...................................................................32 Single-site storage considerations ........................................................................33 Recovery of cloud management platform ..............................................................36 Backup of single-site/single vCenter enterprise hybrid cloud ....................................36
Chapter 5
Dual-Site/Single vCenter Topology ....................................... 37
Overview ..........................................................................................................38 Standard dual-site/single vCenter topology ...........................................................38 Continuous availability dual-site/single vCenter topology ........................................39 Continuous availability network considerations ......................................................40 VPLEX Witness ...................................................................................................45 VPLEX topologies ...............................................................................................46 Continuous availability storage considerations .......................................................53 Recovery of cloud management platform ..............................................................56 Backup in dual-site/single vCenter enterprise hybrid cloud ......................................56 CA dual-site/single vCenter ecosystem .................................................................57
Chapter 6
Dual-Site/Dual vCenter Topology ......................................... 58
Overview ..........................................................................................................59
3
Contents
Standard dual-site/dual vCenter topology .............................................................59 Disaster recovery dual-site/dual vCenter topology ..................................................60 Disaster recovery network considerations .............................................................61 vCenter Site Recovery Manager considerations ......................................................69 vRealize Automation considerations ......................................................................72 Disaster recovery storage considerations ..............................................................73 Recovery of cloud management platform ..............................................................74 Best practices ....................................................................................................75 Backup in dual-site/dual vCenter topology ............................................................75 DR dual-site/dual vCenter ecosystem ...................................................................76
Chapter 7
Data Protection.................................................................. 77
Overview ..........................................................................................................78 Concepts...........................................................................................................79 Standard Avamar configuration............................................................................84 Redundant Avamar/single vCenter configuration ....................................................86 Redundant Avamar/dual vCenter configuration ......................................................90
Chapter 8
Solution Rules and Permitted Configurations ......................... 95
Overview ..........................................................................................................96 Architectural assumptions ...................................................................................96 VMware Platform Services Controller ....................................................................96 VMware vRealize tenants and business groups .......................................................98 EMC ViPR tenants and projects ............................................................................99 General storage considerations .......................................................................... 100 VMware vCenter endpoints ................................................................................ 100 Permitted topology configurations ...................................................................... 101 Permitted topology upgrade paths ...................................................................... 102 Bulk import of virtual machines ......................................................................... 103 DR dual-site/dual vCenter topology restrictions.................................................... 104 Resource sharing ............................................................................................. 106 Data protection considerations ........................................................................... 106 Software resources .......................................................................................... 106 Sizing guidance ............................................................................................... 106
Chapter 9
Conclusion ....................................................................... 107
Conclusion ...................................................................................................... 108
Chapter 10
References ....................................................................... 109
Federation documentation ................................................................................. 110
4
Chapter 1: Executive Summary
This chapter presents the following topics:
Federation solutions ............................................................................................ 6 Document purpose .............................................................................................. 6 Audience ............................................................................................................ 6 Essential reading ................................................................................................ 6 Solution purpose ................................................................................................. 6 Business challenge .............................................................................................. 7 Technology solution............................................................................................. 7 Terminology ....................................................................................................... 8 We value your feedback! ...................................................................................... 9
5
Chapter 1: Executive Summary
EMC II, Pivotal, RSA, VCE, Virtustream, and VMware form a unique Federation of strategically aligned businesses that are free to execute individually or together. The Federation businesses collaborate to research, develop, and validate superior, integrated solutions and deliver a seamless experience to their collective customers. The Federation provides customer solutions and choice for the software-defined enterprise and the emerging third platform of mobile, cloud, big data, and social networking. The Federation Enterprise Hybrid Cloud 3.5 solution is a completely virtualized data center, fully automated by software. The solution starts with a foundation that delivers IT as a service (ITaaS), with options for high availability, backup and recovery, and disaster recovery (DR). It also provides a framework and foundation for add-on modules, such as database as a service (DaaS), platform as a service (PaaS), and cloud brokering.
This Solution Guide provides an introduction to the concepts and architectural options available within the Federation Enterprise Hybrid Cloud solution. It should be used as an aid to deciding on the most suitable configuration for the initial deployment of a Federation Enterprise Hybrid Cloud solution.
This Solution Guide is intended for executives, managers, architects, cloud administrators, and technical administrators of IT environments who want to implement a hybrid cloud IaaS platform. Readers should be familiar with the VMware® vRealize® Suite, storage technologies, general IT functions and requirements, and how a hybrid cloud infrastructure accommodates these technologies and requirements.
The Federation Enterprise Hybrid Cloud 3.5: Reference Architecture Guide describes the reference architecture of a Federation Enterprise Hybrid Cloud solution. The guide introduces the features and functionality of the solution, the solution architecture and key components, and the validated hardware and software environments. The following guides provide further information about various aspects of the Federation Enterprise Hybrid Cloud solution:
Federation Enterprise Hybrid Cloud 3.5: Reference Architecture Guide
Federation Enterprise Hybrid Cloud 3.5: Administration Guide
Federation Enterprise Hybrid Cloud 3.5: Infrastructure and Operations Management Guide
Federation Enterprise Hybrid Cloud 3.5: Security Management Guide
The Federation Enterprise Hybrid Cloud solution enables customers to build an enterpriseclass, scalable, multitenant infrastructure that enables:
6
Complete management of the infrastructure service lifecycle
On-demand access to and control of network bandwidth, servers, storage, and security
Chapter 1: Executive Summary
Provisioning, monitoring, protection, and management of the infrastructure services by the line of business users, without IT administrator involvement
Provisioning from application blueprints with associated infrastructure resources by line-of-business application owners without IT administrator involvement
Provisioning of backup, continuous availability (CA), and DR services as part of the cloud service provisioning process
Maximum asset use
While many organizations have successfully introduced virtualization as a core technology within their data center, the benefits of virtualization have largely been restricted to the IT infrastructure owners. End users and business units within customer organizations have not experienced many of the benefits of virtualization, such as increased agility, mobility, and control. Transforming from the traditional IT model to a cloud-operating model involves overcoming the challenges of legacy infrastructure and processes, such as:
Inefficiency and inflexibility
Slow, reactive responses to customer requests
Inadequate visibility into the cost of the requested infrastructure
Limited choice of availability and protection services
The difficulty in overcoming these challenges has given rise to public cloud providers who have built technology and business models catering to the requirements of end-user agility and control. Many organizations are under pressure to provide similar service levels within the secure and compliant confines of the on-premises data center. As a result, IT departments need to create cost-effective alternatives to public cloud services, alternatives that do not compromise enterprise features such as data protection, DR, and guaranteed service levels.
This Federation Enterprise Hybrid Cloud solution integrates the best of EMC and VMware products and services, and empowers IT organizations to accelerate implementation and adoption of a hybrid cloud infrastructure, while still enabling customer choice for the compute and networking infrastructure within the data center. The solution caters to customers who want to preserve their investment and make better use of their existing infrastructure and to those who want to build out new infrastructures dedicated to a hybrid cloud. This solution takes advantage of the strong integration between EMC technologies and the VMware vRealize Suite. The solution, developed by EMC and VMware product and services teams includes EMC scalable storage arrays, integrated EMC and VMware monitoring, and data protection suites to provide the foundation for enabling cloud services within the customer environment. The Federation Enterprise Hybrid Cloud solution offers several key benefits to customers:
Rapid implementation: The solution can be designed and implemented in as little as
28 days, in a validated, tested, and repeatable way. This increases the time-to-value while simultaneously reducing risk.
Supported solution: Implementing Federation Enterprise Hybrid Cloud through EMC
also results in a solution that is supported by EMC and further reduces risk associated with the ongoing operations of your hybrid cloud.
7
Chapter 1: Executive Summary
Defined upgrade path: Customers implementing the Federation Enterprise Hybrid
Cloud receive upgrade guidance based on the testing and validation completed by the Federation engineering teams. This upgrade guidance enables customers, partners, and EMC services teams to perform upgrades faster, and with reduced risk.
Validated and tested integration: Extensive testing and validation has been
conducted by solutions engineering teams resulting in simplified use, management, and operation.
The EMC Federation EMC II, Pivotal, RSA, VCE, Virtustream, and VMware form a unique Federation of strategically aligned businesses; each can operate individually or together. The Federation provides customer solutions and choice for the software-defined enterprise and the emerging “3rd platform” of mobile, cloud, big data and social, transformed by billions of users and millions of apps.
Table 1 lists the terminology used in this guide. Table 1.
8
Terminology
Term
Definition
ACL
Access control list
AIA
Authority Information Access
API
Application programming interface
Blueprint
A blueprint is a specification for a virtual, cloud, or physical machine and is published as a catalog item in the common service catalog
Business group
A managed object that associates users with a specific set of catalog services and infrastructure resources
CBT
Changed Block Tracking
CDP
CRL Distribution Point
CRL
Certificate Revocation List
CSR
Certificate Signing Request
DHCP
Dynamic Host Configuration Protocol
Fabric group
A collection of virtualization compute resources and cloud endpoints managed by one or more fabric administrators
FQDN
Fully qualified domain name
HSM
Hardware security module
IaaS
Infrastructure as a service
IIS
Internet Information Services
LAG
Link aggregation group that bundles multiple physical Ethernet links between two or more devices into a single logical link can also be used to aggregate available bandwidth, depending on the protocol used.
LDAP
Lightweight Directory Access Protocol
LDAPS
LDAP over SSL
Chapter 1: Executive Summary Term
Definition
MCCLI
Management Console Command Line Interface
PEM
Privacy Enhanced Mail
PKI
Public key infrastructure
PVLAN
Private virtual LAN
SSL
Secure Sockets Layer
TACACS
Terminal Access Controller Access Control System
vRealize Automation blueprint
A specification for a virtual, cloud, or physical machine that is published as a catalog item in the vRealize Automation service catalog
VDC
Virtual device context
vDS
Virtual distributed switch
VLAN
Virtual local area network
VMDK
Virtual machine disk
VRF
Virtual routing and forwarding
VSI
Virtual Storage Integrator
VXLAN
Virtual Extensible LAN
EMC and the authors of this document welcome your feedback on the solution and the solution documentation. Please contact us at
[email protected] with your comments. Authors: Ken Gould, Fiona O’Neill
9
Chapter 2: Cloud Management Platform Options
This chapter presents the following topics:
Overview ..........................................................................................................11 Cloud management platform components .............................................................11 Cloud management platform models ....................................................................14
10
Chapter 2: Cloud Management Platform Options
The cloud management platform supports the entire management infrastructure for this solution. This management infrastructure is divided into three pods (functional areas), which consist of one or more VMware vSphere® ESXi™ clusters and/or vSphere resource groups, depending on the model deployed. Each pod performs a solution-specific function. This chapter describes the components of the management platform and the models available for use. After reading it, you should be able to decide on the model that suits your environment.
Management terminology and hierarchy
To understand how the management platform is constructed, it is important to know how a number of terms are used throughout this guide. Figure 1 shows the relationship between platform, pod, and cluster and their relative scopes as used in the Federation Enterprise Hybrid Cloud.
Figure 1.
Cloud management terminology and hierarchy
The following distinctions exist in terms of the scope of each term:
Platform (Cloud Management Platform) is an umbrella term intended to represent
the entire management environment.
Pod (Management Pod) Each management pod is a subset of the overall
management platform and represents a distinct area of functionality. What each area constitutes in terms of compute resources differs depending on the management models discussed in Cloud management platform models.
Cluster (Technology Cluster) is used in the context of the individual technologies.
While it may refer to vSphere clusters, it can also refer to VPLEX clusters, EMC RecoverPoint® clusters, and so on.
Resource pools Non-default resource pools are used only when two or more
management pods are collapsed onto the same vSphere cluster and are used to control and guarantee resources to each affected pod.
Management platform components
Federation Enterprise Hybrid Cloud sub-divides the management platform into distinct functional areas called management pods. These management pods are:
Core Pod
Network Edge Infrastructure (NEI) Pod
Automation Pod
11
Chapter 2: Cloud Management Platform Options
Figure 2 shows how the components of the management stack are distributed among the management pods.
Figure 2.
Cloud management platform component layout
Core Pod The Core Pod provides the base set of resources to establish the Federation Enterprise Hybrid Cloud solution services. It consists of:
External VMware vCenter Server™ (optional): This vCenter instance hosts only the
Core Pod components and hardware. It is required when using the Distributed management model and may already exist, depending on customer resources.
Cloud VMware vCenter Server: This vCenter instance is used to manage the NEI and
Automation components and compute resources. If using a Collapsed management model or Hybrid management model, it also hosts the Core Pod components and hardware. vRealize Automation uses this vCenter Server as its endpoint from which the appropriate vSphere clusters are reserved for use by vRealize Automation business groups. Note: While Figure 2 depicts vCenter, Update Manager, and PSC as one cell of related components, they are deployed as separate virtual machines.
Microsoft SQL Server: Hosts SQL Server databases used by the Cloud vCenter Server
and VMware Update Manager™. It also hosts the VMware vCenter Site Recovery Manager™ database in a DR dual-site/dual vCenter topology.
12
Chapter 2: Cloud Management Platform Options Note: Figure 2 includes separate SQL Server virtual machines for the External and Cloud vCenter SQL Server databases. This provides maximum resilience. Placing both vCenter databases on the same SQL Server virtual machine in the Core Pod is also supported. The vRealize IaaS SQL Server database must be on its own SQL Server instance in the Automation Pod.
VMware NSX™: Used to deploy and manage the virtual networks for the management
infrastructure and Workload Pods.
EMC SMI-S Provider: Management infrastructure required by EMC ViPR.
Note: When using VCE platforms, the SMI-S provider in the AMP cluster can be used, but must be kept in line with versions specified by Federation Enterprise Hybrid Cloud Simple Support Matrix. This may require a VCE Impact Assessment (IA) to achieve.
The hardware hosting the Core Pod is not enabled as a vRealize Automation compute resource, but the virtual machines it hosts provide the critical services required to instantiate the cloud. All of the virtual machines on the Core Pod are deployed on non-ViPR provisioned storage. The virtual machines can use existing SAN connected storage or any highly availability storage in the customer environment. The Federation Enterprise Hybrid Cloud supports Fibre Channel (FC), iSCSI, and NFS storage from EMC VNX® storage systems for the Core Pod storage. Though not mandatory, FC connectivity is strongly recommended. Note: For continuous availability topologies, this must be SAN-based block storage.
All storage should be RAID protected and all vSphere ESXi servers should be configured with EMC PowerPath/VE for automatic path management and load balancing.
Network Edge Infrastructure (NEI) Pod The NEI Pod is only required where VMware NSX is deployed, and is used to host NSX controllers, north-south NSX Edge Services Gateway (ESG) devices, and NSX Distributed Logical Router (DLR) control virtual machines. Use vSphere DRS rules to ensure that NSX Controllers are separated from each other and also to ensure that primary ESGs are separated from primary DLRs so that a host failure does not affect network availability. This pod provides the convergence point for the physical and virtual networks. Like the Core Pod, storage for this pod should be RAID protected and the Federation recommends Fibre Channel connections. vSphere ESXi hosts should run EMC PowerPath®/VE for automatic path management and load balancing.
Automation Pod The Automation Pod hosts the remaining virtual machines used for automating and managing the cloud infrastructure. The Automation Pod supports the services responsible for functions such as the user portal, automated provisioning, monitoring, and metering. The Automation Pod is managed by the Cloud vCenter Server instance; however it is dedicated to automation and management services. Therefore, the resources from this pod are not exposed to vRealize Automation business groups. The Automation Pod cannot share networks or storage resources with the workload clusters, and should be on a distinctly different Layer 3 network to both the Core and NEI management pods, even when using a collapsed management model. Storage provisioning for the Automation Pod follows the same guidelines as the NEI Pod. Automation Pod networks may be VxLANs managed by NSX. Note: While Figure 2 depicts vRealize IaaS as one cell of related components, the individual vRealize Automation roles are deployed as separate virtual machines.
13
Chapter 2: Cloud Management Platform Options
Workload Pods
Workload Pods are configured and assigned to fabric groups in VMware vRealize™ Automation. Available resources are used to host virtual machines deployed by business groups in the Federation Enterprise Hybrid Cloud environment. All business groups can share the available vSphere ESXi cluster resources. EMC ViPR® service requests are initiated from the vRealize Automation catalog to provision Workload Pod storage. Note: Workload Pods were previously termed resource pods in Enterprise Hybrid Cloud 2.5.1 and earlier.
The Federation Enterprise Hybrid Cloud supports three management models, as shown in Table 2. Note: Minimum host count depends on the compute resources specification being sufficient to support the relevant management virtual machine requirements. The Federation Enterprise Hybrid Cloud sizing tool may recommend a higher number based on the server specification chosen. Table 2.
Federation Enterprise Hybrid Cloud management models
Management model
Pod
vCenter used
Cluster used
Minimum no of hosts
Resource pool used
Distributed
Core
External
Core Cluster
2
N/A
NEI (NSX Only)
Cloud
NEI Cluster
4
N/A
Automation
Cloud
Automation Cluster
2
N/A
Core
Cloud
Collapsed Cluster
2 (w/o NSX)
Core
Collapsed
Hybrid
4 (w/ NSX)
NEI (NSX Only)
Cloud
Collapsed Cluster
NEI
Automation
Cloud
Collapsed Cluster
Core
Cloud
Core Cluster (AMP*)
3*
N/A
NEI (NSX Only)
Cloud
NEI Cluster
4
N/A
Automation
Cloud
Automation Cluster
2
N/A
Automation
* Based on VCE AMP2-HAP configuration. For ultimate resilience and ease of use during maintenance windows, creating vSphere clusters sizes based on N+2 may be appropriate according to customer preference, where N is the calculated CPU and RAM requirements for the hosted virtual machines plus host system overhead. The Federation Enterprise Hybrid Cloud sizing tool sizes vSphere clusters based on an N+1 algorithm. Table 3 indicates the Federation Enterprise Hybrid Cloud topologies supported by each management model. The topologies themselves are described later in this document.
14
Chapter 2: Cloud Management Platform Options Table 3.
Federation Enterprise Hybrid Cloud topologies supported by each management model
Management model
Single site
Standard dualsite/single vCenter
CA dualsite/single vCenter
Standard dualsite/dual vCenter
Disaster recovery dualsite/dual vCenter
Distributed
Supported
Supported
Supported
Supported
Supported
Collapsed
Supported
Supported
Supported
Supported
Supported
Hybrid
Supported
Supported
Supported
Supported
Supported
The following sections describe each of these models and provides guidance on how to choose the appropriate model.
Distributed management model
The distributed management model uses two separate vCenter instances and each management pod has its own distinct vSphere cluster. It requires a minimum of six hosts when used without NSX, or eight hosts when used with NSX. The first External vCenter Server instance manages all vSphere host and virtual machine components for the Core Pod. While the virtual machine running this vCenter instance can also be located within the Core Pod itself, it may be located on a separate system for further levels of high availability. The second Cloud vCenter Server instance located on the cloud management platform manages the NEI, Automation, and Workload Pods supporting the various business groups within the enterprise. This vCenter server acts as the vSphere end-point for vRealize Automation. Figure 3 shows the distributed management model configuration with two vCenters where the first vCenter supports the Core Pod and the second vCenter supports the remaining cloud management pods and tenant resources.
Figure 3.
Distributed Federation Enterprise Hybrid Cloud management model – vSphere view
The distributed management model:
Enables Core Pod functionality and resources to be provided by a pre-existing vSphere instance within your environment.
15
Chapter 2: Cloud Management Platform Options
Collapsed management model
Provides the highest level of resource separation (that is, host level) between the Core, Automation, and NEI Pods.
Places the NEI Pod ESXi cluster as the single intersection point between the physical and virtual networks configured within the solution, which eliminates the need to have critical networking components compete for resources as the solution scales and the demands of other areas of the cloud management platform increase.
Enhances the resilience of the solution because a separate vCenter server and SQL Server instance host the core cloud components.
The collapsed management model uses a single vCenter server to host all Core, Automation, and NEI Pod components as well as the Workload Pods. Each management pod is implemented as an individual vSphere resource pool on a single (shared) vSphere cluster, which ensures that each pod receives the correct proportion of compute and network resources. It requires a minimum of two physical hosts when used without NSX, or four hosts when used with NSX. Figure 4 shows an example of how the vSphere configuration might look with a collapsed management model.
Figure 4.
Collapsed Federation Enterprise Hybrid Cloud management model – vSphere view
The collapsed management model:
16
Provides the smallest overall management footprint for any given cloud size to be deployed.
Allows resource allocation between pods to be reconfigured with minimal effort.
Allows high-availability overhead to be reduced by using a single cluster, but does not alter the CPU, RAM, or storage required to manage the solution.
Chapter 2: Cloud Management Platform Options
Resource pool considerations Given that a single vSphere cluster is used in the collapsed management model, a vSphere resource group is required for each management pod to ensure sufficient resources are reserved for each function. Use the guidelines in Table 4 as the starting point for balancing these resources appropriately. Table 4.
Collapsed management model: Resource groups configuration
Resource
Core
NEI
Auto
CPU
20%
20%
60%
RAM
20%
5%
75%
Note: These figures are initial guidelines and should be monitored in each environment and fine-tuned accordingly. The percentages can be implemented, as shares, in whatever scale is required, as long as the percentage of shares assigned to each resource pool corresponds to the ratio of percentages in Table 4.
Hybrid management model
The hybrid management model uses a single vCenter server to host all Core, Automation, and NEI Pod components as well as the Workload Pods. Each management pod has its own vSphere cluster. Therefore, it requires a minimum of six hosts when used without NSX, or eight hosts when used with NSX. Figure 5 shows an example of how the vSphere configuration might look with a hybrid management model.
Figure 5.
Hybrid Federation Enterprise Hybrid Cloud management model – vSphere view
17
Chapter 2: Cloud Management Platform Options
The hybrid management model:
Deciding on the management model
Provides the highest level of resource separation (that is, host level) between the Core, Automation, and NEI Pods.
Places the NEI Pod ESXi cluster as the single intersection point between the physical and virtual networks configured within the solution, which eliminates the need to have critical networking components compete for resources as the solution scales and the demands of other areas of the cloud management platform increase.
Is compatible with VxBlock NSX factory deployments, and may use the VCE AMP vCenter as the Cloud vCenter.
Use the following key criteria to decide which management model is most suited for your environment:
Reasons to select the distributed management model Use these criteria to decide if this model is suitable for your environment. Reasons for selecting the distributed management model are:
To use existing infrastructure to provide the resources that will host the Core Pod.
To achieve the highest level of resource separation (that is, host level) between the Core, Automation, and NEI Pods.
To minimize the intersection points for north/south traffic to just the hosts that provide compute resources to the NEI Pod.
To maximize the resilience of the solution by using a separate vCenter server and SQL Server instance to host the Core Pod components.
Reasons to select the collapsed management model Use these criteria to decide if this model is suitable for your environment. Reasons for selecting the collapsed management model are:
To deploy the smallest management footprint for any given cloud size.
To reconfigure resource allocation between pods to be reconfigurable with minimal effort.
Reasons to select the hybrid management model Use these criteria to decide if this model is suitable for your environment. Reasons for selecting the hybrid management model are:
Network quality of service considerations
To achieve the highest level of resource separation (that is, host level) between the Core, Automation, and NEI Pods.
To minimize the intersection points for north/south traffic to just the hosts that provide compute resources to the NEI Pod.
To have a management model that overlays easily with VCE VxBlock, and to use the VCE AMP vCenter as the Cloud vCenter.
When using a management model that involves collapsed clusters, it may be necessary to configure network quality of service (QoS) to ensure that each function has a guaranteed minimum level of bandwidth available. Table 5 shows the suggested initial QoS settings. These may be fine-tuned as appropriate to the environment. Note: These values are suggestions based on the logical network Layout 1 in Chapter 3. As this layout is only a sample, you should collapse or divide these allocations according to the network topology you want to implement
18
Chapter 2: Cloud Management Platform Options Table 5.
Suggested network QoS settings
Name
VLAN
DVS shares
DVS % Min
QoS COS
vmk_ESXi_MGMT
100
500
5%
2
vmk_NFS
200
750
7.5%
4
vmk_iSCSI
300
750
7.5%
4
vmk_vMOTION
400
1400
14%
1
DPG_Core
500
500
5%
2
DPG_NEI
600
500
5%
2
DPG_Automation
700
500
5%
2
DPG_Tenant_Uplink
800
2000
20%
0
VXLAN_Transport
900
*
*
*
Avamar_Target (optional)
1000
**
**
**
DPG_AV_Proxies (optional)
1100
600
6%
0
ESG_DLR_Transit
Virtual Wire
1250
12.5%
0
Workload
Virtual Wire
1250
12.5%
0
*This is a VXLAN_Transport VLAN. The shares are associated with the virtual wire networks that use the transport VLAN. **Physical network only. No shares required.
Component high availability
Using vSphere ESXi clusters with VMware vSphere High Availability (vSphere HA) provides general virtual machine protection across the management platform. Additional levels of availability can be provided by using nested clustering between the component virtual machines themselves, such as Windows Failover Clustering, PostgreSQL clustering, load balancer clustering, or farms of machines that work together natively in an N+1 architecture, to provide a resilient architecture.
Distributed vRealize Automation The Federation Enterprise Hybrid Cloud requires the use of distributed vRealize Automation installations. In this model, multiple instances of each vRealize automation role are deployed behind a load balancer to ensure scalability and fault tolerance. All-in-one vRealize Automation installations are not supported for production use. VMware NSX Load Balancing technology is fully supported, tested, and validated by Federation Enterprise Hybrid Cloud. Other load balancer technologies supported by VMware for use in vRealize Automation deployments are also permitted, but configuration assistance for those technologies should be provided by VMware or the vendor. Use of a load balancer not officially supported by Federation Enterprise Hybrid Cloud or VMware for use with vRealize Automation requires a Federation Enterprise Hybrid Cloud request for product qualification (RPQ).
Clustered vRealize Orchestrator Both clustered and stand-alone vRealize Orchestrator installations are supported by Federation Enterprise Hybrid Cloud. Table 6 details the specific component high-availability options, as supported by each management model.
19
Chapter 2: Cloud Management Platform Options Table 6.
vRealize Automation and vRealize Orchestrator High Availability options
Management model
Distributed vRealize Automation
Minimal vRealize Automation (AIO)
Clustered vRealize Orchestrator (Active/Active)
Stand-alone vRealize Orchestrator
Distributed
Supported
Not supported
Supported
Supported
Collapsed
Supported
Not supported
Supported
Supported
Hybrid
Supported
Not supported
Supported
Supported
Highly available VMware Platform Services Controller Highly available configurations for VMware Platform Services Controller are not supported in Federation Enterprise Hybrid Cloud 3.5.
20
Chapter 3: Network Topologies
This chapter presents the following topics:
Overview ..........................................................................................................22 Implications of virtual networking technology options .............................................22 Logical network topologies ..................................................................................24
21
Chapter 3: Network Topologies
This solution provides a network architecture design that is resistant in the event of failure, enables optimal throughput, multitenancy and secure separation. This section presents a number of generic logical network topologies. Further network considerations specific to each topology are presented in the relevant chapters.
Physical connectivity
In designing the physical architecture, the main considerations are high availability, performance, and scalability. Each layer in the architecture should be fault tolerant with physically redundant connectivity throughout. The loss of any one infrastructure component or link should not result in loss of service to the tenant; if scaled appropriately, there is no impact on service performance. Physical network and FC connectivity to the compute layer may be provided over a converged network to converged network adapters on each compute blade, or over any network and FC adapters that are supported by the hardware platform and vSphere.
Supported virtual networking technologies
The Federation Enterprise Hybrid Cloud supports different virtual networking technologies as follows:
VMware NSX for vSphere
VMware vSphere Distributed Switch
The dynamic network services with vRealize Automation showcased in this solution require either NSX. vSphere Distributed Switch supports static networking configurations only, precluding the use of VXLANs.
The following section describes the implications, and features available, when VMware NSX is used with the Federation Enterprise Hybrid Cloud solution compared to non-NSX-based alternatives.
Solution attributes with and without VMware NSX
22
Table 7 compares the attributes, support, and responsibility for various aspects of the Federation Enterprise Hybrid Cloud solution, under its various topologies when used with and without VMware NSX. Table 7.
Comparing solution attributes with and without VMware NSX
Topology
Solution attributes with NSX
Solution attributes without NSX
Single-site
Provides the fully tested and validated load balancer component for vRealize Automation and other Automation Pod components.
vRealize Automation multi-machine blueprints may use networking components provisioned dynamically by NSX.
Requires a non-NSX load balancer for vRealize Automation and other Automation Pod components. Load balancers listed as supported by VMware are permitted, but the support burden falls to VMware or the relevant vendor.
Supports the full range of NSX functionality supported by VMware vRealize.
vRealize Automation blueprints must use pre-defined vSphere networks only (no dynamic provisioning of networking components possible).
Possesses fewer security features due to the absence of NSX.
Reduces network routing efficiency due to lack of east-west kernel level routing options provided by NSX.
Chapter 3: Network Topologies Standard dualsite/single vCenter
Provides the fully tested and validated load balancer component for vRealize Automation and other Automation Pod components.
vRealize Automation multi-machine blueprints may use networking components provisioned dynamically by NSX.
CA dualsite/single vCenter
Standard dualsite/dual vCenter
Supports the full range of NSX functionality supported by VMware vRealize.
Provides the fully tested and validated load balancer component for vRealize Automation and other Automation Pod components.
vRealize Automation multi-machine blueprints may use networking components provisioned dynamically by NSX.
Supports the full range of NSX functionality supported by VMware vRealize.
Enables automatic path fail over when the ‘preferred’ site fails.
Enables VXLAN over L2 or Layer 3 DCI to support tenant workload networks availability in both physical locations.
Provides the fully tested and validated load balancer component for vRealize Automation and other Automation Pod components.
vRealize Automation multi-machine blueprints may use networking components provisioned dynamically by NSX.
Supports the full range of NSX functionality supported by VMware vRealize.
Requires a non-NSX load balancer for vRealize Automation and other Automation Pod components. Load balancers listed as supported by VMware are permitted, but the support burden falls to VMware or the relevant vendor.
vRealize Automation blueprints must use pre-defined vSphere networks only (no dynamic provisioning of networking components possible).
Possesses fewer security features due to the absence of NSX.
Reduces network routing efficiency due to lack of east-west kernel level routing options provided by NSX.
Requires a non-NSX load balancer for vRealize Automation and other Automation Pod components. Load balancers listed as supported by VMware are permitted, but the support burden falls to VMware or the relevant vendor.
vRealize Automation blueprints must use pre-defined vSphere networks only (no dynamic provisioning of networking components possible).
Possesses fewer security features due to the absence of NSX.
Reduces network routing efficiency due to lack of east-west kernel level routing options provided by NSX.
Requires Layer 2 VLANs present at both sites to back tenant virtual machine vSphere port groups.
Requires a non-NSX load balancer for vRealize Automation and other Automation Pod components. Load balancers listed as supported by VMware are permitted, but the support burden falls to VMware or the relevant vendor.
vRealize Automation blueprints must use pre-defined vSphere networks only (no dynamic provisioning of networking components possible).
Possesses fewer security features due to the absence of NSX.
Reduces network routing efficiency due to lack of east-west kernel level routing options provided by NSX.
23
Chapter 3: Network Topologies DR dualsite/dual vCenter
Provides the fully tested and validated load balancer component for vRealize Automation and other Automation Pod components.
Does not support inter-site protection of dynamically provisioned VMware NSX networking artifacts.
Supports consistent NSX security group membership by ensuring virtual machines are placed in corresponding predefined security groups across sites via Federation workflows.
Allows fully automated network reconvergence for tenant resource pods networks on the recovery site via Federation workflows, the redistribution capability of BGP/OSPF and the use of NSX redistribution polices.
Does not honor NSX security tags applied to a virtual machine on the protected site prior to failover.
Requires a non-NSX load balancer for vRealize Automation and other Automation Pod components. Load balancers listed as supported by VMware are permitted, but the support burden falls to VMware or the relevant vendor.
vRealize Automation blueprints must use pre-defined vSphere networks only (no dynamic provisioning of networking components possible).
Possesses fewer security features due to the absence of NSX.
Reduces network routing efficiency due to lack of east-west kernel level routing options provided by NSX.
Requires customer-supplied IP mobility technology.
Requires manual or customer provided re-convergence process for tenant resource pods on the recovery site.
Each logical topology is designed to address the requirements of multitenancy and secure separation of the tenant resources. It is also designed to align with security best practices for segmenting networks according to the purpose or traffic type. In the distributed management platform option, a minimum of one distributed vSwitch is required for each of the External and Cloud vCenters, unless you run the Core Pod components on standard vSwitches. In that case, a minimum of one distributed vSwitch is required for the Cloud vCenter to support NSX networks. Multiple distributed vSwitches are supported in both cases. Note: While the minimum is one distributed vSwitch per vCenter, the Federation recommends two distributed vSwitches in the Cloud vCenter. The first distributed switch should be used for cloud management networks and the second distributed switch for tenant workload networks. The sample layouts provided later in this chapter use this model and indicate which networks are on each distributed switch by indicating vDS1 or vDS2. Additional distributed switches can be created for additional tenants if required.
In the collapsed management platform option, there must be at least one distributed vSwitch in the Cloud vCenter to support NSX. Multiple distributed vSwitches are supported.
24
Chapter 3: Network Topologies
Network layouts
The following network layouts are sample configurations intended to assist in understanding the elements that need to be catered for in a Federation Enterprise Hybrid Cloud network design. They do not represent a prescriptive list of the permitted configurations for logical networks in Federation Enterprise Hybrid Cloud. The network layout should be designed based on individual requirements.
Layout 1 Figure 6 shows one possible logical-to-physical network layout where standard vSphere switches are used for the basic infrastructural networks. This layout may be preferable where:
Additional NIC cards are available in the hosts to be used.
Increased protection against errors in configuration at a distributed vSwitch level is required
It does this by placing the NFS, iSCSI, and vSphere vMotion networks on standard vSwitches.
Dynamic networking technology is required through the use of NSX or vCloud Networking and Security
Note: All VLAN suggestions are samples only and should be determined by the network team in each particular environment.
Figure 6.
Network layout 1
25
Chapter 3: Network Topologies
Descriptions of each network are provided in Table 8. Table 8. Name
Type
Switch type
Location
VLAN
Description
vmk_ESXi_MGMT
VMkernel
Standard vSwitch
vSphere ESXi hosts
100
VMkernel on each vSphere ESXi host that hosts the management interface for the vSphere ESXi host itself. DPG_Core network should be able to reach this network.
vmk_NFS
VMkernel
Standard vSwitch
External vCenter and Cloud vCenter
200
Optional VMkernel used to mount NFS datastores to the vSphere ESXi hosts. NFS File Storage should be connected to the same VLAN / subnet or routable from this subnet.
vmk_iSCSI
VMkernel
Standard vSwitch
External vCenter and Cloud vCenter
300
Optional VMkernel used to mount iSCSI datastores to the vSphere ESXi hosts. iSCSI network portals should be configured to use the same VLAN / subnet or routable from this subnet.
vmk_vMOTION
VMkernel
Standard vSwitch
External vCenter and Cloud vCenter
400
VMkernel used for vSphere vMotion between vSphere ESXi hosts.
DPG_Core
vSphere distributed port group
Distributed vSwitch 1
External vCenter
500
Port group to which the management interfaces of all the core management components connect
DPG_NEI
vSphere distributed port group
Distributed vSwitch 1
Cloud vCenter
600
Port group to which the NSX controllers on the NEI Pod connect. DPG_Core network should be able to reach this network.
DPG_Automation
vSphere distributed port group
Distributed vSwitch 1
Cloud vCenter
700
Port group to which the management interfaces of all the Automation Pod components connect
DPG_Tenant_Uplink
vSphere distributed port group
Distributed vSwitch 2
Cloud vCenter
800
Port group used for all tenant traffic to egress from the cloud. Multiples may exist.
VXLAN_Transport
NSX distributed port group
Distributed vSwitch 2
Cloud vCenter
900
Port group used for VTEP endpoints between vSphere ESXi hosts to allow VXLAN traffic.
ESG_DLR_Transit
NSX logical switch
Distributed vSwitch 2
Cloud vCenter
Virtual wire
VXLAN segments connecting Tenant Edge and Tenant DLRs. Multiples may exist.
Workload
NSX logical switch
Distributed vSwitch 2
Cloud vCenter
Virtual wire
Workload VXLAN segments. Multiples may exist.
Avamar_Target
Primary PVLAN
N/A
Physical switches
1000
Promiscuous primary PVLAN to which physical Avamar grids are connected. This PVLAN has an associated secondary isolated PVLAN (1100) in which the Avamar proxies are placed
(Optional)
26
Network layout 1 descriptions
Chapter 3: Network Topologies Name
Type
Switch type
Location
VLAN
Description
DPG_AV_Proxies
Secondary PVLAN /
Distributed vSwitch 2
Physical switches/ Cloud vCenter
1100
Isolated secondary PVLAN to which Avamar Proxies virtual machines are connected. This PVLAN enables proxies to communicate with Avamar Grids on the Avamar_Target network but prevents proxies from communicating with each other
(Optional)
vSphere distributed port group
Layout 2 Figure 7 shows a second possible logical to physical network layout where distributed vSphere switches are used for all basic infrastructural networks other than the vSphere ESXi management network. This layout may be preferable where:
Fewer NIC cards are available in the hosts to be used.
Increased consolidation of networks is required
It does this by placing all bar the ESXi management interfaces on distributed vSwitches.
Dynamic networking technology is required through the use of NSX or vCloud Networking and Security
Note: All VLAN suggestions are samples only and should be determined by the network team in each particular environment.
Figure 7.
Network layout 2
27
Chapter 3: Network Topologies
Descriptions of each network are provided in Table 9. Table 9.
28
Network layout 2 descriptions
Name
Type
Switch type
Location
VLAN
Description
vmk_ESXi_MGMT
VMkernel
Standard vSwitch
ESXi hosts
100
VMkernel on each vSphere ESXi host that hosts the management interface for the vSphere ESXi host itself. DPG_Core network should be able to reach this network.
vmk_NFS
VMkernel
Distributed vSwitch 1
External vCenter and Cloud vCenter
200
Optional VMkernel used to mount NFS datastores to the vSphere ESXi hosts. NFS File Storage should be connected to the same VLAN / subnet or routable from this subnet.
vmk_iSCSI
VMkernel
Distributed vSwitch 1
External vCenter and Cloud vCenter
300
Optional VMkernel used to mount iSCSI datastores to the vSphere ESXi hosts. iSCSI network portals should be configured to use the same VLAN / subnet or routable from this subnet.
vmk_vMOTION
VMkernel
Distributed vSwitch 1
External vCenter and Cloud vCenter
400
VMkernel used for vSphere vMotion between vSphere ESXi hosts.
DPG_Core
vSphere distributed port group
Distributed vSwitch 1
External vCenter
500
Port group to which the management interfaces of all the core management components connect
DPG_NEI
vSphere distributed port group
Distributed vSwitch 1
Cloud vCenter
600
Port group to which the NSX controllers on the NEI Pod connect. DPG_Core network should be able to reach this network.
DPG_Automation
vSphere distributed port group
Distributed vSwitch 1
Cloud vCenter
700
Port group to which the management interfaces of all the Automation Pod components connect
DPG_Tenant_Uplink
vSphere distributed port group
Distributed vSwitch 2
Cloud vCenter
800
Port group used for all tenant traffic to egress from the cloud. Multiples may exist.
VXLAN_Transport
NSX distributed port group
Distributed vSwitch 2
Cloud vCenter
900
Port group used for VTEP endpoints between vSphere ESXi hosts to allow VXLAN traffic.
ESG_DLR_Transit
NSX logical switch
Distributed vSwitch 2
Cloud vCenter
Virtual wire
VXLAN segments connecting Tenant Edge and Tenant DLRs. Multiples may exist
Workload
NSX logical switch
Distributed vSwitch 2
Cloud vCenter
Virtual wire
Workload VXLAN segments. Multiples may exist.
Chapter 3: Network Topologies Name
Type
Switch type
Location
VLAN
Description
Avamar_Target
Primary PVLAN
N/A
Physical switches
1000
Promiscuous primary PVLAN to which physical Avamar grids are connected. This PVLAN has an associated secondary isolated PVLAN (1100) in which the Avamar proxies are placed
Secondary PVLAN/
Distributed vSwitch
Physical switches/ Cloud vCenter
1100
Isolated secondary PVLAN to which Avamar Proxies virtual machines are connected. This PVLAN enables proxies to communicate with Avamar Grids on the Avamar_Target network but prevents proxies from communicating with each other
(Optional)
DPG_AV_Proxies (Optional)
vSphere distributed port group
Layout 3 Figure 8 shows a third possible logical-to-physical network layout where distributed vSphere switches are used for all networks other than the management network. This layout may be preferable where:
There is no requirement for dynamic networking.
Reduction of management host count is paramount (as it removes the need for the NEI Pod).
Note: All VLAN suggestions are samples only and should be determined by the network team in each particular environment.
Figure 8.
Network layout 3
29
Chapter 3: Network Topologies
Descriptions of each network are provided in Table 10. Table 10. Name
Type
Switch type
Location
VLAN
Description
vmk_ESXi_MGMT
VMkernel
Standard vSwitch
ESXi Hosts
100
VMkernel on each vSphere ESXi host that hosts the management interface for the ESXi Host itself. DPG_Core network should be able to reach this network.
vmk_NFS
VMkernel
Standard vSwitch
External vCenter and Cloud vCenter
200
Optional VMkernel used to mount NFS datastores to the vSphere ESXi hosts. NFS File Storage should be connected to the same VLAN/subnet or routable from this subnet.
vmk_iSCSI
VMkernel
Standard vSwitch
External vCenter and Cloud vCenter
300
Optional VMkernel used to mount iSCSI datastores to the vSphere ESXi hosts. iSCSI network portals should be configured to use the same VLAN / subnet or routable from this subnet.
vmk_vMOTION
VMkernel
Standard vSwitch
External vCenter and Cloud vCenter
400
VMkernel used for vSphere vMotion between vSphere ESXi hosts.
DPG_Core
vSphere distributed port group
Distributed vSwitch 1
External vCenter
500
Port group to which the management interfaces of all the core management components connect
DPG_Automation
vSphere distributed port group
Distributed vSwitch 1
Cloud vCenter
600
Port group to which the management interfaces of all the Automation Pod components connect
DPG_Tenant_Uplink
vSphere distributed port group
Distributed vSwitch 2
Cloud vCenter
700
Port group used for all tenant traffic to egress from the cloud. Multiples may exist.
DPG_Workload_1
vSphere distributed port group
Distributed vSwitch 2
Cloud vCenter
800
Port group used for workload traffic
DPG_Workload_2
vSphere distributed port group
Distributed vSwitch 2
Cloud vCenter
900
Port group used for workload traffic
Avamar_Target
Primary PVLAN
N/A
Physical switches
1000
Promiscuous primary PVLAN to which physical Avamar grids are connected. This PVLAN has an associated secondary isolated PVLAN (1100) in which the Avamar proxies are placed
Secondary PVLAN /
Distributed vSwitch 2
Physical switches/ Cloud vCenter
1100
Isolated secondary PVLAN to which Avamar Proxies virtual machines are connected. This PVLAN enables proxies to communicate with Avamar Grids on the Avamar_Target network but prevents proxies from communicating with each other
(optional)
DPG_AV_Proxies (optional)
30
Network layout 3 descriptions
vSphere distributed port group
Chapter 4: Single-Site/Single vCenter Topology
This chapter presents the following topics:
Overview ..........................................................................................................32 Single-site networking considerations ...................................................................32 Single-site storage considerations ........................................................................33 Recovery of cloud management platform ..............................................................36 Backup of single-site/single vCenter enterprise hybrid cloud ....................................36
31
Chapter 4: Single-Site/Single vCenter Topology
This chapter describes networking and storage considerations for a single-site/single vCenter topology in the Federation Enterprise Hybrid Cloud solution.
When to use the single-site topology
The single-site/single vCenter Federation Enterprise Hybrid Cloud topology should be used when restart or recovery of the cloud to another data center is not required. It can also be used as the base deployment on top of which you may layer the dual-site/single vCenter or dual-site/dual vCenter topology at a later time.
Architecture
Figure 9 shows the single-site/single vCenter architecture for the Federation Enterprise Hybrid Cloud solution including the required sets of resources separated by pod.
Figure 9.
Supported virtual networking technologies
Supported VMware NSX features
32
Federation Enterprise Hybrid Cloud single-site architecture
The Federation Enterprise Hybrid Cloud supports the following virtual networking technologies in the single-site topology:
VMware NSX (recommended)
VMware vSphere Distributed Switch
When using VMware NSX in a single-site architecture, the Federation Enterprise Hybrid Cloud solution supports the full range of NSX functionality supported by VMware vRealize Automation. The integration between these components provides all the required functionality, including but not limited to:
Micro-segmentation
Dynamic provisioning of VMware NSX constructs via vRealize blueprints
Use of NSX security polices, groups, and tags
Integration with the VMware NSX Partner ecosystem for enhanced security
Chapter 4: Single-Site/Single vCenter Topology
NSX best practices
In a single-site topology, when NSX is used, all NSX Controller components reside in the same site and within the NEI Pod. NSX best practice recommends that each NSX Controller is placed on a separate physical host. NSX creates Edge Services Gateways (ESGs) and Distributed Logical Routers (DLRs). Best practice for ESGs and DLRs recommends that they are deployed in HA pairs, and that the ESGs and DLRs are separated from each other onto different physical hosts. Combining these best practices means that a minimum of four physical hosts are required to support the NEI Pod function when NSX is used. VMware anti-affinity rules should be used to ensure that the following conditions are true during optimum conditions:
NSX Controllers reside on different hosts.
NSX ESGs configured for high availability reside on different hosts.
NSX DLR Control virtual machines reside on different hosts.
NSX ESG and DLR Control virtual machines reside on different hosts.
When using the Federation Enterprise Hybrid Cloud Sizing tool, consider the choice of server specification for the NEI Pod to ensure efficient use of hardware resources, as the tool will enforce the four-server minimum when NSX is chosen.
Storage design
This Federation Enterprise Hybrid Cloud solution presents storage in the form of storage service offerings that greatly simplify virtual storage provisioning. The storage service offerings are based on ViPR virtual pools, which are tailored to meet the performance requirements of general IT systems and applications. Multiple storage system virtual pools, consisting of different disk types, are configured and brought under ViPR management. ViPR presents the storage to the enterprise hybrid cloud as virtual storage pools, abstracting the underlying storage details and enabling provisioning tasks to be aligned with the application’s class of service. In Federation Enterprise Hybrid Cloud, each ViPR virtual pool represents a storage service offering can be supported or backed by multiple storage pools of identical performance and capacity. This storage service offering concept is summarized in Figure 10.
33
Chapter 4: Single-Site/Single vCenter Topology
Figure 10.
Storage service offerings for the hybrid cloud
Note: The storage service offerings in Figure 10 are suggestions only. Storage service offerings can be configured and named as appropriate to reflect their functional use.
The storage service examples in Figure 10 suggest the following configurations:
All Flash: Can be provided by either EMC XtremIO™, VNX as all-flash storage, or
VMAX FAST VP where only the flash tier is used.
Tiered: Provides VNX or VMAX block or file-based VMFS or NFS storage devices and is
supported by multiple storage pools using EMC Fully Automated Storage Tiering for Virtual Pools (FAST® VP) and EMC Fully Automated Storage Tiering (FAST®) Cache.
Single Tier: Provides EMC VNX block- or file-based VMFS or NFS storage and is
supported by multiple storage pools using a single storage type of NL-SAS in this example. We suggest these storage service offerings only to highlight what is possible in a Federation Enterprise Hybrid Cloud environment. The full list of supported platforms includes:
EMC VMAX
EMC VNX
EMC XtremIO
EMC ScaleIO®
EMC VPLEX
EMC RecoverPoint
Isilon® (Workload use only)
As a result many other storage service offerings can be configured to suit business and application needs, as appropriate. Note: The Federation recommends that you follow the best practice guidelines when deploying any of the supported platform technologies. The Federation Enterprise Hybrid Cloud does not require any variation from these best practices.
34
Chapter 4: Single-Site/Single vCenter Topology
Storage consumption
vRealize Automation provides the framework to build relationships between vSphere storage profiles and Business Groups so that they can be consumed through the service catalog. Initially, physical storage pools are configured on the storage system and made available to ViPR where they are configured into their respective virtual pools. At provisioning time, LUNs or file systems are configured from these virtual pools and presented to vSphere as VMFS or NFS datastores. The storage is then discovered by vRealize Automation and made available for assignment to business groups within the enterprise. This storage service offering approach greatly simplifies the process of storage administration. Instead of users having to configure the placement of individual virtual machine disks (VMDKs) on different disk types such as serial-attached storage (SAS) and FC, they simply select the appropriate storage service level required for their business need. Virtual disks provisioned on FAST VP storage benefit from the intelligent data placement. While frequently accessed data is placed on disks with the highest level of service, less frequently used data is migrated to disks reflecting that service level. When configuring virtual machine storage, a business group administrator can configure blueprints to deploy virtual machines onto any of the available storage service levels. In the example in Figure 11, a virtual machine can be deployed with a blueprint including a SQL Server database, to a storage service offering named Prod-2, which was designed with the performance requirements of such an application in mind.
Figure 11.
Blueprint storage configuration in vRealize Automation
The devices for this SQL Server database machine have different performance requirements, but rather than assigning different disk types to each individual drive, each virtual disk can be configured on the Prod-2 storage service offering. This allows the underlying FAST technology to handle the best location for each individual block of data across the tiers. The vRealize Automation storage reservation policy ensures that the VMDKs are deployed to the appropriate storage. The storage presented to vRealize Automation can be shared and consumed across the various business groups using the capacity and reservation policy framework in vRealize Automation.
Storage provisioning
Storage is provisioned to the Workload vSphere clusters in the environment using the Provision Cloud Storage catalog item that can provision VNX, VMAX, XtremIO, ScaleIO, and VPLEX Local storage to single-site topology workload clusters. The workflow interacts with both ViPR and vRealize Automation to create the storage, present it to the chosen vSphere cluster and add the new volume to the relevant vRealize Storage Reservation Policy.
35
Chapter 4: Single-Site/Single vCenter Topology
vSphere clusters are made eligible for storage provisioning by tagging them with vRealize Automation custom properties that define them as Unprotected clusters, that is, that they are not involved in any form of inter-site replication relationship. This tagging is done during the installation and preparation of vSphere clusters for use by the Federation Enterprise Hybrid Cloud using the Unprotected Cluster Onboarding workflows provided as part of the Federation Enterprise Hybrid Cloud self-service catalog. Note: Virtual machines on the cluster may still be configured to use backup as a service, as shown in Chapter 7.
As local-only (unprotected) vSphere clusters can also exist in continuous availability and DR topologies, this process ensures that only the correct type of storage is presented to the single-site vSphere clusters and no misplacement of virtual machines intended for inter-site protection occurs.
ViPR virtual pools
For block-based provisioning, ViPR virtual arrays should not contain more than one protocol. For Federation Enterprise Hybrid Cloud this means that ScaleIO storage and FC block storage must be provided via separate virtual arrays. Note: Combining multiple physical arrays into fewer virtual arrays to provide storage to virtual pools is supported.
Single-site topology
Recovery of the management platform does not apply to a single-site topology, because there is no target site to recover to.
Single-site/single vCenter topology backup
The recommended option for backup in a single-site/single vCenter topology is the Standard Avamar configuration, though the Redundant Avamar/single vCenter configuration may also be used to provide additional resilience. Both options are described in Chapter 7.
.
36
Chapter 5: Dual-Site/Single vCenter Topology
This chapter presents the following topics:
Overview ..........................................................................................................38 Standard dual-site/single vCenter topology ...........................................................38 Continuous availability dual-site/single vCenter topology ........................................39 Continuous availability network considerations ......................................................40 VPLEX Witness ...................................................................................................45 VPLEX topologies ...............................................................................................46 Continuous availability storage considerations .......................................................53 Recovery of cloud management platform ..............................................................56 Backup in dual-site/single vCenter enterprise hybrid cloud ......................................56 CA dual-site/single vCenter ecosystem .................................................................57
37
Chapter 5: Dual-Site/Single vCenter Topology
This chapter describes the networking and storage considerations for a dual-site/single vCenter topology in the Federation Enterprise Hybrid Cloud solution.
When to use the dual-site/single vCenter topology
The dual-site/single vCenter Federation Enterprise Hybrid Cloud topology may be used when restart of the cloud to another data center is required. It should only be used in either of the following two scenarios:
Standard dual-site/single vCenter topology
Two sites are present that require management via a single vCenter instance and a single Federation Enterprise Hybrid Cloud management platform/portal.
Note: In this case, the scope of the term ‘site’ is at the user’s discretion. It could be taken to mean separate individual geographical locations, or could also mean independent islands of infrastructure in the same geographical location, such as independent VCE VxBlock platforms.
This topology has no additional storage considerations beyond the single-site/single vCenter topology because each site has completely independent storage.
When used with VMware NSX, this topology employs an additional NEI Pod on the second site to ensure north/south network traffic egresses the second site in the most efficient manner. The local NEI Pod will host the Edge gateway services for its respective site. Note: There is a 1:1 relationship between NSX Manager and vCenter Server, although there is a second NEI Pod, there is still only one NSX manager instance.
Continuous availability dual-site/single vCenter topology Continuous availability is required. This topology also requires that:
EMC VPLEX storage is available.
Stretched Layer 2 VLANs are permitted or the networking technology chosen supports VXLANs.
The latency between the two physical data center locations is less than 10 ms.
The standard dual-site/single vCenter Federation Enterprise Hybrid Cloud topology controls two sites, each with independent islands of infrastructure using a single vCenter instance and Federation Enterprise Hybrid Cloud management stack/portal. This architecture provides a mechanism to extend an existing Federation Enterprise Hybrid Cloud by adding additional independent infrastructure resources to an existing cloud, when resilience of the management platform itself is not required. Figure 12 shows the architecture used for this topology option.
38
Chapter 5: Dual-Site/Single vCenter Topology
Figure 12.
Federation Enterprise Hybrid Cloud standard dual-site/single vCenter architecture
The continuous availability (CA) dual-site/single vCenter Federation Enterprise Hybrid Cloud topology is an extension of the standard dual-site/single vCenter model that stretches the infrastructure across sites, using VMware vSphere Metro Storage Cluster (vMSC), vSphere HA, and VPLEX in Metro configuration. This topology enables multi-site resilience across two sites with automatic restart of both the management platform and workload virtual machines on the surviving site. Figure 13 shows the architecture used for this topology option.
39
Chapter 5: Dual-Site/Single vCenter Topology
Figure 13.
Supported virtual networking technologies
Federation Enterprise Hybrid Cloud CA dual-site/single vCenter architecture
The Federation Enterprise Hybrid Cloud supports the following virtual networking technologies in the dual-site/single vCenter topology:
VMware NSX (recommended)
VMware vSphere Distributed Switch
If vSphere Distributed Switch is used in the CA dual-site/single vCenter topology, then all networks must be backed by a Layer 2 VLAN that is present in both locations. VMware NSX enables you to use VXLANs backed by a Layer 3 DCI.
40
Chapter 5: Dual-Site/Single vCenter Topology
Supported VMware NSX features
NSX best practices
When using VMware NSX in dual-site/single vCenter architecture, the Federation Enterprise Hybrid Cloud solution supports the full range of NSX functionality supported by VMware vRealize Automation. The integration between these components provides all required functionality including, but is not limited to:
Micro-segmentation
Dynamic provisioning of VMware NSX constructs via vRealize blueprints
Use of NSX security polices, groups, and tags
Integration with the VMware NSX Partner ecosystem for enhanced security
In the CA dual-site, single vCenter topology, when NSX is used, all NSX Controller components reside in the NEI Pod, but the NEI Pod is supported by a vSphere Metro Storage (stretched) cluster. NSX best practice recommends that each controller is placed on a separate physical host. NSX creates Edge Services Gateways (ESGs) and Distributed Logical Routers (DLRs). Best practice for ESGs and DLRs recommends that they are deployed in HA pairs, and that the ESGs and DLRs are separated from each other onto different physical hosts. Combining these best practices, and ensuring that each site is fully capable of running the NSX infrastructure optimally means that a minimum of four physical hosts per site (eight in total) are required to support the NEI Pod function when NSX is used. VMware affinity and anti-affinity rules should be used to ensure that the following conditions are true during optimum conditions:
NSX Controllers reside on different hosts.
NSX Edge Services Gateways reside on different hosts.
NSX Distributed Logical Router Control virtual machines reside on different host.
NSX ESG and DLR Control virtual machines do not reside on the same physical hosts.
All NSX Controllers reside on a given site, and move laterally within that site before moving to the alternate site.
When using the Federation Enterprise Hybrid Cloud Sizing tool, appropriate consideration should be given to the choice of server specification for the NEI Pod to ensure efficient use of hardware resources, as the tool will enforce the four server minimum when NSX is chosen.
Data Center Interconnect
Data centers that are connected together over a metro link can use either Layer 2 bridged VLAN connectivity or Layer 3 routed IP connectivity. Both Data Center Interconnect (DCI) options have advantages and disadvantages. However, new standards and technologies, such as Virtual Extensible LAN (VXLAN), address most of the disadvantages.
Layer 2 DCI A Layer 2 DCI should be used in continuous availability scenarios where VMware NSX is not available. Traditional disadvantages of Layer 2 DCI The risks related to Layer 2 extensions between data centers mirror some of the limitations faced in traditional Ethernet broadcast domains. The limiting factor is the scalability of a single broadcast domain. A large number of hosts and virtual machines within a broadcast domain, all of which contend for shared network resources, can result in broadcast storms. The results of broadcast storms are always to the
41
Chapter 5: Dual-Site/Single vCenter Topology
detriment of network availability, adversely affecting application delivery and ultimately leading to a poor user experience. This can affect productivity. As the CA architecture is stretched across both data centers, a broadcast storm could cause disruption in both the primary and secondary data centers. Multiple Layer 2 interconnects create additional challenges for stretched networks. If unknown broadcast frames are not controlled, loops in the Layer 2 extension can form. This can also cause potential disruption across both data centers, resulting in network downtime and loss of productivity. If used, the Spanning Tree Protocol (STP) needs to be run and carefully managed to control loops across the primary and secondary site interconnecting links. Loop avoidance and broadcast suppression mechanisms are available to the IT professional, but must be carefully configured and managed. Traditional advantages of Layer 2 DCI The greatest advantage of Layer 2 DCI is the IP address mobility of physical and virtual machines across both data centers. This simplifies recovery in the event of a failure in the primary data center. Note: Layer 2 connectivity is often necessary for applications where heartbeats and clustering techniques are used across multiple hosts. In some cases, technologies might not be able to span Layer 3 boundaries.
Layer 3 DCI A Layer 2 DCI may be used in continuous availability scenarios where VMware NSX is available. Traditional disadvantages of Layer 3 DCI If an infrastructure failure occurs at the primary site, a machine migrated to the secondary data center must be reconfigured to use an alternate IP addressing scheme. This can be more time consuming and error prone than having a high-availability deployment across a single Layer 2 domain. Inter-site machine clustering may not be supported over a Layer 3 boundary, which can be either multicast or broadcast based. Traditional advantages of Layer 3 DCI Layer 3 DCI does not use extended broadcast domains or require the use of STP. Therefore, there is greater stability of the production and services networks across both primary and secondary data centers. Note: The data center interconnect physical link is subject to the availability of the local telecommunications service provider and the business requirement of the enterprise.
Optimal continuous availability DCI networking solution The network topology used in the CA for Federation Enterprise Hybrid Cloud solution can use the advantages of both Layer 2 and Layer 3 DCI topologies when used with VMware NSX. Layer 2 requirements such as resource and management traffic are handled by the VXLAN implementation enabled by NSX. This offers the advantage of IP mobility across both sites by placing the resource and management traffic on spanned VXLAN segments. It also eliminates the complexity of STP and performance degradation that large broadcast domains can introduce.
42
Chapter 5: Dual-Site/Single vCenter Topology
VXLANs can expand the number of Layer 2 domains or segments beyond the 802.1q limit of 4,096 VLANs to a theoretical limit of 16 million. VXLANs can also extend the Layer 2 environment over Layer 3 boundaries. An underlying Layer 3 data center interconnect runs a dynamic route distribution protocol with rapid convergence characteristics such as Open Shortest Path First (OSPF). OSPF routing metrics route the ingress traffic to the primary data center. If the primary data center is unavailable, the OSPF algorithm automatically converges routes to the secondary data center. This is an important advantage compared to using a traditional Layer 2 DCI and Layer 3 DCI solution in isolation. Note: NSX also supports Border Gateway Protocol (BGP) and Intermediate System to Intermediate System (IS-IS) route distribution protocols. The Federation Enterprise Hybrid Cloud supports OSPF and BGP, but not IS-IS.
In a collapsed management model, all clusters are part of the same vCenter instance and therefore can all be configured to use the security and protection features offered by the same NSX Manager instance. If this is not a requirement for Core Pod, then a stretched Layer 2 network may also be used. In a distributed management model, two vCenter instances are used. Given the 1:1 relationship between a vCenter instance and NSX Manager, a second NSX manager instance would be required if the Core Pod is to use the security and protection NSX provisioned networks. Given the small number of virtual machines present in the external vCenter, it may be appropriate to consider a stretched Layer 2 VLAN for this network if the second NSX manager instance is deemed unnecessary. Figure 14 shows one possible scenario where two data centers are connected using both a Layer 2 and a routed Layer 3 IP link and how the Core, NEI, Automation, and Workload segments could be provisioned.
43
Chapter 5: Dual-Site/Single vCenter Topology
Figure 14.
Continuous availability data center interconnect example using VMware NSX
In this scenario, the following properties are true:
vSphere ESXi stretched clusters are utilized to host the Core, Automation, NEI, and Workload virtual machines. This, with vSphere HA, enables virtual machines to be automatically restarted on the secondary site, if the primary site fails.
The Core Pod virtual machines are connected to a stretched VLAN. This prevents the need for a second NSX manager machine.
The NSX controllers (NEI Pod) are connected to the same stretched VLAN as the Core Pod virtual machines.
The Automation Pod virtual machines are connected to an NSX logical network, backed by VXLAN and available across both sites.
The Workload Pod virtual machines are connected to a NSX logical network, backed by VXLAN and available across both sites.
VXLAN encapsulated traffic must be able to travel between vSphere ESXi hosts at both sites.
One or more NSX Edge Services Gateways (ESGs) are deployed at each site to control traffic flow between the virtual and physical network environments. Note: NSX supports three modes of replication for VXLAN traffic unicast, multicast and hybrid. Unicast mode enables VXLAN traffic to be carried across Layer 3 boundaries without assistance from the underlying physical network, but requires availability of the NSX Controllers.
44
Chapter 5: Dual-Site/Single vCenter Topology
vSphere HA, with VPLEX and VPLEX Witness, enables the cloud-management platform virtual machines to restore the cloud-management service on the secondary site in the event of a total loss of the primary data center. In this scenario, the virtual machines automatically move to and operate from vSphere ESXi nodes residing in the secondary data center.
Edge Services Gateway considerations
All workload virtual machines should use NSX logical switches connected to a Distributed Logical Router (DLR). The DLR can provide the same default gateway to a virtual machine, whether it is running at the primary or secondary site.
DLRs should be connected to at least one ESG at each site and a dynamic route distribution protocol (such as OSPF and others supported by NSX) should be used to direct traffic flow. We recommend that you use both NSX High Availability and vSphere High Availability in conjunction with host DRS groups, virtual machine DRS groups and virtual machine DRS affinity rules to ensure that DLR virtual machines run on the correct site in optimum conditions.
This solution has all the advantages of traditional Layer 2 and Layer 3 solutions. It provides increased flexibility and scalability by implementing VXLANs, and benefits from increased stability by not extending large broadcast domains across the VPLEX Metro.
VPLEX Witness is an optional component deployed in customer environments where the regular preference rule sets are insufficient to provide seamless zero or near-zero recovery time objective (RTO) storage availability in the event of site disasters or VPLEX cluster and inter-cluster failures. Without VPLEX Witness, all distributed volumes rely on configured rule sets to identify the preferred cluster in the event of a cluster partition or cluster/site failure. However, if the preferred cluster fails (for example, as a result of a disaster event), VPLEX is unable to automatically enable the surviving cluster to continue I/O operations to the affected distributed volumes. VPLEX Witness is designed to overcome this. The VPLEX Witness server is deployed as a virtual appliance running on a customer’s vSphere ESXi host that is deployed in a failure domain separate from both of the VPLEX clusters. The third fault domain must have power and IP isolation from both the Site A and Site B fault domains, which host the VPLEX Metro Clusters. This eliminates the possibility of a single fault affecting both the cluster and VPLEX Witness. VPLEX Witness connects to both VPLEX clusters over the management IP network. By reconciling its own observations with the information reported periodically by the clusters, VPLEX Witness enables the clusters to distinguish between inter-cluster network partition failures and cluster failures, and to automatically resume I/O operations in these situations. Figure 15 shows an example of a high-level deployment of VPLEX Witness and how it can augment an existing static preference solution. The VPLEX Witness server resides in a fault domain separate from the VPLEX clusters on Site A and Site B.
45
Chapter 5: Dual-Site/Single vCenter Topology
Figure 15.
High-level deployment of EMC VPLEX Witness
VMware classifies the stretched VPLEX Metro cluster configuration with VPLEX into the following categories:
Deciding on VPLEX topology
Uniform host access configuration with VPLEX host Cross-Connect—vSphere ESXi hosts in a distributed vSphere cluster have a connection to the local VPLEX system and paths to the remote VPLEX system. The remote paths presented to the vSphere ESXi hosts are stretched across distance.
Non-uniform host access configuration without VPLEX host Cross-Connect— vSphere ESXi hosts in a distributed vSphere cluster have a connection only to the local VPLEX system.
Use the following guidelines to help you decide which topology suits your environment:
Uniform host access configuration with VPLEX host Cross-Connect
Uniform (Cross-Connect) is typically used where:
Inter-site latency is less than 5ms.
Stretched SAN configurations are possible.
Non-Uniform (without Cross-Connect) is typically used where:
Inter-site latency is between 5 ms and 10 ms.
Stretched SAN configurations are not possible.
EMC GeoSynchrony® supports the concept of a VPLEX Metro cluster with Cross-Connect. This configuration provides a perfect platform for a uniform vSphere stretched-cluster deployment. VPLEX with host Cross-Connect is designed for deployment in a metropolitantype topology with latency that does not exceed 5 ms round-trip time (RTT). vSphere ESXi hosts can access a distributed volume on the local VPLEX cluster and on the remote cluster in the event of a failure. When this configuration is used with VPLEX Witness, vSphere ESXi hosts are able to survive through multiple types of failure scenarios. For example, in the event of a VPLEX cluster or back-end storage array failure, the vSphere ESXi hosts can still access the second VPLEX cluster with no disruption in service. In the unlikely event that the preferred site fails, VPLEX Witness intervenes and ensures that access to the surviving cluster is automatically maintained. In this case, vSphere HA automatically restarts all affected virtual machines.
46
Chapter 5: Dual-Site/Single vCenter Topology
Figure 16 shows that all ESXi hosts are connected to the VPLEX clusters at both sites. This can be achieved in a number of ways:
Merge switch fabrics by using Inter-Switch Link (ISL) technology used to connect local and remote SANs.
Connect directly to the remote data center fabric without merging the SANs.
Figure 16.
Deployment model with VPLEX host Cross-Connect
This type of deployment is designed to provide the highest possible availability for a Federation Enterprise Hybrid Cloud environment. It can withstand multiple failure scenarios including switch, VPLEX, and back-end storage at a single site with no disruption in service. For reasons of performance and availability, the Federation recommends that separate host bus adapters be used for connecting to local and remote switch fabrics. Note: VPLEX host Cross-Connect is configured at the host layer only and does not imply any cross connection of the back-end storage. The back-end storage arrays remain locally connected to their respective VPLEX clusters.
From the host perspective, in the uniform deployment model with VPLEX host CrossConnect, the vSphere ESXi hosts are zoned to both the local and the remote VPLEX clusters. Figure 17 displays the VPLEX storage views for a host named DRM-ESXi088, physically located in Site A of our environment. Here the initiators for the host are registered and added to both storage views with the distributed device being presented from both VPLEX clusters.
47
Chapter 5: Dual-Site/Single vCenter Topology
Figure 17.
VPLEX storage views with host Cross-Connect
This configuration is transparent to the vSphere ESXi host. The remote distributed volume is presented as an additional set of paths. Figure 18 shows the eight available paths that are presented to host DRM-ESXi088, for access to the VPLEX distributed volume hosting the datastore named CC-Shared-M3. The serial numbers of the arrays are different because four of the paths are presented from the first VPLEX cluster and the remaining four are presented from the second.
Figure 18.
Datastore paths in a VPLEX with host Cross Connect configuration
PowerPath/VE autostandby mode Neither the host nor the native multipath software can by themselves distinguish between local and remote paths. This poses a potential impact on performance if remote paths are used for I/O in normal operations because of the cross-connect latency penalty.
48
Chapter 5: Dual-Site/Single vCenter Topology
PowerPath/VE provides the concept of autostandby mode, which automatically identifies all remote paths and sets them to standby (asb:prox is the proximity-based autostandby algorithm). This feature ensures that only the most efficient paths are used at any given time. PowerPath/VE groups paths internally by VPLEX cluster. The VPLEX cluster with the lowest minimum path latency is designated as the local/preferred VPLEX cluster, while the other VPLEX cluster within the VPLEX Metro system is designated as the remote/non-preferred cluster. A path associated with the local/preferred VPLEX cluster is put in active mode, while a path associated with the remote/non-preferred VPLEX cluster is put in autostandby mode. This forces all I/O during normal operations to be directed towards the local VPLEX cluster. If a failure occurs where the paths to the local VPLEX cluster are lost, PowerPath/VE activates the standby paths and the host remains up and running on the local site, while accessing storage on the remote site.
Non-uniform host access configuration without VPLEX Cross-Connect
The non-uniform host configuration can be used for a Federation Enterprise Hybrid Cloud deployment if greater distances are required. The supported latency of this configuration requires that the round-trip time be within 5 ms to comply with VMware HA requirements. Without the cross-connect deployment, vSphere ESXi hosts at each site have connectivity to only that sites VPLEX cluster. Figure 19 shows that hosts located at each site have connections to only their respective VPLEX cluster. The VPLEX clusters have a link between them to support the VPLEX Metro configuration, and the VPLEX Witness is located in a third failure domain.
Figure 19.
VPLEX architecture without VPLEX Cross-Connect
The major benefit of this deployment option is that greater distances can be achieved in order to protect the infrastructure. With the EMC VPLEX AccessAnywhereTM feature, the nonuniform deployment offers the business another highly resilient option that can withstand various types of failures including front-end and back-end single path failure, single switch failure, and single back-end array failure. Figure 20 shows the storage views from VPLEX cluster 1 and cluster 2. In the example nonuniform deployment, hosts DRM-ESXi077 and DRM-ESXi099 represent hosts located in different data centers. They are visible in their site-specific VPLEX cluster’s storage view. With AccessAnywhere, the hosts have simultaneous write access to the same distributed device, but only via the VPLEX cluster on the same site.
49
Chapter 5: Dual-Site/Single vCenter Topology
Figure 20.
VPLEX Storage Views without VPLEX Cross-Connect
Figure 21 shows the path details for one of the hosts in a stretched cluster that has access to the datastores hosted on the VPLEX distributed device. The World Wide Name (WWN) on the Target column shows that all paths to that distributed device belong to the same VPLEX cluster. PowerPath/VE has also been installed on all of the hosts in the cluster, and it has automatically set the VPLEX volume to the adaptive failover mode. The autostandby feature is not used in this case because all the paths to the device are local.
Figure 21.
vSphere Datastore Storage paths without VPLEX Cross-Connect
With vSphere HA, the virtual machines are also protected against major outages, such as network partitioning of the VPLEX WAN link or an entire site failure. In order to prevent any unnecessary down time, the Federation recommends that the virtual machines reside on the site that would win ownership of the VPLEX distributed volume in the event of such a partitioning occurring.
Site affinity for management platform machines
50
When using the CA dual-site/single vCenter topology, the Federation recommends that all platform components are bound to a given site using VMware affinity ‘should’ rules. This ensures minimum latencies between components while still allowing them to move to the surviving site in the case of a site failure.
Chapter 5: Dual-Site/Single vCenter Topology
Site affinity for tenant virtual machines
The solution uses VMware Host Distributed Resource Scheduler (DRS) groups to subdivide the vSphere ESXi hosts in each workload and management cluster into groupings of hosts corresponding to their respective sites. It does this by defining two VMware host DRS groups in the format SiteName_Hosts where the site names of both sites are defined during the installation of the Federation Enterprise Hybrid Cloud foundation package. VMware virtual machine DRS groups are also created in the format Sitename_VMs during the preparation of the ESXi cluster for continuous availability. Storage reservation polices (SRPs) created by the Federation Enterprise Hybrid Cloud storage as service workflows are automatically named to indicate the preferred site in which that storage type is run. Note: In this case, the preferred site setting means that in the event of a failure that results in the VPLEX units being unable to communicate, that this site will be the one that continues to provide read/write access to the storage.
During deployment of a virtual machine though the vRealize portal, the user is asked to choose from a list of storage reservation policies. Federation Enterprise Hybrid Cloud custom workflows use this information to place the virtual machine on a vSphere ESXi cluster with access to the required storage type and placing the virtual machine into the appropriate virtual machine DRS group. Virtual machines to host DRS rules are then used to bind virtual machines to the preferred site by configuring the SiteName_VMs virtual machine DRS group with a setting of “should run” on the respective SiteName_Hosts host DRS group. This ensures virtual machines run on the required site, while allowing them the flexibility of failing over if the infrastructure on that site becomes unavailable. Figure 22 shows how the virtual machine DRS groups and affinity rules might look in a sample configuration.
Figure 22.
Sample view of site affinity DRS group and rule configuration
51
Chapter 5: Dual-Site/Single vCenter Topology Note: The values “SiteA” and “SiteB” shown in both Figure 22 and Figure 23 can and should be replaced with meaningful site names in a production environment. They must correlate with the site name values entered during the Federation Enterprise Hybrid Cloud Foundation package initialization for site affinity to work correctly.
Figure 23 shows a simple example of two scenarios where virtual machines are deployed to a vMSC and how the logic operates to place those virtual machines on their preferred sites.
Figure 23.
Deploying virtual machines with site affinity
Scenario 1: Deploy VM1 with affinity to Site A This scenario describes deploying a virtual machine (VM1) with affinity to Site A: 1.
During virtual machine deployment, the user chooses a storage reservation policy named SiteA_Preferred_CA_Enabled.
1.
This storage reservation policy choice filters the cluster choice to only those clusters with that reservation policy. In this case cluster 1.
2.
Based on the selected storage reservation policy, Federation Enterprise Hybrid Cloud workflows programmatically determine that Site A is the preferred location, and therefore locates the virtual machine DRS affinity group corresponding with Site A, namely SiteA_VMs.
3.
The expected result is: a.
VM1 is deployed into SiteA_VMs, residing on host CL1-H1 or CL1H2.
b.
VM1 is deployed onto a datastore from the SiteA_Preferred_CA_Enabled storage reservation policy, for example: VPLEX_Distributed_LUN_SiteA_Preferred_01 or VPLEX_Distributed_LUN_SiteA_Preferred_02
Scenario 2: Deploy VM2 with affinity to Site B This scenario describes deploying a virtual machine (VM2) with affinity to Site B: 1.
52
During virtual machine deployment, the user chooses a storage reservation policy named SiteB_Preferred_CA_Enabled.
Chapter 5: Dual-Site/Single vCenter Topology
2.
This storage reservation policy choice filters the cluster choice to only those clusters with that reservation policy. In this case cluster 1.
3.
Based on the selected storage reservation policy, Federation Enterprise Hybrid Cloud workflows programmatically determine that Site B is the preferred location, and therefore locates the virtual machine DRS affinity group corresponding with Site B, namely SiteB_VMs.
4.
The expected result is: a.
VM2 is deployed into SiteB_VMs, meaning it resides on hosts CL1-H3 or CL1H4.
b.
VM1 is deployed onto a datastore from the SiteB_Preferred_CA_Enabled storage reservation policy. For example: VPLEX_Distributed_LUN_SiteB_Preferred_01 or VPLEX_Distributed_LUN_SiteB_Preferred_02
ViPR virtual arrays
There must be at least one virtual array for each site. By configuring the virtual arrays in this way, ViPR can discover the VPLEX and storage topology. You should carefully plan and perform this step because it is not possible to change the configuration after resources have been provisioned, without first disruptively removing the provisioned volumes.
ViPR virtual pools
ViPR virtual pools for block storage offer two options under High Availability: VPLEX local and VPLEX distributed. When you specify local high availability for a virtual pool, the ViPR storage provisioning services create VPLEX local virtual volumes. If you specify VPLEX distributed high availability for a virtual pool, the ViPR storage provisioning services create VPLEX distributed virtual volumes. To configure a VPLEX distributed virtual storage pool through ViPR:
Ensure a virtual array exists for both sites, with the relevant physical arrays associated with those virtual arrays. Each VPLEX cluster must be a member of the virtual array at its own site only.
Before creating a VPLEX high-availability virtual pool at the primary site, create a local pool at the secondary site. This is used as the target virtual pool when creating VPLEX distributed virtual volumes. When creating the VPLEX high-availability virtual pool on the source site, select the source storage pool from the primary site, the remote virtual array, and the remote pool created in Step 2. This pool is used to create the remote mirror volume that makes up the remote leg of the VPLEX virtual volume. Note: This pool is considered remote when creating the high availability pool because it belongs to VPLEX cluster 2 and we are creating the high availability pool from VPLEX cluster 1.
Figure 24 shows this configuration, where VPLEX High Availability Virtual Pool represents the VPLEX high-availability pool being created.
53
Chapter 5: Dual-Site/Single vCenter Topology
Figure 24.
Interactions between local and VPLEX distributed pools
As described in Site affinity for tenant virtual machines, Federation Enterprise Hybrid Cloud workflows leverage the ‘winning’ site in a VPLEX configuration to determine which site to map virtual machines to. To enable active/active clusters, it is therefore necessary to create two sets of datastores – one set that will win on Site A and another set than will win on Site B. To enable this, you need to configure an environment similar to Figure 24 for Site A, and the inverse of it for Site B (where the local pool is on Site A, and the high availability pool is configured from Site B).
ViPR and VPLEX consistency groups interaction
VPLEX uses consistency groups to maintain common settings on multiple LUNs. To create a VPLEX consistency group using ViPR, a ViPR consistency group must be specified when creating a new volume. ViPR consistency groups are used to control multi-LUN consistent snapshots and have a number of important rules associated with them when creating VPLEX distributed devices:
All volumes in any given ViPR consistency group must contain only LUNs from the same physical array. As a result of these considerations, the Federation Enterprise Hybrid Cloud STaaS workflows create a new consistency group per physical array, per vSphere cluster per site.
All VPLEX distributed devices in a given ViPR consistency group must have source and target backing LUNS from the same pair of arrays.
As a result of these two rules, it is a requirement of the Federation Enterprise Hybrid Cloud that an individual ViPR virtual pool is created for every physical array that provides physical pools for use in a VPLEX distributed configuration.
Virtual Pool Collapser function
Federation Enterprise Hybrid Cloud STaaS workflows use the name of the ViPR virtual pool chosen as part of the naming for the vRealize Storage Reservation Policy (SRP) that the new datastore is added to. The Virtual Pool Collapser (VPC) function of Federation Enterprise Hybrid Cloud collapses the LUNs from multiple virtual pools into a single SRP. The VPC function can be used in the scenario where multiple physical arrays provide physical storage pools of the same configuration or service level to VPLEX, but through
54
Chapter 5: Dual-Site/Single vCenter Topology
different virtual pools, and where required to ensure that all LUNS provisioned across those physical pools are collapsed into the same SRP. VPC can be enabled or disabled at a global Federation Enterprise Hybrid Cloud level. When enabled, the Federation Enterprise Hybrid Cloud STaaS workflows examine the naming convention of the virtual pool selected to determine which SRP it should add the datastore to. If the virtual pool has the string ‘_VPC-‘ in it, then Federation Enterprise Hybrid Cloud knows that it should invoke VPC logic.
Virtual Pool Collapser example Figure 25 shows an example of VPC in use. In this scenario, the administrator has enabled the VPC function and created two ViPR virtual pools
GOLD_VPC-000001, which has physical pools from Array 1
GOLD_VPC-000002, which has physical pools from Array 2
When determining how to construct the SRP name to be used, the VPC function will only use that part of the virtual pool name that exists before ‘_VPC-‘. In this example that results in the term ‘GOLD’ which then contributes to the common SRP name of SITEA_GOLD_CA_Enabled. This makes it possible to conform to the rules of ViPR consistency groups as well as providing a single SRP for all datastores of the same type, which maintains abstraction and balanced datastore usage at the vRealize layer.
Figure 25.
Virtual Pool Collapser example
In the example shown in Figure 25, all storage is configured to win on a single site (Site A). To enable true active/active vSphere Metro Storage clusters, additional pools should be configured in the opposite direction, as mentioned in Continuous availability storage considerations.
Storage provisioning
VPLEX distributed storage is provisioned to the Workload vSphere clusters in the environment using the Federation Enterprise Hybrid Cloud catalog item named Provision Cloud Storage. As shown in Figure 24, these VPLEX volumes can be backed by VMAX, VNX, or XtremIO arrays. Note: The Federation recommends that you follow the best practice guidelines when deploying any of the supported platform technologies. The Federation Enterprise Hybrid Cloud does not require any variation from these best practices.
55
Chapter 5: Dual-Site/Single vCenter Topology
The workflow interacts with both ViPR and vRealize Automation to create the storage, presents it to the chosen vSphere cluster, and adds the new volume to the relevant vRealize storage reservation policy. As with the single-site topology, vSphere clusters are made eligible for storage provisioning by ‘tagging’ them with vRealize Automation custom properties. However, in this case they are defined as CA Enabled clusters, that is, they are part of a vMSC that spans both sites in the environment. This tagging is done during the installation and preparation of vSphere clusters for use by the Federation Enterprise Hybrid Cloud using the CA Cluster Onboarding workflow provided as part of the Federation Enterprise Hybrid Cloud self-service catalog. As local-only vSphere clusters can also be present in CA topology, the Provision Cloud Storage catalog item will automatically present only ViPR VPLEX distributed virtual storage pools to provision from when you attempt to provision to a CA-enabled vSphere cluster.
56
Standard dualsite/single vCenter topology
This model provides no resilience/recovery for the cloud management platform. To enable this you should use the CA dual-site/single vCenter variant.
CA dualsite/single vCenter topology
As all of the management pods reside on vMSC, management components are recovered through vSphere HA mechanisms. Assuming the VPLEX Witness has been deployed in a third fault domain, this should happen automatically.
Dual-site/single vCenter topology backup
The primary option for backup in a dual-site/single vCenter topology is the Redundant Avamar/single vCenter configuration though the Standard Avamar configuration may also be used if backup is only required on one of the two sites. Both options are described in Chapter 7.
Chapter 5: Dual-Site/Single vCenter Topology
Ecosystem interactions
Figure 26 shows how the concepts in this chapter interact in a CA dual-site/single vCenter configuration. Data protection concepts from Chapter 7 are also included.
Figure 26.
CA dual-site/single vCenter ecosystem
57
Chapter 6: Dual-Site/Dual vCenter Topology
This chapter presents the following topics:
Overview ..........................................................................................................59 Standard dual-site/dual vCenter topology .............................................................59 Disaster recovery dual-site/dual vCenter topology ..................................................60 Disaster recovery network considerations .............................................................61 vCenter Site Recovery Manager considerations ......................................................69 vRealize Automation considerations ......................................................................72 Disaster recovery storage considerations ..............................................................73 Recovery of cloud management platform ..............................................................74 Best practices ....................................................................................................75 Backup in dual-site/dual vCenter topology ............................................................75 DR dual-site/dual vCenter ecosystem ...................................................................76
58
Chapter 6: Dual-Site/Dual vCenter Topology
This chapter describes networking and storage considerations for a dual-site/dual vCenter topology in the Federation Enterprise Hybrid Cloud solution.
When to use the dual-site/dual vCenter topology
The dual-site/single vCenter Federation Enterprise Hybrid Cloud topology may be used in either of the following scenarios.
Standard dual-site/dual vCenter topology Two sites are present that require management via independent vCenter instances and a single Federation Enterprise Hybrid Cloud management stack/portal. Each site must have their own storage and networking resources, otherwise this model has no additional considerations per site than those listed in the single-site/single vCenter model. This is because each site has totally independent infrastructure resources with independent vCenters, but is managed by the same Federation Enterprise Hybrid Cloud management platform/portal. Note: In this case, the scope of the term ‘site’ is at the users’ discretion. This can be separate individual geographical locations, or independent islands of infrastructure in the same geographical location, such as independent VxBlock platforms.
Disaster recovery dual-site/single vCenter topology Disaster recovery (restart of virtual machines on another site through the use of VMware Site Recovery Manager) is required. This topology also requires that EMC RecoverPoint is available. Note: Typically this model is used when the latency between the two physical data center locations exceeds the required latency for the use of vSphere metro storage clusters using VPLEX distributed storage (10 ms).
The standard dual-site/dual vCenter Federation Enterprise Hybrid Cloud architecture controls two sites, each with independent islands of infrastructure, each using its own vCenter instance but controlled by a single Federation Enterprise Hybrid Cloud management platform/portal. This architecture provides a mechanism to extend an existing Federation Enterprise Hybrid Cloud by adding additional independent infrastructure resources to an existing cloud, when resilience of the management platform itself is not required, but where the resources being added either already belong to an existing vCenter or it is desirable for them to do so. Figure 27 shows the architecture used for this topology option.
59
Chapter 6: Dual-Site/Dual vCenter Topology
Figure 27.
Federation Enterprise Hybrid Cloud standard dual-site/dual vCenter architecture
The DR dual-site/dual vCenter topology for the Federation Enterprise Hybrid Cloud solution provides protection and restart capability for workloads deployed to the cloud. Management and workload virtual machines are placed on storage protected by RecoverPoint and are managed from VMware vCenter Site Recovery Manager™. This topology allows for multi-site resilience across two sites with DR protection for both the management platform and workload virtual machines on the surviving site. Figure 28 shows the overall architecture of the solution.
60
Chapter 6: Dual-Site/Dual vCenter Topology
Figure 28.
Physical network design
Federation Enterprise Hybrid Cloud DR dual-site/dual vCenter architecture
The Federation Enterprise Hybrid Cloud solution deploys a highly resilient and fault-tolerant network architecture for intra-site network, compute, and storage networking. To achieve this, it uses features such as redundant hardware components, multiple link aggregation technologies, dynamic routing protocols, and high availability deployment of logical networking components. The DR dual-site/dual vCenter topology of the Federation Enterprise Hybrid Cloud solution requires network connectivity across two sites using WAN technologies. It maintains the resiliency of the Federation Enterprise Hybrid Cloud by implementing a similarly high-availability and fault tolerant network design with redundant links and dynamic routing protocols. The high-availability features of the solution, which can minimize downtime and service interruption, address any component-level failure within the site. Throughput and latency requirements are other important aspects of physical network design. To determine these requirements, consider carefully both the size of the workload and data that must be replicated between sites and the requisite RPOs and RTOs for your
61
Chapter 6: Dual-Site/Dual vCenter Topology
applications. Traffic engineering and QOS capabilities can be used to guarantee the throughput and latency requirements of data replication.
Requirements based on the management model
The DR dual-site/dual vCenter topology is supported on all Federation Enterprise Hybrid Cloud management models. The Automation Pod components must be on a different Layer 3 network to the Core and NEI Pod components so that they can be failed over using VMware Site Recovery Manager, and the Automation network re-converged without affecting the Core and NEI Pod components on the source site.
Supported virtual networking technologies
The Federation Enterprise Hybrid Cloud supports the following virtual networking technologies in the dual-site/dual vCenter topology:
Supported VMware NSX features
Unsupported VMware NSX features
VMware NSX (recommended)
VMware vSphere Distributed Switch backed by non-NSX technologies for network reconvergence
When using VMware NSX in dual-site /dual vCenter architecture, the Federation Enterprise Hybrid Cloud supports many NSX features, including but not limited to the following:
Micro-segmentation
Use of NSX security polices and groups
The following VMware NSX Features are not supported in dual-site/dual vCenter topology:
Inter-site protection of dynamically provisioned VMware NSX networking components
NSX security tags
Note: NSX security tags are not honored on failover as part of the out-of-the-box solution, but may be implemented as a professional services engagement.
NSX best practices
In a DR dual-site/dual vCenter topology, when NSX is used, NSX Controllers reside on each site’s corresponding NEI Pod. NSX best practice recommends that each controller is placed on a separate physical host. NSX will create ESGs and DLRs. Best practice for ESGs and DLRs recommends that they are deployed in HA pairs, and that the ESGs and DLRs are separated onto different physical hosts. Combining the above best practices means that a minimum of four physical hosts per site (eight hosts in total) are required to support the NEI pod function when NSX is used. VMware anti-affinity rules should be used to ensure that the following conditions are true during optimum conditions:
NSX controllers reside on different hosts.
NSX ESGs reside on different hosts.
NSX DLR Control virtual machines reside on different host.
NSX ESG and DLR Control virtual machines do not reside on the same physical hosts.
When using the Federation Enterprise Hybrid Cloud Sizing tool, appropriate consideration should be given to the choice of server specification for the NEI Pod to ensure efficient use of hardware resources, as the tool will enforce the four server per site minimum when NSX is chosen. Figure 29 shows how the various NSX components are deployed independently on both sites within the topology.
62
Chapter 6: Dual-Site/Dual vCenter Topology
Figure 29.
NEI Pods from the cloud vCenter Server instances on Site A and Site B
Perimeter NSX Edge When used with VMware NSX, the Federation Enterprise Hybrid Cloud solution provides multitier security support and security policy enforcement by deploying NSX Edges as perimeter firewalls. An NSX Edge can be deployed at different tiers to support tiered security policy control. Each site's NSX Manager deploys corresponding NSX Edge Services Gateways (ESGs) configured for services such as firewall, DHCP, NAT, VPN, and SSL-VPN.
Logical switches When used with VMware NSX, the Federation Enterprise Hybrid Cloud solution provides logical networking support through NSX logical switches that corresponds to VXLAN segments. These logical switches support the extension of Layer 2 connections between various virtual machines and other networking components such as NSX Edges and logical routers. The use of VXLAN also increases the scalability of the solution. For the DR for the Federation Enterprise Hybrid Cloud solution topology, transit logical switches are required on both sites to provide connections between the DLRs and NSX Edges, as shown in Figure 30 and Figure 31. Duplicate logical switches are also needed on both sites for use by the workload virtual machines.
63
Chapter 6: Dual-Site/Dual vCenter Topology
Figure 30.
Logical switches on Site A
Figure 31.
Logical switches on Site B
Distributed logical router When VMware NSX is used with the DR dual-site/dual vCenter topology, the NSX network elements of logical switches, DLRs and ESGs must be in place before configuring DRprotected blueprints to enable DR-protected workload provisioning. The DLRs perform east-west routing between the NSX logical switches. Additionally, the DLR can provide gateway services such as NAT for the virtual machines connected to the preprovisioned application with the ESG performing north-south routing between the DLR to the physical core network. Note: These network elements must be created either using the NSX UI or by direct API calls.
When these elements are in place, vRealize Automation blueprints can be configured to connect a machine's network adapter to their respective logical switch. The DLR control virtual machine is deployed on the NEI Pod in high-availability mode. In this mode, two virtual machines are deployed on separate hosts as an active/passive pair. The active/passive pair maintains state tables and verifies each other's availability through heartbeats. When a failure of the active DLR is detected, the passive DLR immediately takes over and maintains the connection state and workload availability.
64
Chapter 6: Dual-Site/Dual vCenter Topology
A DLR kernel module is deployed to each NSX-enabled Workload Pod host to provide east/west traffic capability and broadcast reduction. To provide default gateway services on both sites, a corresponding DLR must be deployed on both sites, as shown in Figure 32.
Figure 32.
IP mobility between the primary and recovery sites
DLR interfaces on Site A and Site B
The Federation Enterprise Hybrid Cloud solution supports migration of virtual machines to a recovery site without the need to change the IP addresses of the virtual machines. It does this by fully automating network re-convergence of tenant resource pods during disaster recovery failover when using VMware NSX only.
Non-NSX network technology requirements The use of vSphere Distributed Switch, backed by other non-NSX networking technologies is permitted, but requires that the chosen technology supports IP mobility. Additionally, it requires that network re-convergence for both the Automation Pod and tenant resource pods is carried out manually in accordance with the chosen network technology, or that automation of that network re-convergence is developed as a professional services engagement. Maintenance of the alternative network convergence strategy is outside the scope of Federation Enterprise Hybrid Cloud support.
VMware NSX-based IP mobility Default gateways on each site are created using DLRs. By configuring the DLRs on both sites identically, the same IP addresses and IP subnets are assigned to their corresponding
65
Chapter 6: Dual-Site/Dual vCenter Topology
network interfaces, as shown in Figure 32. In this way, there is no need to reconfigure workloads default gateway settings in a recovery scenario. A dynamic routing protocol is configured for the logical networking and is integrated with the physical networking to support dynamic network convergence and IP mobility for the networks (subnets) supported for DR. This approach simplifies the solution and eliminates the need to deploy additional services to support IP address changes. As shown in Figure 33, route distribution requires that an IP prefix created for each logical switch that is connected to the DLR. Note: For DR protection the IP Prefix must be configured with the same name and IP network values on both primary and recovery DLRs.
Figure 33.
Route redistribution policy on Site A and Site B
A route redistribution policy is configured by adding one or more prefixes to the Route Distribution table, so that logical switch networks defined in the prefix list can be redistributed to the dynamic routing protocol on the primary site DLR where the virtual machines are deployed and running. The route redistribution policy on the recovery site DLR
66
Chapter 6: Dual-Site/Dual vCenter Topology
is configured to deny redistribution of networks connected to the recovery site, as shown in Figure 30. In the event of a disaster or a planned migration, you should execute a recovery plan in VMware Site Recovery Manager. After the virtual machines are powered off, the Federation Enterprise Hybrid Cloud network convergence scripts automatically (when using VMware NSX) determine the networks relevant to the cluster being failed over, and modify the action settings of those networks on the primary site DLR to deny redistribution of the networks associated with the cluster being failed. Note: Only the protected networks contained in the specific Site Recovery Manager protection plan being executed will be set to ‘Deny’.
A subsequent recovery step uses the same network convergence scripts to modify the route redistribution policy on the recovery site DLR to permit redistribution of the corresponding recovery site networks before powering on the virtual machines. This dynamic network convergence ensures that the virtual machines can reach infrastructure services, such as Domain Name System (DNS) and Microsoft Active Directory on the recovery site, and reduces the recovery time. You can implement an additional level of routing control from a site to the WAN peering point to ensure that only appropriate networks are advertised. To enable network failover with the same IP subnet on both sites, a network can be active only on the primary site or the recovery site. To support this, the unit of failover for a network is restricted to a single compute cluster. All virtual machines on a compute cluster can fail over to the recovery site without affecting virtual machines running on other compute clusters. If the network spans multiple clusters, the administrator must configure the recovery plan to ensure that all virtual machines on the same network are active only on one site.
Supported routing protocols
The Federation Enterprise Hybrid Cloud has validated network designs using both OSPF and BGP in disaster recovery environments. BGP is recommended over OSPF, but both have been validated.
VMware NSXbased security design
This section describes the additional multitier security services available to virtual machines deployed in the Federation Enterprise Hybrid Cloud solution when used with VMware NSX.
NSX security policies NSX security policies use security groups to simplify security policy management. A security group is a collection of objects, such as virtual machines, to which a security policy can be applied. To enable this capability the machines contained in the multi-machine blueprint must be configured with one or more security groups. A network security administrator or application security administrator configures the security policies to secure application traffic according to business requirements. To ensure consistent security policy enforcement for virtual machines on the recovery site, you must configure the security policies on both the primary and recovery sites.
NSX perimeter Edge security Perimeter edges are deployed using NSX Edges on both the primary and recovery sites. The perimeter NSX Edge provides security features, such as stateful firewalls, and other services such as DHCP, NAT, VPN, and load balancer. The configuration of various services must be manually maintained on both the primary and recovery site perimeter edges. This ensures consistent security policy enforcement in case of DR or planned migration of virtual machines to the recovery site.
67
Chapter 6: Dual-Site/Dual vCenter Topology
NSX distributed firewall The Federation Enterprise Hybrid Cloud solution supports the distributed firewall capability of NSX to protect virtual machine communication and optimize traffic flow. The distributed firewall is configured though the Networking and Security -> Service Composer -> Security Groups section of the vSphere web client. Figure 34 shows various security groups that may be pre-created in the NSX security configuration.
Figure 34.
Security groups on the primary and recovery sites
The Federation Enterprise Hybrid Cloud solution provides an option to associate security group information with a machine blueprint. When a business user deploys the blueprint, the virtual machine is included in the security group configuration. This ensures enforcement of the applicable security policy as soon as the virtual machine is deployed. As shown in Figure 35, a corresponding security group of the same name must be created on the recovery site. To ensure that workloads are consistently protected after failover, both primary and recovery site security policies must be identically configured.
Figure 35.
68
Security group on the recovery site
Chapter 6: Dual-Site/Dual vCenter Topology
Overview
This DR for Federation Enterprise Hybrid Cloud solution incorporates storage replication using RecoverPoint, storage provisioning using ViPR, and integration with Site Recovery Manager to support DR services for applications and virtual machines deployed in the hybrid cloud. Site Recovery Manager natively integrates with vCenter and NSX to support DR, planned migration, and recovery plan testing.
RecoverPoint and ViPR Storage Replication Adapters
Site Recovery Manager integrates with EMC RecoverPoint storage replication and ViPR automated storage services via EMC Storage Replication Adapters (SRAs). The SRAs control the EMC RecoverPoint replication process. The EMC RecoverPoint SRA controls the Automation Pod datastores. The ViPR SRA controls protected Workload Pod datastores.
Site mappings
To support DR services, the Site Recovery Manager configuration must include resource mappings between the vCenter Server instance on the protected site and the vCenter Server instance on the recovery site. The mappings enable the administrator to define automated recovery plans for failing over application workloads between the sites according to defined RTOs and RPOs. The resources you need to map include resource pools, virtual machine folders, networks, and the placeholder datastore. The settings must be configured on both the protected and recovery sites to support application workload recovery between the two sites.
Resource pool mappings A Site Recovery Manager resource pool specifies the compute cluster, host, or resource pool that is running a protected application. Resource pools must be mapped between the protected site and the recovery site in both directions so that, when an application fails over, the application can then run on the mapped compute resources on the recovery site.
Folder mappings When virtual machines are deployed using the Federation Enterprise Hybrid Cloud solution, the virtual machines are placed in particular folders in the vCenter Server inventory to simplify administration. By default, virtual machines are deployed in a folder named VRM. This folder must be mapped between the protected and recovery sites in both directions. When used with Federation Enterprise Hybrid Cloud backup services, the folders used by backup as a service are automatically created in both vCenters and mapped in Site Recovery Manager.
Network mappings Virtual machines may be configured to connect to different networks when deployed. Applications deployed with DR support must be deployed on networks that have been configured as defined in the Disaster recovery network considerations section. The networks must be mapped in Site Recovery Manager between the protected and recovery sites in both directions. For testing recovery plans, you should deploy a test network and use test network mappings when you create the recovery plan. Note: A Layer 3 network must be failed over entirely. Active machines in a given Layer 3 network must reside only in the site with the "permit" route redistribution policy.
Placeholder datastore For every protected virtual machine, Site Recovery Manager creates a placeholder virtual machine on the recovery site. The placeholder virtual machine retains the virtual machine properties specified by the global inventory mappings or specified during protection of the individual virtual machine.
69
Chapter 6: Dual-Site/Dual vCenter Topology
A placeholder datastore must be accessible to the compute clusters that support the DR services. The placeholder datastore must be configured in Site Recovery Manager and must be associated with the compute clusters.
Disaster recovery support for Automation Pod vApps
The Federations Enterprise Hybrid Cloud used several components that are deployed as a vSphere vApp. Currently this list includes:
EMC ViPR Controller
EMC ViPR SRM
Site Recovery Manager protects virtual machines, but does not preserve the vApp structure required for EMC ViPR Controller and EMC ViPR SRM virtual machines to function. The high-level steps to achieve recovery of vApps are: 1.
Deploy the vApp identically in both sites.
2.
Vacate the vApp on the recovery site (delete the virtual machines, but retain the virtual machine container).
3.
Protect the vApp on the protected site through Site Recover Manager, mapping the vApp containers from both sites.
4.
Reapply virtual machine vApp settings on placeholder virtual machines.
For additional details on the process and if other vApps in the environment are required, see the VMware Knowledge Base topic: vCenter Operations Manager 5.0.x: Using Site Recovery Manager to Protect a vApp Deployment.
Protection groups
A protection group is the unit of failover in Site Recovery Manager. The Federation Enterprise Hybrid Cloud solution supports failover at the granularity of the Workload Pod. In the context of the DR dual-site/dual vCenter topology, two Workload Pods are assigned to a DR pair, where one pod is the primary and is considered the protected cluster, and the second pod is the alternate site and is considered the recovery cluster. All protection groups associated with a DR pair and all the virtual machines running on a particular pod must failover together. For the DR dual-site/dual vCenter topology there is a 1:1 mapping between a DR pair and a recovery plan, and each recovery plan will contain one or more protection groups. Each protection group contains a single replicated vSphere datastore, and all the virtual machines that are running on that datastore. When you deploy new virtual machines on a Workload Pod vRealize Automation, those virtual machines are automatically added to the corresponding protection group and fail over with that protection group.
Recovery plans
Recovery plans enable administrators to automate the steps required for recovery between the primary and recovery sites. A recovery plan may include one or more protection groups. You can test recovery plans to ensure that protected virtual machines recover correctly to the recovery site.
Tenant Pod recovery plans The automated networking re-convergence capabilities of this DR for Federation Enterprise Hybrid Cloud solution eliminate the need to change the IP addresses of workload virtual machines when they failover from one site to the other. Instead, the tenant networks move with the virtual machines and supports virtual machine communication outside the network when on the recovery site.
70
Chapter 6: Dual-Site/Dual vCenter Topology
When using VMware NSX, Federation Enterprise Hybrid Cloud can automate network reconvergence of the tenant workload pods via a custom VMware Site Recovery Manager step of the Site Recovery Manager recovery plan, ensuring security policy compliance on the recovery site during a real failover. However, running a test Site Recovery Manager recovery plan with VMware NSX does not affect the production virtual machines, because the network convergence automation step has the required built-in intelligence to know that the networks should not be re-converged in that scenario. If non-NSX alternatives are used, then this network re-convergence is not automated, and therefore needs to be done manually during a pause in the Site Recovery Manager recovery plan, or via an automated Site Recovery Manager task created as part of a professional services engagement. Note: A recovery plan must be manually created per DR-enabled cluster before any STaaS operations are executed - two per pair to enable failover and failback.
Automation Pod recovery plans Network re-convergence of the network supporting the Federation Enterprise Hybrid Cloud Automation Pod is a manual task irrespective of the presence of VMware NSX. Note: This reflects the out-of-the-box solution experience. Automated network re-convergence for the Automation Pod can be achieved via a professional services engagement.
Collapsed management model
When configuring the protection group and recovery plans for the Automation Pod components under a collapsed management model, you must exclude all Core and NEI Pod components from the configurations. This is to ensure that system does not attempt to fail over the Core and NEI components from one site to the other.
71
Chapter 6: Dual-Site/Dual vCenter Topology
Configuring primary and recovery site endpoints
The Federation Enterprise Hybrid Cloud solution uses vRealize Automation to provide automated provisioning and management of cloud resources such as storage and virtual machines. To support DR services for cloud resources, you must configure vRealize Automation two virtual (vCenter) endpoints. The first endpoint is configured to support IaaS services for the first site; this endpoint uses the vCenter Server instance where the storage and virtual machines for the first site are deployed. The second endpoint is configured to serve as the recovery site for the resources of the first site. If required, workloads can also be configured to run in the secondary site, with recovery in the first site by configuring multiple DR cluster pairs with a protected cluster in each site, and a corresponding recovery cluster on the other site. To configure each endpoint, a separate vCenter agent must be installed on the IAAS server that is running vRealize Automation.
Configuring the infrastructure for disaster recovery services
The vRealize Automation IaaS administrator must assign the compute resources for the Workload Pods, on both the protected and recovery sites, to the fabric administrator for allocation to business groups. In the dual-site/dual vCenter DR configuration, you must designate Workload Pods (clusters) as DR-enabled, and all workloads deployed to that cluster will be DR-protected. If you have additional workloads that don’t require DR support then additional local (unprotected) workload pods should be provisioned to accommodate this. When replicated storage is provisioned to a protected Workload Pod, the fabric administrator must update the reservation policies for the relevant business groups to allocate the newly provisioned storage. Federation Enterprise Hybrid Cloud STaaS workflows automatically add newly provisioned storage to the appropriate protection group. This ensures that the virtual machines deployed on the storage are automatically protected and are included in the recovery plans defined for the Workload Pod.
Configuring application blueprints for disaster recovery
Storage reservation policies are used to deploy virtual machine disks to a datastore that provides the required RPO. The vRealize Automation IaaS administrator must create storage reservation policies to reflect the RPOs of different datastores. The fabric administrator must then assign the policies to the appropriate datastores of the compute clusters. Business Group administrators can configure the blueprints for virtual machines so that business users can select an appropriate storage reservation policy when deploying an application. The business user requests a catalog item in the Federation Enterprise Hybrid Cloud tenant portal, selects storage for the virtual machines, and assigns an appropriate storage reservation policy for the virtual machines disks based on the required RPO. The choice made at this point also dictates whether the virtual machine will be DR protected or not. The virtual machines disks are then placed on datastores that support the required RPO. The virtual machines are automatically deployed with the selected DR protection service and associated security policy for both the primary and recovery sites.
72
Chapter 6: Dual-Site/Dual vCenter Topology
ViPR managed Workload Pod storage
For the Workload Pods, ViPR SRA manages the protection of ViPR-provisioned storage. ViPR SRA provides an interface between Site Recovery Manager and ViPR Controller. ViPR Controller, which is part of the Automation Pod, must be running and accessible before the ViPR SRA can instruct ViPR to control the EMC RecoverPoint replication functions. This means that the Automation Pod and ViPR vApp must be functioning before Site Recovery Manager can execute a recovery of the Workload Pods.
Storage at each site
The Core and NEI clusters on each site require site-specific storage that does not need to be protected by EMC RecoverPoint. Site Recovery Manager also requires site-specific datastores on each site to contain the placeholder virtual machines for the tenant and automation pods. The Automation Pod storage must be distinct from the Core and NEI storage and protected by EMC RecoverPoint.
ViPR virtual arrays
There must be at least one virtual array for each site. By configuring the virtual arrays in this way, ViPR can discover the EMC RecoverPoint and storage topology. You should carefully plan and perform this step because it is not possible to change the configuration after resources have been provisioned, without first disruptively removing the provisioned volumes.
ViPR virtual pools
When you specify EMC RecoverPoint as the protection option for a virtual pool, the ViPR storage provisioning services create the source and target volumes and the source and target journal volumes, as shown in Figure 36.
Figure 36.
ViPR/EMC RecoverPoint protected virtual pool
Each DR-protected/recovery cluster pair has storage that replicates (under normal conditions) in a given direction, for example, from Site A to Site B. To allow active/active site configuration, additional DR cluster pairs should be configured whose storage replicates in the opposite direction. You must create two sets of datastores – one set that will replicate from Site A and another set that will replicate from Site B. To enable this, you need to configure an environment similar to Figure 36 for Site A, and the inverse of it for Site B (where the protected source pool is Site B, and local target pool is on Site A).
RecoverPoint journal considerations
Every RecoverPoint-protected LUN requires access to a journal LUN to maintain the history of disk writes to the LUN. The performance of the journal LUN is critical in the overall performance of the system attached to the RecoverPoint-protected LUN and therefore its
73
Chapter 6: Dual-Site/Dual vCenter Topology
performance capability should be in line with the expected performance needs of that system. By default, ViPR uses the same virtual pool for both the target and the journal LUN for a RecoverPoint copy, but it does allow you to specify a separate or dedicated pool. In both cases, the virtual pool and its supporting physical pools should be sized to provide adequate performance.
Storage provisioning
EMC RecoverPoint protected storage is provisioned to the Workload vSphere clusters in the environment using the catalog item named Provision Cloud Storage. Note: The Federation recommends that you follow the best practice guidelines when deploying any of the supported platform technologies. The Federation Enterprise Hybrid Cloud does not require any variation from these best practices.
The workflow interacts with both ViPR and vRealize Automation to create the storage, present it to the chosen vSphere cluster and add the new volume to the relevant vRealize storage reservation policy. As with the single-site topology, vSphere clusters are made eligible for storage provisioning by tagging them with vRealize Automation custom properties. However, in this case they are defined as DR-enabled clusters, that is, they are part of a Site Recovery Manager configuration that maps protected clusters to recovery clusters. This tagging is done during the installation and preparation of vSphere clusters for use by the Federation Enterprise Hybrid Cloud using the DR Cluster Onboarding workflow provided as part of the Federation Enterprise Hybrid Cloud self-service catalog. As local-only vSphere clusters can also be present in a DR dual-site/dual vCenter topology, when you attempt to provision to a DR-enabled vSphere cluster, the Provision Cloud Storage catalog item will automatically present only EMC RecoverPoint-protected virtual storage pools to provision from.
Standard dualsite/dual vCenter topology
This model provides no resilience/recovery for the cloud management platform. To enable this you should use the DR dual-site/dual vCenter variant.
DR dual-site/dual vCenter topology
In the DR dual-site/dual vCenter topology, EMC RecoverPoint and Site Recovery Manager protect the Automation Pod. This allows for recovery between Site A and Site B in planned and unplanned recovery scenarios. EMC RecoverPoint SRA for Site Recovery Manager is used to interact with EMC RecoverPoint during a failover of the Automation Pod’s resources. The Core and NEI Pods (when NSX is used) are created manually on both sites to mirror functionality such as NSX dynamic routing, NSX security groups, NSX security policies (firewall rules) and to host the Site Recovery Manager servers. As a result, there is no need to protect them using EMC RecoverPoint or Site Recovery Manager. In a distributed management model, this is accomplished by excluding the Core and NEI Pods from the process of creating associated datastore replications, protection groups, and recovery plans for the vSphere ESXi clusters hosting those functions. In a collapsed management model, all components are on the same vSphere ESXi cluster, so the Core and NEI components must be excluded from Site Recovery Manager recovery plans and protections groups for that cluster. Despite residing on the same vSphere cluster, the Automation Pod components should be on a distinct network and a distinct set of datastores, so that they can be failed over between sites without affecting Core or NEI components.
74
Chapter 6: Dual-Site/Dual vCenter Topology
Tenant workload networks are automatically re-converged to the recovery site by the Federation Enterprise Hybrid Cloud solution when used with VMware NSX. When non-NSX alternatives are used, tenant network re-convergence is not automated by the Federation Enterprise Hybrid Cloud. Automation Pod network re-convergence is a manual step with or without the presence of VMware NSX. The vCenter Server instances on each site manage the NEI, Automation, and Workload Pods on their respective sites and act as the vSphere end-points for vRealize Automation. The vCenter Server instances are integrated using Site Recovery Manager, which maintains failover mappings for the networks, clusters, and folders between the two sites.
Naming conventions
VMware vCenter Site Recovery Manager protection groups Protection group names must match the Workload Pod names—for example, if SAComputePod2 is the name of Workload Pod 2 on Site A, then the Site Recovery Manager protection group must also be named SAComputePod2. The solution relies on this correspondence when performing several of the automation tasks necessary for successful failover and subsequent virtual machine management through vRealize Automation.
VMware NSX security groups Security group names must be the same on both sites.
VMware NSX security policies Security policy names must be the same on both sites.
EMC ViPR virtual pools ViPR virtual pool names must be meaningful because they are the default names for storage reservation policies. For example, when creating Tier 1 DR protected storage with an RPO of 10 minutes, Tier 1 – DR Enabled – 10 Minute RPO is an appropriate name.
NSX logical networks
Each Workload Pod (compute cluster) must have its own transport zone. The NEI Pod must be a member of each transport zone. If a transport zone spans multiple compute clusters, the corresponding Site Recovery Manager protection groups must be associated with the same Site Recovery Manager recovery plan. The reason for this is that, when a transport zone spans multiple compute clusters, network mobility from Site A to Site B affects the virtual machines deployed across these clusters; therefore, the clusters must be failed over as a set.
DR dual-site/dual vCenter topology backup
The recommended option for backup in a DR dual-site/dual vCenter topology is the Redundant Avamar/dual vCenter configuration. This option is described in detail in Chapter 7.
75
Chapter 6: Dual-Site/Dual vCenter Topology
Ecosystem interactions
Figure 37 shows how the concepts in this chapter interact in a DR dual-site/dual vCenter configuration. Data protection concepts from Chapter 7 are also included.
Figure 37.
76
DR ecosystem
Chapter 7: Data Protection
This chapter presents the following topics:
Overview ..........................................................................................................78 Concepts...........................................................................................................79 Standard Avamar configuration............................................................................84 Redundant Avamar/single vCenter configuration ....................................................86 Redundant Avamar/dual vCenter configuration ......................................................90
77
Chapter 7: Data Protection
This chapter discusses the considerations for implementing data protection, also known as backup as a service (BaaS) in the context of the Federation Enterprise Hybrid Cloud. Backup and recovery of a hybrid cloud is a complicated undertaking in which many factors must be considered, including:
Backup type and frequency
Impact and interaction with replication
Recoverability methods and requirements
Retention periods
Automation workflows
Interface methods (workflows, APIs, GUI, CLI, scripts, and so on)
Implementation in a CA or DR-enabled environment
VMware vRealize Orchestrator™, which is central to all of the customizations and operations used in this solution, manages operations across several EMC and VMware products, including:
VMware vRealize Automation
VMware vCenter
EMC Avamar and EMC Data Protection Advisor™
This solution uses Avamar as the technology to protect your datasets. Using Avamar, this backup solution includes the following characteristics:
Abstracts and simplifies backup and restore operations for cloud users
Uses VMware Storage APIs for Data Protection, which provides Changed Block Tracking for faster backup and restore operations
Provides full image backups for running virtual machines
Eliminates the need to manage backup agents for each virtual machine in most cases
Minimizes network traffic by deduplicating and compressing data
Note: The Federation recommends that you engage an Avamar product specialist to design, size, and implement a solution specific to your environment and business needs.
78
Chapter 7: Data Protection
Scalable backup architecture
The Federation Enterprise Hybrid Cloud backup configurations add scalable backup by adding the ability to configure an array of Avamar instances. Federation Enterprise Hybrid Cloud BaaS workflows automatically distribute the workload in a round-robin way across the available Avamar instances, and provides a catalog item to enable additional Avamar instances (up to a maximum of 15 Avamar replication pairs) to be added to the configuration. When new Avamar instances are added, new virtual machine workloads are automatically assigned to those new instances until an equal number of virtual machines are assigned to all Avamar instances in the environment. Once that target has been reached, virtual machines are assigned in a round-robin way again. The configuration of the Avamar instances is stored by the Federation Enterprise Hybrid Cloud workflows for later reference when reconfiguring or adding instances.
Avamar replication pairs
An Avamar replication pair is defined as a relationship configured between two Avamar instances, and is used by the Federation Enterprise Hybrid Cloud workflows to ensure backup data is protected against the loss of a physical Avamar instance. Normally this is used to ensure that data backed up on one site is available to restore on a secondary site, but it could also be used to provide extra resilience on a single site if required. The Federation Enterprise Hybrid Cloud provides two different redundant Avamar configurations that use an array of Avamar replication pairs to achieve the same scalability as the standard Avamar configuration but with the added resilience that every instance of Avamar has a replication partner, to which it can replicate any backup sets that it receives. Note: In the standard Avamar configuration, each instance is technically configured as the first member of an Avamar replication pair. In this case, no redundancy exists, but it can be added later by adding a second member to each replication pair.
To achieve this, the Federation Enterprise Hybrid Cloud uses the concepts of primary and secondary Avamar instances within each replication pair, and the ability to reverse these personalities so that, in the event of a failure, backup and restore operations can continue. The primary Avamar instance is where all scheduled backups are executed. It is also the instance that Federation Enterprise Hybrid Cloud on-demand backup and restore features communicate with in response to dynamic user requests. The primary Avamar instance also has all the currently active replication groups, making it responsible for replication of new backup sets to the secondary Avamar instance. The secondary Avamar instance has the same configurations for backup and replication policies, except that BaaS workflows initially configure these policies in a disabled state. If the primary Avamar instance becomes unavailable, the policies on the secondary Avamar instance can be enabled via the Toggle Single Avamar Pair Designations catalog item to enable backup and replication operations to continue. Note: Replication operations do not catch up until the original primary Avamar instance (now designated as secondary) becomes available again, at which time replication automatically transmits newer backup sets to the secondary system.
In this solution, after a redundant Avamar configuration is enabled, the Federation Enterprise Hybrid Cloud workflows configure all subsequent backups with replication enabled. If one member of the Avamar replication pair is offline, backups taken to the surviving member of the pair will automatically be replicated once the offline member is brought back online.
79
Chapter 7: Data Protection
How each Avamar instance in a replication pair operates varies based on which backup topology is configured, and is described in the context of each individual topology later in this chapter.
VMware vCenter folder structure and backup service level relationship
When a backup service level is created via the Create Backup Service Level vRealize Automation catalog item, it creates an associated set of folders in the cloud vCenter (or both cloud vCenters if done in a dual-site/dual vCenter environment). The number of folders created depends on how many Avamar pairs are present, and these folders become part of the mechanism for distributing the backup load. Note: In a DR dual-site/dual vCenter environment, the Create a Backup Service Level catalog item automatically creates Site Recovery Manager folder mappings between the new folders created in the first cloud vCenter and their corresponding folders in the second vCenter.
Example If you create a backup service level named Daily-7yr in your environment, and four Avamar replication pairs (numbered 0 through 3) are present, then the following folders are created in the relevant cloud vCenter servers:
Daily-7yr-Pair0
Daily-7yr-Pair1
Daily-7yr-Pair2
Daily-7yr-Pair3
When you assign a virtual machine to the Daily-7yr backup policy, the workflows use a selection algorithm to determine the Avamar pair with least load, and then assign the virtual machine to the associated folder. So if Avamar-Pair2 is determined to be the best target, then the virtual machine is moved to the Daily-7yr-Pair2 vCenter folder and automatically backed up by Avamar-Pair2 as a result. How the Avamar instances are assigned to monitor and backup these folders differs dependent on which backup topology is deployed, and is described in the context of each individual topology later in this chapter.
Avamar pair to vSphere cluster association
Avamar image-level backups work by mounting snapshots of VMDKs to Avamar proxy virtual machines and then backing up the data to the Avamar instance that the Avamar proxy is registered with. In a fully deployed Federation Enterprise Hybrid Cloud with up to 10,000 user virtual machines and hundreds of vSphere clusters, this could lead to Avamar proxy sprawl if not properly configured and controlled. To do this, the Federation Enterprise Hybrid Cloud associates vSphere clusters to a subset of Avamar replications pairs. This means that a reduced number of Avamar proxy virtual machines are required to service the cloud. Associations between a vSphere cluster and an Avamar pair is done via the Federation Enterprise Hybrid Cloud BaaS Associate Avamar Pairs with vSphere Cluster catalog item. Note: In a DR dual-site/dual vCenter topology, when a protected cluster is associated with an Avamar pair, the associated recovery cluster is automatically associated with the same Avamar pair to ensure continuity of service on failover.
80
Chapter 7: Data Protection
Avamar designations
In the redundant Avamar/single vCenter configuration, there are two Avamar instances in each pair, and both are assigned to monitor the same vCenter folder and to backup any virtual machines that folder contains. To ensure that this does not result in both instances backing up the same virtual machine and then replicating each backup (four copies in total), the Federation Enterprise Hybrid Cloud uses primary and secondary Avamar instances within each replication pair, and the ability to reverse these personalities so that, in the event of a failure, backup and restore operations can continue. The primary Avamar instance is where all scheduled backups are executed. It is also the instance that Federation Enterprise Hybrid Cloud on-demand backup and restore features communicate with in response to dynamic user requests. The primary Avamar instance also has all the currently active replication groups, making it responsible for replication of new backup sets to the secondary Avamar instance. The secondary Avamar instance has the same configurations for backup and replication policies, except that BaaS workflows initially configure these policies in a disabled state. If the primary Avamar instance becomes unavailable, the policies on the secondary Avamar instance can be enabled via the Toggle Single Avamar Pair Designations catalog item to enable backup and replication operations to continue. Note: Avamar designations are only relevant in the redundant Avamar/single vCenter topology, because the standard Avamar configuration does not have replication, and in the redundant Avamar/dual vCenter configuration each member of a pair is configured to monitor a folder from only one of the two vCenters.
Avamar proxy server configuration
To associate an Avamar pair with a vSphere cluster, an Avamar proxy virtual machine needs to be deployed to that cluster.
Standard Avamar configuration In single-site topologies, all proxies are on the same site. Therefore, the minimum number of proxy virtual machines required per Avamar pair for each cluster is one. Two is recommended for high availability, if there is scope within the overall number of proxies that can be deployed to the environment. Ideally, this number should be in the region of 60 to 80 proxies.
Redundant Avamar/single vCenter configuration As the virtual machines on every vSphere cluster could be backed up by either of the members of an Avamar replication pair at different points in time, proxies for both the primary and secondary Avamar instances of every associated Avamar replicated pair should be deployed to every vSphere cluster. This means a minimum of two proxies is required. Four proxies would provide additional resilience, if the scope exists within the overall number of proxies that can be deployed to the environment. If the environment also includes CA, then the proxies for the Site A instances should be bound to Site A using virtual machine DRS affinity groups with a DRS virtual machine to host rule that sets those virtual machines to must run on a host DRS group that contains the Site A hosts. Similarly, proxies for the Site B Avamar instance should be bound to Site B hosts. This ensures that no unnecessary cross-WAN backups occur, as Avamar can use vStorage APIs for Data Protection to add VMDKs (from the local leg of the VPLEX volume) to proxy virtual machines bound to physical hosts on the same site as the primary Avamar instance.
81
Chapter 7: Data Protection
Redundant Avamar/dual vCenter configuration In a dual-site/dual vCenter configuration, each vSphere cluster must have an Avamar proxy virtual machine for the local Avamar instance of every Avamar replicated pair associated with it. This ensures backups are taken locally and replicated to the other member of the Avamar pair. In a dual-site/dual vCenter configuration with DR, when a failover occurs, virtual machines will be moved from the vCenter folders on Site A to their corresponding vCenter folders on Site B, at which point the other member of the Avamar replication pair will assume responsibility for backing up and restoring those virtual machines. Therefore, each vSphere cluster still only requires a minimum of one Avamar proxy for every Avamar instance that is associated with it. Two will provide extra resilience. Note: In this configuration, if a failure of a single Avamar instance occurs without the failure of the vCenter infrastructure on the same site, then the second member of the Avamar replication pair will not automatically assume responsibility to backup virtual machines. To further protect against this scenario, additional resilience can be added on each site by using an Avamar RAIN grid.
Avamar administratively full
Determining that a backup target, in this case an Avamar instance, has reached capacity can be based on a number of metrics of the virtual machines it is responsible for protecting, including:
The number of virtual machines assigned to the instance
The total capacity of those virtual machines
The rate of change of the data of those virtual machines
The effective deduplication ratio that can be achieved while backing up those virtual machines
The available network bandwidth and backup window size
Because using these metrics can be somewhat subjective, the Federation Enterprise Hybrid Cloud provides the ability for an administrator to preclude an Avamar instance or Avamar replication pair from being assigned further workload by setting a binary Administrative Full flag set via the Set Avamar to Administrative Full vRealize Automation catalog item. When a virtual machine is enabled for data protection via Federation Enterprise Hybrid Cloud BaaS workflows, the available Avamar instances are assessed to determine the most suitable target. If an Avamar instance or Avamar replication pair has had the Administrative Full flag set, then that instance/pair is excluded from the selection algorithm but continues to back up its existing workloads through on-demand or scheduled backups. If workloads are retired and an Avamar instance or pair is determined to have free capacity, again the Administrative Full flag can be toggled back, including it in the selection algorithm.
Policy-based replication
82
Policy-based replication provides granular control of the replication process. With policybased replication, you create replication groups in Avamar Administrator to define the following replication settings:
Members of the replication group, which are either entire domains or individual clients
Priority for the order in which backup data replicates
Types of backups to replicate based on the retention setting for the backup or the date on which the backup occurred
Maximum number of backups to replicate for each client
Chapter 7: Data Protection
Destination server for the replicated backups
Schedule for replication
Retention period of replicated backups on the destination server
The redundant Avamar configurations automatically create a replication group associated with each backup policy and configure it with a 60-minute stagger to the interval associated with the backup policy. This enables the backups to complete before the replication starts. Note: This schedule can be manually altered within the Avamar GUI, but it is important that you make changes to both the primary and secondary versions of the replication group schedule so that replication operates as required if the Avamar personalities are reversed.
Replication control
If Data Domain is used as a backup target, Avamar is responsible for replication of Avamar data from the source Data Domain system to the destination Data Domain system. As a result, all configuration and monitoring of replication is done via the Avamar server. This includes the schedule on which Avamar data is replicated between Data Domain units. You cannot schedule replication of data on the Data Domain system separately from the replication of data on the Avamar server. There is no way to track replication by using Data Domain administration tools. Note: Do not configure Data Domain replication to replicate data to another Data Domain system that is configured for use with Avamar. When you use Data Domain replication, the replicated data does not refer to the associated remote Avamar server.
83
Chapter 7: Data Protection
Architecture
This section describes the features of the standard Avamar configuration shown in Figure 38 and the environments where it may be used.
Figure 38.
Scenarios for use
Standard Avamar configuration architecture
Best use The most logical fit for a standard Avamar configuration is a single-site Federation Enterprise Hybrid Cloud deployment.
Alternate uses The standard Avamar configuration can be used in topologies such as CA dual-site and DR dual-site topologies with the following caveats:
84
The architecture provides no resilience on the secondary site in either of the dual-site topologies. If the site that hosts the Avamar instances is lost, then there is no ability to restore from backup.
In the CA dual-site/single vCenter topology, any virtual machines that reside on the site with no Avamar instances present will back up across the WAN connection.
In the DR dual-site/dual vCenter topology, any virtual machines that reside on the recovery site (and therefore are registered with a different vCenter) have no ability to back up.
Chapter 7: Data Protection
In the standard Avamar configuration, if the Create Backup Service level workflow creates a folder named Daily-7yr, and there are four Avamar replications pairs present, then it will configure the following backup policies with the Avamar replication pairs:
Avamar-Pair0: Assigned to monitor vCenter folder Daily-7yr-Pair0
Avamar-Pair1: Assigned to monitor vCenter folder Daily-7yr-Pair1
Avamar-Pair2: Assigned to monitor vCenter folder Daily-7yr-Pair2
Avamar-Pair3: Assigned to monitor vCenter folder Daily-7yr-Pair3
In this case, each pair has only one member, and therefore only one Avamar instance is monitoring each folder.
Characteristics
The characteristics of the standard Avamar configuration are:
All Avamar instances are standalone, that is, backup sets are not replicated to a secondary Avamar system.
It works in the context of a single cloud vCenter only.
All Avamar instances contain active backup policies.
Note: An Avamar instance can be set to administratively full and still have active backup policies.
Distribution examples
All Avamar instances are considered to be on the same site, and therefore the roundrobin distribution of virtual machines to vCenter folders includes all Avamar instances that are:
Assigned to the vSphere cluster that the virtual machine is on.
Are not set to Administratively Full.
The following scenarios convey how virtual machines are assigned to vCenter folders to distribute load evenly across Avamar instances, assuming the following configuration, as shown in Figure 38:
Four Avamar instances and two vSphere clusters exist
AV_REP_PAIR0 and AV_REP_PAIR1 are assigned to Cluster 1
AV_REP_PAIR2 and AV_REP_PAIR3 are assigned to Cluster 2
Note: In this example all virtual machines are deployed to the backup policy named Daily-7yr.
Scenario 1: VM1 is deployed to Cluster 1 - No other workload virtual machines exist
AV_REP_PAIR2 and AV_REP_PAIR3 are ruled out because they are not assigned to
Cluster 1.
AV_REP_PAIR0 and AV_REP_PAIR1 are identified as potential targets.
The expected results are:
The virtual machine is deployed to Cluster 1.
It is placed in a folder named Daily-7yr-Pair0, indicating assignment to AV_REP_PAIR0. AV_REP_PAIR1 is an equally viable candidate as both grids are empty, but AV_REP_PAIR0 is selected based on numerical order.
85
Chapter 7: Data Protection
Scenario 2: VM2 is deployed to Cluster 1 - VM1 exists
AV_REP_PAIR2 and AV_REP_PAIR3 are ruled out because they are not assigned to
Cluster 1.
AV_REP_PAIR0 and AV_REP_PAIR1 are identified as potential targets.
The expected results are:
The virtual machine is deployed to Cluster 1.
It is placed in a folder named Daily-7yr-Pair1 indicating assignment to AV_REP_PAIR1 because the round-robin algorithm determined that AV_REP_PAIR1 had fewer virtual machines than the other candidate AV_REP_PAIR0.
VM3 follows a similar logic and ends up being managed by AV_REP_PAIR2 while VM4 is managed by AV_REP_PAIR3.
Architecture
This section describes the features of the redundant Avamar/single vCenter configuration shown in Figure 39 and the environments where it can be used.
Figure 39.
86
Redundant Avamar/single vCenter configuration
Chapter 7: Data Protection
Scenarios for use
Best use The most logical fit for a redundant Avamar/single vCenter configuration is a dual-site/single vCenter Federation Enterprise Hybrid Cloud deployment.
Alternate uses The redundant Avamar/single vCenter configuration can be used in the single-site topology with no caveats to provide a backup infrastructure that can tolerate the loss of a physical Avamar instance. Note: The redundant Avamar/single vCenter should not be used in a DR dual-site topology because doing so imposes caveats that can be overcome using the redundant Avamar/dual vCenter configuration without the need for any extra components.
vCenter folder assignments
In the redundant Avamar/single vCenter configuration, if the Create Backup Service level workflow creates a folder called Daily-7yr and there are four Avamar replications pairs present then it will configure the following backup policies with the Avamar replication pairs:
Avamar-Pair0: Assigned to monitor vCenter folder Daily-7yr-Pair0
Avamar-Pair1: Assigned to monitor vCenter folder Daily-7yr-Pair1
Avamar-Pair2: Assigned to monitor vCenter folder Daily-7yr-Pair2
Avamar-Pair3: Assigned to monitor vCenter folder Daily-7yr-Pair3
As there is only one vCenter, and therefore only one vCenter folder per Avamar replication pair, each Avamar instance in the pair is configured to monitor the same vCenter folder. At this point, the concept of primary and secondary Avamar members are employed to ensure that only one member of the pair is actively backing up and replicating the virtual machines at any given point in time.
Characteristics
The characteristics of the redundant Avamar/single vCenter configuration are:
All Avamar instances are configured in pairs and all backups are replicated.
It works in the context of a single cloud vCenter only.
Fifty percent of the Avamar instances have active backup and replication polices at any given point in time (50 percent of the Avamar instances are primary, 50 percent are secondary.) Note: Primary means that the backup policies on that instance are enabled. An Avamar instance can be set to administratively full and still be considered primary.
Avamar replication pairs are defined as split across sites, and therefore the roundrobin distribution of virtual machines to vCenter folders includes all Avamar pairs that:
Are assigned to the vSphere cluster that the virtual machine is on.
Have their primary member on the same site as the virtual machine DRS Affinity group that the virtual machine is a member of.
Are not set to Administratively Full.
87
Chapter 7: Data Protection
Distribution examples
The following scenarios convey how virtual machines are assigned to vCenter folders in order to distribute load evenly across Avamar instances, assuming the following configuration (as shown in Figure 39):
Six primary Avamar instances, six secondary instances. and two vSphere clusters exist
AV_REP_PAIR0 through AV_REP_PAIR3 are assigned to Cluster 1
AV_REP_PAIR4 and AV_REP_PAIR5 are assigned to Cluster 2
Note: In this example, all virtual machines are deployed to a backup policy named Daily-7yr.
Scenario 1: VM1 is deployed to Cluster 1, Site A - No other workload virtual machines exist
AV_REP_PAIR4 and AV_REP_PAIR5 are ruled out because they are not assigned to
Cluster 1.
AV_REP_PAIR1 and AV_REP_PAIR3 are ruled out for being primary on Site B.
AV_REP_PAIR0 and AV_REP_PAIR2 are identified as potential targets.
The expected results are:
The virtual machine is deployed to Cluster 1 – Host CL1-H1.
It is placed in a folder named Daily-7yr-Pair0 indicating assignment to AV_REP_PAIR0. AV_REP_PAIR2 is an equally viable candidate as both grids are empty, but AV_REP_PAIR0 is chosen based on numerical order.
Scenario 2: VM2 is deployed to Cluster 1, Site A - VM1 exists
AV_REP_PAIR4 and AV_REP_PAIR5 are ruled out because they are not assigned to
Cluster 1.
AV_REP_PAIR1 and AV_REP_PAIR3 are ruled out because their primary instances
are on Site B.
AV_REP_PAIR0 and AV_REP_PAIR2 are identified as potential targets.
The expected results are:
The virtual machine is deployed to Cluster 1 – Host CL1-H1.
It is placed in a folder named Daily-7yr-Pair2 indicating assignment to AV_REP_PAIR2 because the round-robin algorithm determined that AV_REP_PAIR2 had fewer virtual machines than the other candidate AV_REP_PAIR0.
Scenario 3: VM3 is deployed to Cluster 1, Site B - VM1 and VM2 exist
AV_REP_PAIR4 and AV_REP_PAIR5 are ruled out because they are not assigned to
Cluster 1.
AV_REP_PAIR0 and AV_REP_PAIR2 are ruled out because their primary instances
are on Site A.
88
AV_REP_PAIR1 and AV_REP_PAIR3 are identified as potential targets.
The expected results are:
The virtual machine is deployed to Cluster 1 – Host CL1-H2.
It is placed in a folder named Daily-7yr-Pair1 indicating assignment to AV_REP_PAIR1. AV_REP_PAIR3 is an equally viable candidate as both grids are empty, but AV_REP_PAIR1 is selected based on numerical order.
Chapter 7: Data Protection
Scenario 4: VM4 is deployed to Cluster 1, Site B - VM1, VM2, and VM3 exist
AV_REP_PAIR4 and AV_REP_PAIR5 are ruled out because they are not assigned to
Cluster 1.
AV_REP_PAIR0 and AV_REP_PAIR2 are ruled out because their primary instances
are on Site A.
AV_REP_PAIR1 and AV_REP_PAIR3 are identified as potential targets.
The expected results are:
The virtual machine is deployed to Cluster 1 – Host CL1-H2.
It is placed in a folder named Daily-7yr-Pair3 indicating assignment to AV_REP_PAIR3 because the round-robin algorithm determined that AV_REP_PAIR3 had fewer virtual machines than the other candidate AV_REP_PAIR1.
When VM5 and VM6 are deployed to Cluster 2, the same logic dictates that VM5 is managed by AV_REP_PAIR4 while VM6 is managed by AV_REP_PAIR5 based on the Cluster to Avamar Pair mappings.
89
Chapter 7: Data Protection
Architecture
This section describes the features of the redundant Avamar/dual vCenter configuration shown in Figure 40 and the environments where it may be used.
Figure 40.
Scenarios for use
Redundant Avamar/dual vCenter configuration
Best use The most logical fit for a redundant Avamar/dual vCenter configuration is a dual-site/dual vCenter Federation Enterprise Hybrid Cloud deployment.
Alternate uses There are no valid alternate uses for this configuration as no other topology uses dual-cloud vCenters.
90
Chapter 7: Data Protection
vCenter folder assignments
In the redundant Avamar/dual vCenter configuration, if the Create Backup Service level workflow creates a folder named Daily-7yr, and there are four Avamar replications pairs present, then it will configure the following backup policies with the Avamar replication pairs:
Avamar-Pair0: Assigned to monitor vCenter folder Daily-7yr-Pair0
Avamar-Pair1: Assigned to monitor vCenter folder Daily-7yr-Pair1
Avamar-Pair2: Assigned to monitor vCenter folder Daily-7yr-Pair2
Avamar-Pair3: Assigned to monitor vCenter folder Daily-7yr-Pair3
Because there are two vCenters, each Avamar instance in a pair is configured to monitor one of the two corresponding vCenter folders, that is, the instance on Site A monitors the folder from the Site A vCenter, and the instance from Site B monitors the folder from the Site B vCenter. As a virtual machine can only be one of the two folders (even in a DR dualsite/dual vCenter topology) there is no duplication of backups. Note: When VMware Site Recovery Manager is used, placeholder virtual machines are created as part of the Site Recovery Manager protection process. To ensure that Avamar does not detect these placeholder virtual machines, additional folders are created in each vCenter with a ‘_PH’ suffix, and placeholder virtual machines are located in these folders via Site Recovery Manager folder mappings. Before failing over a DR cluster, run the Prepare for DP Failover catalog item. This moves the production virtual machines out of their service level folders on the protected site, so that their placeholders are not created in an Avamar monitored folders when Site Recovery Manager re-protects the virtual machine after failover.
Characteristics
The characteristics of the redundant Avamar/single vCenter configuration are:
All Avamar instances are configured in pairs and all backups are replicated.
It works in the context of a dual-cloud vCenter only. Note: An Avamar instance can be set to Administratively Full and still have active backup and replication polices.
Distribution examples
Avamar replication pairs are defined as being split across sites, and therefore the round-robin distribution of virtual machines to vCenter folders include all Avamar pairs that are:
Assigned to the vSphere cluster that the virtual machine is on.
Not set to Administratively Full.
The following scenarios convey how virtual machines are assigned to vCenter folders in order to distribute load evenly across Avamar instances, assuming the following configuration (as shown in Figure 39):
Six active Avamar instances (in three Avamar replication pairs), three protected vSphere clusters, three recovery vSphere clusters, and two local clusters exist
AV_REP_PAIR0 through AV_REP_PAIR1 are assigned to Clusters 1 through 6
AV_REP_PAIR2 is assigned to Clusters 7 and 8
Note: All virtual machines are deployed to the backup policy named Daily-7yr for the example.
Scenario 1: VM1 is deployed to Cluster 1 - No other workload virtual machines exist
AV_REP_PAIR2 is ruled out because it is not assigned to Cluster 1.
AV_REP_PAIR0 and AV_REP_PAIR1 are identified as potential targets.
91
Chapter 7: Data Protection
The expected results are:
The virtual machine is deployed to Cluster 1 – Host CL1-H1.
It is placed in a folder named Daily-7yr-Pair0 indicating assignment to AV_REP_PAIR0. AV_REP_PAIR1 is an equally viable candidate as both grids are empty, but AV_REP_PAIR0 is selected based on numerical order.
Because Cluster 1 is on Site A, AV_INSTANCE_00 will back up the virtual machine and replicate the backups to AV_INSTANCE_01.
Scenario 2: VM2 is deployed to Cluster 1 - VM1 exists
AV_REP_PAIR2 is ruled out as it is not assigned to Cluster 1.
AV_REP_PAIR0 and AV_REP_PAIR1 are identified as potential targets
The expected results are:
The virtual machine is deployed to Cluster 1 – Host CL1-H1.
It is placed in a folder named Daily-7yr-Pair1 indicating assignment to AV_REP_PAIR1 because the round-robin algorithm determined that AV_REP_PAIR2 had fewer virtual machines than the other candidate, AV_REP_PAIR0.
Because Cluster 1 is on Site A, AV_INSTANCE_02 will back up the virtual machine and replicate the backups to AV_INSTANCE_03.
Scenario 3: VM3 is deployed to Cluster 3 - VM1 and VM2 exist
AV_REP_PAIR2 is ruled out as it is not assigned to Cluster 3.
AV_REP_PAIR0 and AV_REP_PAIR1 are identified as potential targets.
The expected results are:
The virtual machine is deployed to Cluster 3 – Host CL3-H1.
It is placed in a folder named Daily-7yr-Pair0 indicating assignment to AV_REP_PAIR0. AV_REP_PAIR1 is an equally viable candidate as both have equal virtual machines (one assigned to each pair globally, but none to the instances on Site B), but AV_REP_PAIR0 is selected based on numerical order.
Scenario 4: VM4 is deployed to Cluster 3 - VM1, VM2, and VM3 exist
AV_REP_PAIR2 is ruled out, because it is not assigned to Cluster 3.
AV_REP_PAIR0 and AV_REP_PAIR1 are identified as potential targets.
The expected results are:
The virtual machine is deployed to Cluster 3 – Host CL3-H1.
It is placed in a folder named Daily-7yr-Pair1 indicating assignment to AV_REP_PAIR1 because the round-robin algorithm determined that AV_REP_PAIR1 had fewer virtual machines than the other candidate AV_REP_PAIR0.
Because Cluster 3 is on Site B, AV_INSTANCE_03 will back up the virtual machine and replicate the backups to AV_INSTANCE_02.
Scenario 5: VM5 is deployed to Cluster 5 - VM1, VM2, VM3 and VM4 exist
92
AV_REP_PAIR2 is ruled out, because it is not assigned to Cluster 5.
AV_REP_PAIR0 and AV_REP_PAIR1 are identified as potential targets.
The expected results are:
The virtual machine is deployed to Cluster 5 – Host CL5-H1.
It is placed in a folder named Daily-7yr-Pair0 indicating assignment to AV_REP_PAIR0. AV_REP_PAIR1 is an equally viable candidate because both have
Chapter 7: Data Protection
equal virtual machines (two assigned to each pair globally), but AV_REP_PAIR0 is selected based on numerical order.
Because Cluster 5 is on Site A, AV_INSTANCE_00 will back up the virtual machine and replicate the backups to AV_INSTANCE_01.
Scenario 6: VM6 is deployed to Cluster 6 - VM1, VM2, VM3, VM4 and VM5 exist
AV_REP_PAIR2 is ruled out, because it is not assigned to Cluster 6.
AV_REP_PAIR0 and AV_REP_PAIR1 are identified as potential targets.
The expected results are:
The virtual machine is deployed to Cluster 6 – Host CL6-H1.
It is placed in a folder named Daily-7yr-Pair1 indicating assignment to AV_REP_PAIR1 because the round-robin algorithm determined that AV_REP_PAIR1 had fewer virtual machines than the other candidate AV_REP_PAIR0.
Because Cluster 6 is on Site B, AV_INSTANCE_03 will back up the virtual machine and replicate the backups to AV_INSTANCE_02.
Scenario 7: VM7 is deployed to Cluster 7 - VM1, VM2, VM3, VM4, VM5 and VM6 exist
AV_REP_PAIR0 and AV_REP_PAIR1 are ruled out, because they are not assigned to
Cluster 7.
AV_REP_PAIR2 is identified as the only potential target.
The expected results are:
The virtual machine is deployed to Cluster 7 – Host CL7-H1.
It is placed in a folder named Daily-7yr-Pair2 indicating assignment to AV_REP_PAIR3. We found no other viable candidates.
Because Cluster 7 is on Site A, AV_INSTANCE_05 will back up the virtual machine and replicate the backups to AV_INSTANCE_05.
93
Chapter 7: Data Protection
Redundant Avamar/dual vCenter proxy example
Figure 41 shows an example of how proxies might be configured in a redundant Avamar/dual vCenter environment
Figure 41.
94
Redundant Avamar/dual vCenter proxy example
Chapter 8: Solution Rules and Permitted Configurations
This chapter presents the following topics:
Overview ..........................................................................................................96 Architectural assumptions ...................................................................................96 VMware Platform Services Controller ....................................................................96 VMware vRealize tenants and business groups .......................................................98 EMC ViPR tenants and projects ............................................................................99 General storage considerations .......................................................................... 100 VMware vCenter endpoints ................................................................................ 100 Permitted topology configurations ...................................................................... 101 Permitted topology upgrade paths ...................................................................... 102 Bulk import of virtual machines ......................................................................... 103 DR dual-site/dual vCenter topology restrictions.................................................... 104 Resource sharing ............................................................................................. 106 Data protection considerations ........................................................................... 106 Software resources .......................................................................................... 106 Sizing guidance ............................................................................................... 106
95
Chapter 8: Solution Rules and Permitted Configurations
This chapter looks at the rules, configurations, and dependencies between the Federation Enterprise Hybrid Cloud components and their constructs, outlining how this influences the supported configurations within the cloud.
Assumption and justifications
The following assumptions and justifications apply to the Federation Enterprise Hybrid Cloud architecture:
The appliance based version of vCenter is not supported. The vCenter Server full installation is used because it:
Provides support for an external Microsoft SQL Server database
Resides on a Windows System that also supports the VMware Update Manager™ service, enabling minimal resource requirements in smaller configurations
VMware Platform Services Controller is used instead of the vRealize Automation Identity Appliance because it supports the multisite, single sign-on requirements of the solution
Windows-based Platform Services Controllers are used as they are the natural upgrade path from previous versions of Federation Enterprise Hybrid Cloud
The appliance-based versions of Platform Services Controllers are not supported
This solution uses VMware Platform Services Controller in place of the vRealize Automation Identity Appliance. VMware Platform Services Controller is deployed on the dedicated virtual machine server in each Core Pod (multiple Core Pods exist in the DR dual-site/dual vCenter topology) and an additional Platform Services Controller (Auto-Platform Services Controller) is deployed on a server in the Automation Pod. The Auto-Platform Services Controller server provides authentication services to all the Automation Pod management components requiring Platform Services Controller integration. This configuration enables authentication services to fail over with the other automation components and enables a seamless transition between Site A and Site B. There is no need to change IP addresses, DNS, or management component settings.
Platform Services Controller domains
The Federation Enterprise Hybrid Cloud uses one or more Platform Services Controller domains depending on the management platform deployed. Platform Services Controller instances are configured within those domains according to the following model:
96
External Platform Services Controller domain (distributed management model only)
First external Platform Services Controller
Second external Platform Services Controller (DR dual-site/dual vCenter topology only)
Cloud SSO domain (All topologies)
First Cloud Platform Services Controller
Automation Pod Platform Services Controller
Second Cloud Platform Services Controller (DR dual-site/dual vCenter topology only)
Chapter 8: Solution Rules and Permitted Configurations
Figure 42 shows the Platform Services Controller domains and how each Platform Services Controller instance and domain required are configured.
Figure 42.
First Platform Services Controller instance in each single sign-on domain
SSO domain and vCenter SSO instance relationships
This first VMware Platform Services Controller deployed in each single sign-on domain is deployed by creating a new vCenter Single Sign-On domain, enabling it to participate in the default vCenter Single Sign-On namespace (vsphere.local). This primary Platform Services Controller server supports identity sources, such as Active Directory, OpenLDAP, local operating system users, and SSO embedded users and groups. This is the default deployment mode when you install VMware Platform Services Controller.
97
Chapter 8: Solution Rules and Permitted Configurations
Subsequent vCenter Single Sign-On instances in each single sign-on domain
Additional VMware Platform Services Controller instances are installed by joining the new Platform Services Controller to an existing single sign-on domain, making them part of the existing domain, but in a new SSO site. When you create Platform Services Controller servers in this fashion, the deployed Platform Services Controller instances become members of the same authentication namespace as the Platform Services Controller instance. This deployment mode should only be used after you have deployed the first Platform Services Controller instance in each single sign-on domain. In vSphere 6.0, VMware Platform Services Controller single sign-on data (such as policies, solution users, application users, and identity sources) is automatically replicated between each Platform Services Controller instance in the same authentication namespace every 30 seconds.
vRealize tenant design
The Federation Enterprise Hybrid Cloud can operate using single or multiple vRealize Automation tenants. STaaS operations rely on the tenant URL value configured as part of the vRealize Automation tenant, and therefore require an individual vRealize Orchestrator server per additional tenant, if the ability for multiple tenants to execute STaaS operations is required. The Federation Enterprise Hybrid Cloud foundation package needs to be installed on each of the vRealize Orchestrator servers, entering the relevant tenant URL during installation. This is also required in order to populate the vRealize Automation catalog in each tenant with the relevant STaaS catalog items.
vRealize tenant best practice
As the vRealize Automation IaaS administrator is a system-wide role, having multiple tenants configure endpoints and execute STaaS operations may not provide any additional value over and above the use of a single tenant with multiple business groups. Therefore, while multiple tenants are supported, Federation Enterprise Hybrid Cloud is normally deployed with a single tenant with respect to STaaS and Data Protection operations.
vRealize business group design
Federation Enterprise Hybrid Cloud uses two system business groups in each non-default tenant. The first, EHCSystem, is used as the target for installation of the vRealize Automation advanced services STaaS catalog items. It does not require any compute resources. The second system business group, EHCOperations, is used as the group where Federation Enterprise Hybrid Cloud storage administrators are configured. It is given entitlements to the STaaS and Cluster Onboarding catalog items. It has no compute resource requirements.
vRealize business group best practice
The Federation recommends that applications provisioned using vRealize Automation Application Services each have a separate business group per application type to enable administrative separation of blueprint creation and manipulation. Figure 43 shows an example where the EHCSystem and EHCOperations system business groups are configured alongside three tenant business groups (IT, HR, and Manufacturing) and three application business groups used by vRealize Automation Application Services for Microsoft SharePoint, Oracle, and Microsoft Exchange.
98
Chapter 8: Solution Rules and Permitted Configurations
Figure 43.
Software-defined data center tenant design and endpoints
ViPR tenants
The Federation Enterprise Hybrid Cloud uses a single ViPR tenant. The default provider tenant or an additional non-default tenant may be used.
ViPR projects
Federation Enterprise Hybrid Cloud STaaS operations rely on a correlation between the tenant URL value of the user executing the request and a ViPR project name. Therefore, to enable STaaS for an additional vRealize tenant, you must create a corresponding ViPR project whose name and case match that of the vRealize tenant URL. As each project can have a maximum total storage capacity (quota) associated with it that cannot be exceeded, the use of multiple ViPR projects enables multiple vRealize Automation tenants within the Federation Enterprise Hybrid Cloud to provision storage from the same storage endpoints in a controlled or limited fashion.
ViPR consistency groups
ViPR consistency groups are an important component of the CA and DR topologies for the Federation Enterprise Hybrid Cloud solution. Consistency groups logically group volumes within a project to ensure that a set of common properties is applied to an entire group of volumes during a fault event. This ensures host-to-cluster or application-level consistency when a failover occurs. Consistency groups are created by Federation Enterprise Hybrid Cloud STaaS operations and are specified when CA or DR-protected volumes are provisioned. Consistency group names must be unique within the ViPR environment. When used with VPLEX in the CA dual-site/single vCenter, these consistency groups are created per physical array, per vSphere cluster, and per site. When used with RecoverPoint in a DR dual-site/dual vCenter configuration, these consistency groups are created in 1:1 relationship with the vSphere datastore/LUN.
99
Chapter 8: Solution Rules and Permitted Configurations
vSphere datastore clusters
VMware Raw Device Mappings (RDMs)
The Federation Enterprise Hybrid Cloud does not support datastore clusters for the following reasons: 1.
Linked clones do not work with datastore clusters, causing multi machine blueprints to fail unless configured with explicitly different reservations for edge devices.
2.
vRealize Automation already performs capacity analysis during initial placement.
3.
Can result in inconsistent behavior when virtual machines report their location to vRealize Automation. Misalignment with vRealize reservations can makes the virtual machine un-editable.
4.
Federation Enterprise Hybrid Cloud STaaS operations do not place new LUNs into datastore clusters, therefore all datastore clusters would have to be manually maintained.
5.
Specific to DR: a.
Day 2 storage DRS migrations between datastores would break the Site Recovery Manager protection for the virtual machines moved.
b.
Day 2 storage DRS migrations between datastores would result in rereplicating the entire virtual machine to the secondary site.
VMware Raw Device Mappings are not created by or supported by Federation Enterprise Hybrid Cloud STaaS services. If created outside of STaaS services, then any issues arising from their use will not be supported by Federation Enterprise Hybrid Cloud customer support. If changes are required in the environment to make them operate correctly, then a Federation Enterprise Hybrid Cloud RPQ should be submitted first for approval. Federation Enterprise Hybrid Cloud support teams may request that you back out any change made to the environment to facilitate RDMs.
Multiple vCenter endpoints are supported within Federation Enterprise Hybrid Cloud. However depending on the topology chosen, there are certain considerations as outlined in this section.
Single-site/single vCenter and dualsite/single vCenter topologies
These topologies can support:
Only one vCenter per tenant with the ability to execute STaaS services.
More than one vCenter per tenant as long as the second and subsequent vCenter endpoints do not require STaaS or BaaS services.
Enabling additional STaaS-enabled vCenter endpoints To enable additional STaaS-enabled vCenter endpoints, these topologies require a separate vRealize tenant per vCenter endpoint for the following reasons:
100
Federation Enterprise Hybrid Cloud STaaS catalog items use vRealize Orchestrator through the vRealize Automation advanced server configuration, which only allows one vRealize Orchestrator to be configured per tenant.
This vRealize Orchestrator stores important vCenter configuration details gathered during the Federation Enterprise Hybrid Cloud Foundation installation process.
To store additional vCenter configuration details, an additional vRealize Orchestrator is required.
Specifying the additional vRealize Orchestrator as an advanced server configuration requires an additional tenant.
Chapter 8: Solution Rules and Permitted Configurations
Each vCenter endpoint requires its own independent vRealize Orchestrator server and NSX Manager instance.
The vRealize Orchestrator consideration is based on the additional tenant consideration above.
The NSX Manager requirement is based on the VMware requirement for a 1:1 relationship between vCenter and NSX.
Enabling additional BaaS-enabled vCenter endpoints These topologies require independent Avamar instances for each vCenter endpoint to enable BaaS services. Note: As backup service level names use a common vRealize dictionary, backup service levels created by each tenant will be visible to all tenants. Therefore, it is advisable to name backup service levels to indicate which tenant they were created for. This enables an operator to identify the backup service levels relevant to them.
Dual-site/dual vCenter topologies
This topology can support:
Two vCenters per tenant with the ability to execute STaaS and BaaS services.
More than two vCenters per tenant as long as the third and subsequent vCenter endpoints do not require STaaS, BaaS or DRaaS services.
Enabling additional STaaS and BaaS-enabled vCenter endpoints Additional STaaS and BaaS enabled vCenter endpoints require additional tenants and Avamar instances similar to the single vCenter topologies. Note: As backup service level names use a common vRealize dictionary, backup service levels created by each tenant will be visible to all tenants. Therefore it is advisable to name backup service levels to indicate which tenant they were created for. This enables an operator to identify the backup service levels relevant to them.
Combining topologies
The following configurations are permitted for each Federation Enterprise Hybrid Cloud instance:
Local only (single-site/single vCenter)
Local plus CA combined
Uses the CA dual-site/single vCenter topology and provides local-only and CA functionality via distinct Workload Pods with separate networks and storage
Local plus DR combined
Uses the DR dual-site/dual vCenter topology and provides local-only and DR functionality via distinct Workload Pods with separate networks and storage
Note: Federation Enterprise Hybrid Cloud 3.5 does not support both DR and CA functionality on the same Federation Enterprise Hybrid Cloud instance.
101
Chapter 8: Solution Rules and Permitted Configurations
Single site to continuous availability upgrade
Single-site Federation Enterprise Hybrid Cloud deployments can be upgraded to CA dualsite/single vCenter topology by adopting either an online or offline upgrade approach with the following considerations.
Considerations
The topology upgrade is an EMC professional services engagement and provides three basic methods of conversion based on the original storage design
NFS to VPLEX Distributed VMFS (Online via Storage vMotion)
Standard VMFS to VPLEX Distributed VMFS (Offline via VPLEX encapsulation)
VPLEX Local VMFS to VPLEX Distributed VMFS (Online via VPLEX Local to VPLEX Metro conversion)
If NFS is in use for management platform storage, then new VPLEX storage is required.
In non-BaaS environments, local workloads can be migrated to new CA clusters using storage vMotion if required. Note: Federation Enterprise Hybrid Cloud 3.5 does not currently provide an automated mechanism to achieve this. Contact EMC Professional Services to assist in this process.
Existing local-only workload clusters may remain as local-only clusters or be converted to CA-enabled clusters after the topology upgrade. Note: EMC Professional Services should execute this process
In BaaS environments, virtual machines requiring CA protection should remain on the original cluster and the cluster should be converted to a CA-enabled cluster. This is due to the need to carefully manage the relationships of vSphere clusters, Avamar grids, Avamar proxies, and vCenter folder structure to preserve the ability to restore backups taken prior to the topology upgrade.
Single-site to disaster recovery upgrade
After the topology upgrade, new clusters can be provisioned to provide CA or localonly functionality for new tenant virtual machines.
Single-site Federation Enterprise Hybrid Cloud deployments can be upgraded to DR dualsite/dual vCenter topology by adopting with the following considerations:
Considerations
Additional Core and NEI Pod infrastructure and components need to be deployed on the second site.
Additional Automation Pod infrastructure needs to be deployed on the second site to become the target for the Automation Pod failover.
EMC RecoverPoint needs to be installed and configured and all Automation Pod LUNs replicated to the second site. Note: If NFS volumes were used for Automation Pod storage then new FC-based block datastores should be provided, and the Automation Pod components migrated to the new storage using Storage vMotion.
102
Prior to the upgrade, the Automation Pod components must be deployed on a distinct network segment from the Core and NEI Pods.
Chapter 8: Solution Rules and Permitted Configurations
A Microsoft SQL Server instance and a vCenter Single Sign-On role must be deployed to a server in the Automation Pod during the initial deployment.
Migration of previously existing virtual machines from local to DR clusters is not currently supported with default functionality.
Note: If there is a requirement to DR-enabled pre-existing tenant workloads, contact EMC Services teams to provide this as custom functionality.
Importing from virtual machines and adding Federation Enterprise Hybrid Cloud services
For environments that require existing virtual machines to be imported into the Federation Enterprise Hybrid Cloud, the bulk import feature of vRealize Automation enables the import of one of more virtual machines. This functionality is available only to vRealize Automation users who have Fabric Administrator and Business Group Manager privileges. The Bulk Import feature imports virtual machines intact with defining data such as reservation, storage path, blueprint, owner, and any custom properties. Federation Enterprise Hybrid Cloud offers the ability to layer Federation Enterprise Hybrid Cloud services onto pre-existing virtual machines by using and extending the bulk import process. Before beginning the bulk import process, the following conditions must be true:
Target virtual machines are located in an Federation Enterprise Hybrid Cloud vCenter endpoint Note: This is not an additional IaaS-only vCenter endpoint if they are also present.
Target virtual machines must be located on the correct vRealize Automation managed compute resource cluster and that cluster must already be on-boarded as a Federation Enterprise Hybrid Cloud cluster.
In cases where DR services are required for the target virtual machines, then they must be on a DR-enabled cluster.
In cases where data protection services are required for the target virtual machines, then they must be on a cluster that is associated with an Avamar pair.
Target virtual machines must be located on the correct vRealize Automation managed datastore.
In cases where DR services are required for the target virtual machines, then they must be on a datastore protected by EMC RecoverPoint.
In cases where data protection services are required for the target virtual machines, then they must be on a datastore that is registered with an EMC Avamar grid.
Note: The process for importing these virtual machines and adding Federation Enterprise Hybrid Cloud services is documented in the Federation Enterprise Hybrid Cloud 3.5: Administration Guide.
103
Chapter 8: Solution Rules and Permitted Configurations
Multimachine blueprints
Load balancers cannot be deployed as part of a protected multimachine blueprint. However, you can manually edit the upstream Edge to include load-balancing features for a newly deployed multimachine blueprint.
vRealize Automation
Failover state operations Provisioning of virtual machines to a protected DR cluster is permitted at any time, as long as that site is operational. If you provision a virtual machine while the recovery site is unavailable due to vCenter Site Recovery Manager disaster recovery failover, you need to run the DR Remediation catalog item to bring it into protected status when the recovery site is back online. During STaaS provisioning of a protected datastore, Federation Enterprise Hybrid Cloud workflows issue a DR auto-protect attempt for the new datastore with vCenter Site Recovery Manager. If both sites are operational when the request is issued, this should be successful. If, however, one site is offline (vCenter Site Recovery Manager Disaster Recovery Failover) when the request is made, the datastore will be provisioned, but you must run the DR Remediation catalog item to bring it into a protected status. Note: The DR Remediation catalog item can be run at any time to ensure that all DR items are protected correctly.
104
Failover granularity
While replication is at the datastore level, the unit of failover for in a DR configuration is a DR-enabled cluster. It is not possible to failover a subset of virtual machines on a single DRprotected cluster. This is because all networks supporting these virtual machines are converged to the recovery site during a failover.
RecoverPoint cluster limitations
There is also a limit of 64 consistency groups per RecoverPoint appliance and 128 consistency groups per RecoverPoint cluster. Therefore, the number of nodes deployed in the RecoverPoint cluster should be sized to allow appropriate headroom for surviving appliances to take over the workload of failed appliance.
RecoverPoint licensing
The Federation Enterprise Hybrid Cloud supports RecoverPoint CL-based licensing only. It does not support RecoverPoint SE or RecoverPoint EX, as these versions are not currently supported by EMC ViPR.
VMware Site Recovery Manager limitations
Protection maximums Table 11 shows the maximums that apply for Site Recovery Manager-protected resources. Table 11.
Site Recovery Manager protection maximums
Total number of
Maximum
Virtual machines configured for protection using array-based replication
5,000
Virtual machines per protection group
500
Protection groups
250
Recovery plans
250
Protection groups per recovery plan
250
Virtual machines per recovery plan
2,000
Replicated datastores (using array-based replication) and >1 RecoverPoint cluster
255
Chapter 8: Solution Rules and Permitted Configurations
Recovery maximums Table 12 shows the maximums that apply for Site Recovery Manager recovery plans. Table 12.
Implied Federation Enterprise Hybrid Cloud storage maximums
Site Recovery Manager protection maximums
Total number of
Maximum
Concurrently executing recovery plans
10
Concurrently recovering virtual machines using array-based replication
2,000
Table 13 indicates the storage maximums in a Federation Enterprise Hybrid Cloud DR environment, when all other maximums are taken into account. Table 13.
Implied Federation Enterprise Hybrid Cloud storage maximums
Total number of
Maximum
DR enabled datastores per RecoverPoint Consistency Group
1
DR enabled datastores per RecoverPoint Cluster
128
DR enabled datastores per Federation Enterprise Hybrid Cloud environment
250
To ensure maximum protection for DR-enabled vSphere clusters, the Federation Enterprise Hybrid Cloud STaaS workflows create each LUN in its own RecoverPoint consistency group. This ensures that ongoing STaaS provisioning operations have no effect on either the synchronized state of existing LUNs or the history of restore points for those LUNs maintained by EMC RecoverPoint. Because there is a limit of 128 consistency groups per EMC RecoverPoint cluster, there is therefore a limit of 128 Federation Enterprise Hybrid Cloud STaaS provisioned LUNs per RecoverPoint cluster. To extend the scalability further, additional EMC RecoverPoint clusters are required. Each new datastore is added to its own Site Recovery Manager protection group. As there is a limit of 250 protection groups per Site Recovery Manager installation, this limits the total number of datastores in a DR environment to 250, irrespective of the number of RecoverPoint clusters deployed.
Storage support
Supports VMAX, VNX, XtremIO, and VMAX3 (behind VPLEX) only.
Network support
The Federation Enterprise Hybrid Cloud provides fully automated network re-convergence during disaster recovery failover when using VMware NSX only. The use of vSphere Distributed Switch backed by other networking technologies is also permitted, but requires that network re-convergence is carried out manually in accordance with the chosen network technology, or that automation of network re-convergence is developed as a professional services engagement.
NSX security support
Only supports the assignment of blueprint virtual machines to a security group. Does not support the assignment of blueprints to security policies or security tags.
105
Chapter 8: Solution Rules and Permitted Configurations
Resource isolation
As vRealize Automation endpoints are visible to all vRealize Automation IaaS administrators, resource isolation in the truest sense is not possible. However, use of locked blueprints and storage reservation policies can be used to ensure that certain types of workload (such as those whose licensing is based on CPU count) can be restricted to only a subset of the Workload Pods available in the environment. This includes the ability to control those licensing requirements across tenants by ensuring that all relevant deployments are on the same set of compute resources.
Resource sharing
All endpoints configured across the vRealize Automation instance by an IaaS administrator are available to be added to fabric groups, and therefore consumed by any business group across any of the vRealize Automation tenants. Provisioning to vCenter endpoints, however, can still only be done through the tenant configured as part of the Federation Enterprise Hybrid Cloud foundation installation in that tenant and its vRealize Orchestrator server.
106
Application tenant integration
The Federation recommends that applications provisioned using vRealize Automation Application Services each have their own business group by application type to enable administrative separation of blueprint creation and manipulation.
Supported Avamar platforms
The Federation Enterprise Hybrid Cloud supports physical Avamar infrastructure only. It does not support Avamar Virtual Edition
Scale out limits
Federation Enterprise Hybrid Cloud 3.5 supports a maximum of 15 Avamar replication pairs (30 individual physical instances).
Federation Enterprise Hybrid Cloud software resources
For information about qualified components and versions required for the initial release of the Federation Enterprise Hybrid Cloud 3.5 solution, refer to the Federation Enterprise Hybrid Cloud 3.5: Reference Architecture Guide. For up-to-date supported version information, refer to the EMC Simple Support Matrix: EMC Hybrid Cloud 3.5: elabnavigator.emc.com.
Federation Enterprise Hybrid Cloud sizing
For all Federation Enterprise Hybrid Cloud sizing operations, refer to the EMC Mainstay Sizing tool: mainstayadvisor.com/go/emc.
Chapter 9: Conclusion
This chapter presents the following topic:
Conclusion ...................................................................................................... 108
107
Chapter 9: Conclusion
The Federation Enterprise Hybrid Cloud solution provides on-demand access and control of infrastructure resources and security while enabling customers to maximize asset use. Specifically, the solution integrates all the key functionality that customers demand of a hybrid cloud and provides a framework and foundation for adding other services. This solution provides the following features and functionality:
Continuous availability
Disaster recovery
Data protection
Automation and self-service provisioning
Multitenancy and secure separation
Workload-optimized storage
Elasticity and service assurance
Monitoring
Metering and chargeback
The solution uses the best of EMC and VMware products and services to empower customers to accelerate the implementation and adoption of hybrid cloud while still enabling customer choice for the compute and networking infrastructure within the data center.
108
Chapter 10: References
This chapter presents the following topic:
Federation documentation ................................................................................. 110
109
Chapter 10: References
These documents are available on EMC.com. Access to Online Support depends on your login credentials. If you do not have access to a document, contact your Federation representative.
110
Federation Enterprise Hybrid Cloud 3.5: Reference Architecture Guide
Federation Enterprise Hybrid Cloud 3.5: Infrastructure and Operations Management Guide
Federation Enterprise Hybrid Cloud 3.5: Security Management Guide
Federation Enterprise Hybrid Cloud 3.5: Administration Guide
VCE Foundation for Federation Enterprise Hybrid Cloud Addendum 3.5
VCE Foundation Upgrade from 3.1 to 3.5 Process
