Skip to main content

VCF 9.1.x Management Domain DR: An End-to-End Guide with VCF Protection and Recovery

· VCF 9.1 · Disaster Recovery

The question this post answers

How do you fail over VCF Operations to another site without L2 stretch or NSX Federation, and keep the same IPs?

Many VCF customers run two sites with no stretched Layer 2 network, no NSX Federation and no routed IP mobility. Yet VCF Operations must come back at the recovery site with the same IP addresses and DNS names, so that users, adapters and integrations keep working after a disaster.

This post shows a proven design that does exactly that on VCF 9.1.1, using VCF Protection and Recovery (P&R) 9.1.1 with vSphere Replication. It is based on a real implementation runbook and aligned with Broadcom's VCF 9.1 documentation.

The answer in short

  1. Replicate the data, not the network. P&R replicates the VCF Operations nodes from Site A to Site B every 15 minutes over a separate, routed replication network.

  2. Pre-build an identical VLAN at Site B and keep it dormant. It has the same VLAN ID, subnet and gateway IP as Site A, but its gateway stays shut down and unadvertised.

  3. Move the route, not the IPs. During failover, the network team withdraws the subnet at Site A and activates it at Site B. The recovered nodes power on with their original IPs; nothing is re-addressed and DNS does not change.

  4. Gate every power-on behind the network. The recovery plan's first step is a network cutover prompt, so VCF Operations never runs with the subnet active at both sites.

What this post covers

PartWhat you will learn
1. Recovery modelWhich VCF 9.1 management components fail over to the other site, and which are restored in place
2. Reference architectureHow the two sites, the P&R appliances, the replication network and the dormant VLAN fit together
3. PlanningDesign decisions, RPO/RTO, prerequisites, IP budget, firewall flows and subnet states
4. ImplementationThe six build stages, from replication networking to the recovery plan and its prompts
5. Recovery workflowsNetwork cutover, test recovery, disaster failover, reprotect and planned failback
6. Day-2 operationsKeeping the DR design valid: drift, new hosts, upgrades, testing and monitoring

Environment at a glance: two VCF 9.1.1 instances (Site A protected, Site B recovery), separate management vCenters and SSO domains, vSAN ESA principal storage, a 15-minute RPO and a 4-hour RTO target.

Who it is for: VCF architects, VMware engineers, and the network and operations teams who build and run DR for VCF.

The VCF 9.1 recovery model: fleet level vs instance level

In VCF 9.1, fleet-level components can be recovered at another VCF Instance when a site is lost. Instance-level components are never failed over; each VCF Instance is restored in place from its own backups. Every DR design starts by placing each management component on one side of that line.

Broadcom covers the two paths in separate guides: VCF Component Backup and Restore, VCF Instance Backup and Restore and VCF Fleet Disaster Recovery.

Fleet components recover at the other instance; instance components restore in placeVCF 9.1 management components and their site-failure recovery pathFleet levelRecovered at the other VCF InstanceInstance levelNever failed over; restored in place from backupVCF Operations (this post)Fail over with P&R and vSphere ReplicationVCF Operations for NetworksOwn P&R protection group and recovery planManagement Services and logsRedeploy runtime at recovery site, restore SFTPVCF AutomationDeploy a new instance, restore the SFTP backupVCF License ServerRedeploy and reconfigure licensingvCenter and vSphere SupervisorFile-based restore from SFTPNSX Managers and EdgesFile-based restore; Edge config in NSX backupSDDC ManagerFile-based restore; back it up after all othersESX hosts and vSAN ESARebuild hosts; data returns through restoresP&R applianceRedeploy, re-register, re-pair, then reprotectInstance restore order: DNS, NTP, identity, certificates, SFTP → hosts and vSAN → vCenter → NSX → SDDC Manager →VCF Operations → Management Services → Automation, Ops for Networks → P&R → workload domains → Supervisor
VCF 9.1 recovery model · fleet level vs instance level

Only the left column ever crosses to the other site. Everything on the right is rebuilt where it stood, in the order shown underneath; the table below adds the backup method for each component.

ComponentLevelBackup methodComponent failureSite or instance failure
VCF Operations (analytics cluster)FleetImage-based, all nodes together, to a third-party repositoryRestore the cluster from a consistent image backupFail over with P&R (vSphere Replication) to the other instance
VCF Operations for NetworksFleetImage-based, third-party repositoryImage restoreFail over with its own P&R protection group and recovery plan
VCF Management Services (fleet and SDDC lifecycle, depot, identity, log services)FleetFile-based, SFTPRestore or redeploy the affected serviceRedeploy the services runtime at the recovery instance and restore from SFTP
Log managementFleetIncluded in the Management Services backupRestore with Management ServicesRecovered with Management Services
VCF AutomationFleetFile-based, SFTPRedeploy and restoreDeploy a new instance in the recovery VCF Instance and restore the backup
VCF License ServerFleetNoneRedeploy and reconfigure licensingRedeploy and reconfigure licensing
vCenter (incl. vSphere Supervisor)InstanceFile-based, SFTP (image-based optional)File-based restoreRestored in place with the VCF Instance
NSX Manager cluster and NSX EdgesInstanceFile-based, SFTP (Edge config inside the NSX Manager backup)Redeploy the failed node, or restore the clusterRestored in place with the VCF Instance
SDDC ManagerInstanceFile-based, SFTP; credentials database through a separate API backupFile-based restore, then inventory reconciliationRestored in place; back it up after all other components
ESX hostsInstanceNone; keep as-built documentationReinstall the host and re-add it to the clusterRebuilt during VCF Instance restore
vSAN ESA (management)InstanceNo backup; data protected by the storage policy (FTT)vSAN rebuilds from surviving copiesData returns through VM and file-based restores
P&R 9.1.1 applianceInstanceNot listed as a backup itemRedeploy, re-register and re-pairRedeploy at the restored instance, then reprotect

Two placement rules follow from this table. Keep the SFTP server and the image-backup repository outside the site they protect, or replicate them, so backups survive a site loss. And treat backup and replication as complementary: replication brings VCF Operations back fast at the other site, while backups bring it back from corruption that replication would faithfully copy.

When a whole instance must be rebuilt, Broadcom's restore order is: external dependencies (DNS, NTP, identity, certificates, SFTP reachability), management hosts and vSAN, vCenter, NSX Manager, SDDC Manager, VCF Operations, VCF Management Services, VCF Automation and VCF Operations for Networks, P&R, workload domains, then vSphere Supervisor.

What VCF Protection and Recovery 9.1 brings

VCF Protection and Recovery 9.1 is a single appliance that combines the orchestration of Site Recovery Manager with vSphere Replication. Broadcom states that standalone SRM and vSphere Replication appliances are deprecated in 9.1, and that 9.0.2 standalone appliances cannot be upgraded in place (KB 444864).

The appliance contains the protection and recovery service, vSphere Replication, and vSAN snapshots and replication. One appliance can be deployed per vCenter. A scale-out appliance (protection and recovery service only) is needed only beyond 5,000 protected VMs or for fan-in and fan-out topologies.

For the fleet components that are redeployed rather than replicated, Broadcom ships supporting scripts for the VCF 9.1 Fleet DR failover and failback tasks (KB 443506). They update fleet component registration, the Identity Broker instance association and component VIPs, and should only be used from the documentation steps that reference them.

Reference architecture: replicate the data, move the subnet

VCF Operations data replicates continuously over a routed replication network. The VCF Operations subnet moves between sites only through a gated network cutover, because the identical Site B VLAN stays dormant until then.

Data replicates continuously; the subnet moves only at cutoverVCF Operations DR with P&R 9.1.1 and an identical, dormant VLAN at Site BSite A · protectedSite B · recoveryManagement stackvCenter, NSX, SDDC Manager (local)P&R 9.1.1 appliance (pr-a)Orchestration + vSphere ReplicationVCF Operations nodesPrimary, replica, data: runningVCF Operations VLAN10.10.30.0/24 · gateway activeReplication VLAN 1650vmk3 on every management hostManagement stackvCenter, NSX, SDDC Manager (local)P&R 9.1.1 appliance (pr-b)Orchestration + vSphere ReplicationPlaceholder VMsReplicas on Site B vSAN ESAIdentical VCF Operations VLAN10.10.30.0/24 · gateway dormantReplication VLAN 2650vmk3 on every management hostIsolated test port groupNo route out · test recovery onlysite pairreplicasRPO 15 minone siteactive at a timerouted,always onAccent: running VCF Operations nodesDashed: dormant until network cutoverEnhanced VR data: TCP 32032 host to host
Reference architecture · two VCF 9.1.1 instances, P&R per site, identical dormant VLAN

Read the gap between the sites row by row: the P&R appliances are paired, replicas flow one way, the replication VLANs are routed all the time, and the VCF Operations VLAN is active at only one site.

Two networks behave very differently in this design. The replication VLANs are routed between sites all the time. The VCF Operations VLAN exists at both sites but is routed from only one of them at any moment.

ComponentSite A (protected)Site B (recovery)Rule
Management vCenter, NSX, SDDC ManagerExistingExistingNever replicated
P&R 9.1.1 appliance (with vSphere Replication)11Paired
VCF Operations nodesRunning on the Ops port groupReplicas on vSAN, placeholder VMsSame IPs after failover
VCF Operations VLAN and gatewayActive and advertisedIdentical, dormantExactly one site active
Replication VLAN and VMkernel per hostVLAN 1650, routedVLAN 2650, routedAlways on
Test networkNoneIsolated test port groupTests only
Cloud ProxiesLocal, not replicatedLocal, not replicatedHandled by recovery plan prompts

Scope of the worked example. In scope: the VCF Operations analytics cluster (primary, replica and data nodes), Cloud Proxies handled as operator prompts, one P&R appliance per site, the network cutover of the VCF Operations VLAN, and the full test, failover, reprotect and failback cycle. Out of scope: VCF Operations for Networks, VCF Automation and VCF Management Services; vCenter, NSX Managers and SDDC Manager, which each site keeps and never replicates; array-based replication; and L2 extension, NSX Federation or re-addressing.

Key risk: duplicate subnets. If the Site A and Site B VCF Operations subnets are ever active and routed at the same time, traffic is black-holed or split. Every failover, test and failback step in this design includes an explicit network gate to prevent it.

Planning the DR design

Planning settles seven design decisions, the replication settings, the prerequisites and the network model before anything is deployed. Every prerequisite must be confirmed and evidenced; any item that fails is a blocker.

Design decisions

vSphere Replication protects each VCF Operations node with a 15-minute RPO onto Site B vSAN ESA, and recovery keeps the original IPs by activating the identical Site B VLAN.

#DecisionValueRationale
D1Replication methodvSphere Replication through the P&R 9.1.1 applianceDocumented VCF Fleet DR method; no array dependency
D2Replication scopeVCF Operations analytics nodes onlyManagement components stay local to each site
D3IP handlingSame IPs; no IP customization in the recovery planIdentical VLAN and subnet pre-built at Site B
D4Network failoverManual, gated cutover by the network teamNo L2 stretch, NSX Federation or routed IP mobility
D5Replication networkDedicated, routed replication VLAN per site, one VMkernel per management hostReplication isolated from management and vSAN traffic
D6Test recovery networkIsolated, non-routed port group at Site BTest VMs use production IPs and must not reach production
D7Target RTO4 hours or less, including network cutoverDesign objective; must be proven in testing

Replication settings per VCF Operations node

SettingValueNote
RPO15 minutesRange is 1 minute to 24 hours; below 5 minutes needs an Advanced Cyber Compliance subscription
Point-in-time instances3 copies over 24 hoursAdds vSAN capacity at Site B
Guest OS quiescingDisabledCrash-consistent, per the VCF Fleet DR guidance
Network compressionEnabledUses more host CPU at both ends
Network encryptionEnabledRequired for Enhanced vSphere Replication from vSphere Replication 9.0.3
Target datastoreSite B management vSAN ESA datastore
Target VM storage policyvSAN ESA RAID-5 (FTT=1) or the cluster default ESA policyOn 4 to 5 hosts, RAID-5 2+1 is about 1.5× raw
Disk formatSame as sourceThin on vSAN

Prerequisites checklist

Platform

  • ☐ Both sites run VCF 9.1.1 with healthy management domains on vSAN ESA, and two separate management vCenters in separate SSO domains (P&R cannot replicate within one vCenter)

  • ☐ Site B management vSAN has capacity for the replicas × policy overhead plus point-in-time growth, and stays below about 65% raw utilisation on a 4-host cluster

  • ☐ The P&R 9.1.1 binaries and entitlement are available for both sites, and the vCenter versions are supported in the Broadcom Interoperability Matrix

  • ☐ A VCF Operations backup exists and has been tested independently of replication

P&R appliance (per site)

ItemRequirement
ApplianceOne P&R 9.1.1 appliance per management vCenter; scale-out appliance not used
vCPU and memory8 vCPU, 24 GB RAM
DiskAbout 826 GB thick provisioned (per the OVA)
NetworkOne static IP on the VM management network, with forward and reverse DNS
TimeNTP on vCenter, ESX hosts and P&R (KB 1318)
NamesFQDNs everywhere, never short hostnames, to avoid certificate thumbprint and DNS failures

VCF Operations network at Site B (identical, dormant VLAN)

  • ☐ Same VLAN ID, subnet, mask and gateway IP as Site A

  • ☐ A Site B distributed port group on that VLAN, with the same teaming and MTU as Site A

  • ☐ The Site B gateway SVI is shut down or not advertised in normal operation

  • ☐ The network team has a written, tested cutover and cutback procedure and a named on-call owner

  • ☐ Firewall, NAT and load-balancer rules for VCF Operations exist at Site B, or are scripted for cutover

  • ☐ DNS records stay unchanged (the IPs do not change), and DNS and NTP servers from both sites are configured on every node

Replication network

  • ☐ A replication VLAN and subnet at each site, routed between the sites. The "no L3 for failover" constraint applies only to the VCF Operations subnet; vSphere Replication cannot work without routed connectivity between replication VMkernels

  • ☐ MTU validated end to end: jumbo frames where supported, or a consistent 1500

  • ☐ Bandwidth for the initial full sync of all nodes and 15-minute deltas

Test isolation and operations

  • ☐ An isolated Site B test port group with no route to production, or the P&R auto-created isolated network

  • ☐ Change approval, an incident commander role and network on-call defined for DR invocation

IP budget

With 4 management hosts per site, the build needs 10 new IPs and 2 new routed VLANs. VCF Operations needs no new IPs, because Site B reuses the identical VLAN. The formula is 2 + H(A) + H(B), where H is the number of management hosts per site.

ItemSite ASite BTotal
P&R appliance IP (management network)112
Replication VMkernel IPs (1 per management host)448
New IPs5510
New replication VLAN and port group (L3 routable)112
VCF Operations IPsExistingSame IPs0 new

Addressing worksheet (example values)

ObjectSite ASite B
P&R appliancepr-a.sitea.corp.local, 10.10.10.30/24pr-b.siteb.corp.local, 10.20.10.30/24
Replication VLAN, subnet, gateway1650, 10.10.50.0/28, 10.10.50.12650, 10.20.50.0/28, 10.20.50.1
Replication port groupdpg-mgmt-a-vr-1650dpg-mgmt-b-vr-2650
Replication VMkernel IPs10.10.50.11 to .1410.20.50.11 to .14
VCF Operations VLAN, subnet, gateway10.10.30.0/24, 10.10.30.1Same 10.10.30.0/24, 10.10.30.1
VCF Operations node IPs10.10.30.11 (primary), .12 (replica), .13 (data)Same

Firewall flows

#SourceDestinationPurpose
F1P&R A and P&R BLocal vCenterRegistration and operations
F2P&R A ↔ P&R BEach otherSite pairing
F3P&R A ↔ vCenter B, P&R B ↔ vCenter APeer vCenterCross-site operations
F4Replication subnet A ↔ replication subnet BBoth directionsvSphere Replication data
F5Replication subnetsRemote P&R applianceReplication control
F6Admin jump hostP&R A and B on 443 and 5480UI and appliance management
F7VCF Operations subnet (Site B when active)vCenters, NSX, DNS, NTP, LDAP, SMTP, adapter endpointsSame rules as Site A, staged at Site B
F8ESX hosts running protected VMsESX hosts of the target cluster, TCP 32032Enhanced vSphere Replication data

Take the exact ports for Protection and Recovery 9.1 from ports.broadcom.com.

Subnet states

The VLAN can exist on the Site B switches and port group at all times; only the routing (SVI or route advertisement) moves between sites.

The VCF Operations subnet is routed from exactly one site in every stateOnly the routing moves; the VLAN exists at both sites. Never both gateways up at once.Test recoverySite A gateway: activeSite B: dormant, VMs isolatedNormalSite A gateway: activeSite B gateway: dormantFailed overSite A: down or withdrawnSite B: active, advertisedFailback (planned)Site B: deactivatedSite A: re-activatedTest, then CleanupDR or planned migrationnetwork cutover A → Breprotect B → A,planned migrationcutback B → A,reprotect A → B
VCF Operations subnet states · normal, test, failed over, failback

A test never changes routing. Only a declared DR or an approved planned migration moves the gateway, and the way back always runs through reprotect and a planned migration with a cutback.

Implementation: from replication network to recovery plan

The build runs in six stages, each repeated at both sites where it applies: replication networking, P&R deployment, registration, site pairing and mappings, replication, then the protection group and recovery plan.

Six build stages, each validated before the next startsStages 1 to 3 repeat at both sites; pairing, replication and the plan are created from Site A1 · Replication networkPort group + vmk3 per hostvmkping in both directions2 · Deploy P&R 9.1.1One appliance per siteCA certificate, service account3 · Register P&RLocal vCenter, port 443Default extension ID4 · Pair, license, mapDifferent SSO domainsReverse and test mappings5 · Replicate VMsRPO 15 min, 3 PIT copiesQuiesce off, encryption on6 · Group and planPriority groups 1, 2, 3Seven operator prompts
Implementation sequence · 6 stages

Each stage has a check that gates the next: vmkping before deployment, Running services before pairing, Connected sites before mapping, a completed full sync before the protection group.

Stage 1: Replication networking (both sites)

Each management host gets one replication VMkernel on a dedicated port group, so vSphere Replication traffic leaves on its own routed VLAN rather than on management (vmk0) or vSAN.

  1. In the management vCenter, create a distributed port group on the management VDS: dpg-mgmt-a-vr-1650 (Site B dpg-mgmt-b-vr-2650), static binding, elastic allocation, 8 ports, VLAN 1650 (Site B 2650), load balancing "Route based on physical NIC load".

  2. On every management host, add a VMkernel adapter on that port group with the vSphere Replication and vSphere Replication NFC services. Every host sends during failover and receives during failback.

  3. Set the agreed MTU (9000 only if jumbo frames are validated end to end), a static IP from the worksheet, and Override default gateway for this adapter with the replication gateway.

  4. On vSAN hosts the adapter is normally vmk3 (vmk0 management, vmk1 vMotion, vmk2 vSAN).

  5. Validate in both directions from a host shell, using -s 1472 for a 1500 MTU:

vmkping -I vmk3 -d -s 8972 10.20.50.11
vmkping -I vmk3 10.20.10.30

Operational rule: any host added to a management cluster later must get a replication VMkernel before it can run or receive protected VMs.

Stage 2: Deploy the P&R 9.1.1 appliance (per site)

Download the appliance from the Broadcom Support Portal under My Downloads > Protection and Recovery > VMware Protection and Recovery > 9.1.1, and verify the checksum. The scale-out appliance is not needed.

  1. In the local management vCenter, right-click the management cluster and select Deploy OVF Template.

  2. Name the VM (pr-a at Site A, pr-b at Site B), select the management cluster, the management vSAN ESA datastore and default policy, and map Network 1 to the VM management port group.

  3. On Customize template: enable SSHD only if policy allows (disable after the build); hostname = the lowercase FQDN with matching A and PTR records; root password for the OS and admin password for the Appliance Management Interface; NTP and DNS servers from both sites; IPv4 static addressing from the worksheet.

  4. Power on, then confirm https://<pr-fqdn>:5480 opens and forward and reverse DNS resolve.

  5. In the appliance management interface, use Certificate > Generate CSR (FQDN and IP as SANs), have the enterprise CA sign it, and upload it with Certificate > Change.

  6. Create a service account in each vCenter (for example svc-pr-a@vsphere.local) in the Administrators group or a custom P&R role, and store it in the vault.

Stage 3: Register P&R with the local vCenter

Each appliance registers with the management vCenter at its own site.

  1. Open https://<pr-fqdn>:5480, log in as admin, and select Summary > Configure appliance.

  2. Enter the vCenter FQDN in lowercase, port 443 and the service account, then verify the certificate thumbprint.

  3. On Name and Extension: a unique site name (Site-A, Site-B), the DR team mailbox, the appliance FQDN as local host, and keep the default extension ID com.vmware.vcDr. Both appliances in a pair must use the same extension ID; a custom ID is only for shared recovery site topologies.

  4. Confirm the services show Running and the P&R plug-in appears in the vSphere Client after logging out and back in.

Stage 4: Pair sites, license and map inventory

The network mapping is the core of this design: the Site A Ops port group maps to the identical-VLAN Site B port group for real recoveries, and to the isolated test port group for tests.

  1. In P&R at Site A, select New Site Pair and choose Pair with a peer vCenter located in a different Single Sign-On domain. Enter the Site B vCenter address exactly as it was used when P&R B was registered.

  2. Confirm both sites show Connected, then apply and verify the P&R licence at both sites.

  3. Create mappings with Prepare mappings manually. Automatic mapping matches objects by name and is available only for networks and folders; resource mappings are always manual.

  4. Select Prepare reverse mappings for each mapping; reprotect and failback need them.

  5. On Test networks, select the isolated test port group (or accept the auto-created isolated network). Never use the production Site B port group for tests, because test VMs keep production IPs.

  6. Leave IP customization unconfigured. No IP subnet mapping rules are needed because the subnet is identical at both sites.

MappingSite ASite BReverse
Network (recovery)Ops port group AOps port group B (identical VLAN, same subnet)Yes
Network (test)Isolated test port group (no route)
FolderVCF Ops folder AVCF Ops folder BYes
ResourceManagement cluster AManagement cluster BYes
Storage policy (if used)Management vSAN ESA policy AManagement vSAN ESA policy BYes

Stage 5: Configure replication

The RPO sets the maximum tolerable data loss: a smaller RPO loses less data but uses more bandwidth. Before starting, confirm network encryption is enabled for Enhanced replication and TCP 32032 is open from the source ESX hosts to the target cluster hosts.

  1. In P&R, open Replications > Outgoing > Create new replication and accept automatic vSphere Replication server assignment.

  2. Select the VCF Operations primary, replica and data nodes only.

  3. Target the Site B management vSAN ESA datastore with the agreed policy and Same as source disk format. Leave Select seeds cleared unless pre-seeded copies exist (wrong seeds can cause data loss), and keep Auto-include new disks selected.

  4. Run all Replication mapping tests and confirm they pass.

  5. Apply the settings from the planning table: RPO 15 minutes, 3 point-in-time instances per day for 1 day, quiescing off, compression and encryption on.

  6. Choose Do not add to protection group now and Do not add to recovery plan now, finish, and watch the Initial Full Sync until every VM shows OK. Record how long it took.

Stage 6: Protection group and recovery plan

One replication set, one protection group and one recovery plan cover all VCF Operations nodes. VCF Operations for Networks uses its own protection group outside this scope.

  1. Create protection group VCF-Operations-PG, direction Site A → Site B, type Individual VMs (vSphere Replication), with all VCF Operations nodes. Confirm every VM shows Protected (OK) and placeholder VMs exist at Site B.

  2. Create recovery plan VCF-Operations-RP with that protection group and keep the site-level test network mapping.

  3. Set startup priorities: primary analytics node in priority group 1, replica node in 2, data nodes in 3. For each VM, confirm IP customization is off.

  4. Add the operator prompts as user steps of type Prompt, shown in the recovery sequence below. The VCF 9.1 procedure for customizing the VCF Operations recovery plan expects the load balancer, Cloud Proxy and node-status prompts; the network cutover gate is this design's own addition.

Recovery workflows: test, failover, reprotect and failback

One recovery plan serves four workflows. What changes between them is where the plan runs, whether the network cuts over, and what node state the operator expects at each prompt.

The recovery plan sequence and the network gate

The network cutover gate is the first prompt, before any priority 1 VM powers on. It is never dismissed until the network team confirms in writing; powering on VCF Operations with the subnet active at both sites causes a duplicate-IP outage. Confirm the real prompt order during the first test.

The network gate comes before any VM powers onVCF-Operations-RP: seven operator prompts around three priority groupsStart the planTest, DR or planned1 · Network gateWritten confirmation2 · Load balancerVIP up at target3 · Cloud ProxiesShut down per mode4 · Check primaryState in /adminPower on P1Primary node5 · Check replicaSame as primaryPower on P2Replica node6 · Check dataData node statePower on P3Data nodes7 · Bring onlineProxies on, all OnlineRecovery completeRecord end time, RTONumbered boxes are prompts the operator dismisses; shaded boxes are P&R power-on steps.Node targets differ by mode: DR expects Running / Waiting for Analytics, planned migration expects Offline.
Recovery plan sequence · 7 prompts, 3 priority groups

The plan pauses at every numbered box until an operator dismisses it, so each node is checked before the next priority group starts.

OrderPositionPromptContent
1First step, before power-on of priority 1Network cutover gateReal recovery: Site A subnet withdrawn or down; Site B SVI up and advertised; Site B firewall, NAT and LB rules active; change number recorded. Test: VMs on the isolated test port group and Site B still dormant
2Before priority 1Verify load balancerVCF Operations LB or VIP available at Site B
3Before priority 1Shut down Cloud ProxiesDR: shut down reachable Cloud Proxies at Site B. Planned migration: at both sites
4Before priority 1Check primary nodeIn /admin. Planned: Not Running / Offline. DR: Running / Waiting for Analytics
5Before priority 2Check replica nodeSame as the primary check
6Before priority 3Check data nodesPlanned: Not Running / Offline. DR: Running / Online
7After the last power-onPower on Cloud Proxies, bring cluster onlineSelect Bring Cluster Online if needed and wait for all nodes Online

Network cutover procedure (network team)

The VCF Operations subnet must be active at exactly one site. P&R waits at the gate prompt while the network team works.

  1. Confirm the trigger: DR declared, or planned migration approved.

  2. Deactivate at Site A: shut the Ops VLAN SVI or withdraw its route, and confirm it is no longer advertised. If Site A is down, confirm it is unreachable instead.

  3. Activate at Site B: bring up the SVI with the same gateway IP, advertise the subnet into the core and WAN, and activate the staged firewall, NAT and LB rules.

  4. Verify: traceroute to the gateway reaches Site B; only one route to the subnet exists; DNS, NTP, LDAP and the vCenter B endpoints are reachable.

  5. Record the change number and time. Only then does the P&R operator dismiss the gate.

Cutback for failback runs the same steps in reverse. Rollback: if Site B activation fails and Site A is still healthy, do not dismiss the gate; re-activate Site A, cancel the recovery plan run if no VMs have powered on, and escalate.

Never have both gateways up and advertised at the same time. If Site A comes back on its own after a DR, for example after a power restore, keep the Site A SVI down until the planned failback.

Workflow 1: Non-disruptive test recovery

A test powers on copies of the VCF Operations nodes at Site B on the isolated test port group, with their production IPs, while production keeps running at Site A. No network cutover happens during a test.

  • Before: protection group OK, recovery plan Ready, replication OK for every node; the network team confirms the test port group has no gateway, uplink route or path to production; a test jump host is on the test port group; a change record is open.

  • Run: select Test with Replicate recent changes to recovery site. At the gate, confirm isolation. At the Cloud Proxy prompt, do not shut down production Cloud Proxies; dismiss with a note. Check node state from the jump host at each stage.

  • Measure: record elapsed time to Test complete as the test RTO. Estimate network cutover time separately with the network team.

  • Clean up: run Cleanup; the test VMs are powered off and unregistered and the test images discarded. The plan returns to Ready.

  • Pass criteria: all nodes Online inside the bubble; prompt order as designed; no duplicate-IP alarms or production impact; test RTO plus network cutover estimate within 4 hours.

Workflow 2: Unplanned disaster failover

Run only when Site A, or VCF Operations at Site A, is lost and DR is declared. Everything runs from Site B.

  1. Activation gate: the incident commander declares DR and records the time; the change owner gets emergency approval; network on-call is engaged; Site B vCenter, vSAN health, DNS, NTP and the load balancer are confirmed available.

  2. In the Site B vCenter, open Protection and Recovery (Site A shows as unreachable), select VCF-Operations-RP > Run > Disaster recovery, and acknowledge the permanent changes.

  3. P&R skips the protected-site steps (errors there are expected) and recovers the VMs from the latest replica on Site B vSAN.

  4. At the gate, the network team runs the cutover. Dismiss only after their written confirmation.

  5. Work through the LB, Cloud Proxy and node prompts using the DR targets, then power on Cloud Proxies and wait for all nodes Online.

  6. Validate: cluster healthy; users log in through the usual FQDN or VIP (DNS unchanged, now served from Site B); adapters and Cloud Proxies collecting; a single active route to the subnet; the DR event record complete.

Workflow 3 and 4: Reprotect and planned failback

Failback has three stages and needs both sites up: reprotect (replicate B → A), a planned migration with a network cutback, then reprotect again to restore A → B.

  • Prerequisites: Site A vCenter, hosts, vSAN, replication VMkernels and P&R A healthy, site pair Connected; Site A gateway still down; last recovery run successful; Site A vSAN capacity for the reverse replicas; cutback window approved.

  • Reprotect B → A: select Reprotect on the plan. The first sync can take as long as the initial full sync. If it stops part-way, fix the cause and run Reprotect again. Recommended: run a test towards Site A on a Site A isolated port group.

  • Planned migration A ← B: select Run > Planned migration. P&R shuts down VCF Operations at Site B, runs a final sync and recovers at Site A. The network team runs the cutback at the gate. Shut down Cloud Proxies at both sites; the node prompts expect Not Running / Offline, the normal in-between state. Then bring the cluster online and validate.

  • Restore protection A → B: select Reprotect again, confirm direction Site A → Site B and plan Ready, confirm the Site B gateway is dormant, then run a test and cleanup before closing the change.

Day-2 operations: keeping the DR design true

The two ongoing risks are the dormant Site B VLAN drifting from Site A's configuration, and new hosts or nodes missing from replication. A DR design that is not tested and maintained decays quietly.

EventActionOwner
Any change to the Site A VCF Operations VLAN, subnet, firewall, NAT or LBApply the identical change to the dormant Site B config in the same change recordNetwork
VCF Operations upgrade or node add (Fleet LCM)Confirm every node is replicated (new nodes need replication and the protection group), then run a testVCF admin
New management ESX host at either siteAdd the replication VMkernel before the host runs protected VMsVCF admin
P&R 9.1.x patch or upgradeUpgrade both sites in the documented order, then confirm replication is OKVCF admin
Site B vSAN capacity above 70%Expand capacity or reduce point-in-time instancesVCF admin
QuarterlyTest recovery plus a tabletop of the network cutoverDR owner and Network
Every 6 to 12 monthsFull planned migration and failback, including the real network cutoverDR owner and Network

Monitoring: alert on replication status other than OK and on RPO violations; alert on recovery plan status other than Ready; and have the network team monitor that the VCF Operations subnet is advertised from exactly one site.

Implementation sign-off

  • ☐ Replication port groups and VMkernels on every management host; vmkping passes in both directions

  • ☐ P&R appliances deployed with CA-signed certificates, registered, paired and licensed

  • ☐ Identical VLAN, subnet and gateway built at Site B and kept dormant; firewall, NAT and LB rules staged

  • ☐ Network, test network, folder, resource and storage-policy mappings in both directions

  • ☐ All nodes replicating with RPO 15 minutes, 3 point-in-time copies, quiescing off, compression and encryption on

  • ☐ Protection group OK, recovery plan Ready, startup priorities set, all seven prompts configured

  • ☐ Network cutover procedure signed off by the network team and tested in a maintenance window

  • ☐ Test recovery passed, cleanup completed, RTO recorded

Record every DR event

Keep one record per test, failover, reprotect or migration: event type, change or incident number, who declared and approved, start and end time (UTC), network cutover change and time, who verified the single route, prompt outcomes and node states, measured RTO including cutover, recovery point used (latest or point in time), issues and actions, and sign-off. Over time these records are your proof that the 4-hour RTO is real.

References

Comments

Popular posts from this blog

Changing the FQDN of the vCenter appliance (VCSA)

This article states how to change the system name or the FQDN of the vCenter appliance 6.x You may not find any way to change the FQDN from the vCenter GUI either from VAMI page of from webclient as the option to change the hostname always be greyed out. Now the option left is from the command line of VCSA appliance. Below steps will make it possible to change the FQDN of the VCSA from the command line. Access the VCSA from console or from Putty session. Login with root permission Use above command in the command prompt of VCSA : /opt/vmware/share/vami/vami_config_net Opt for option 3 (Hostname) Change the hostname to new name Reboot the VCSA appliance.   After reboot you will be successfully manage to change the FQDN of the VCSA . Note: Above step is unsupported by VMware and may impact your SSL certificate and face problem while logging to vSphere Web Client. If you are using self-signed certificate, you can regenerate the certificate with...

VCF 9 Networking : Centralized vs Distributed Connectivity

  VCF 9 Networking : Centralized vs Distributed Connectivity With the release of VMware Cloud Foundation 9 (VCF 9.0) , VMware has redefined the private cloud networking model by introducing both centralized and distributed connectivity. This dual approach provides flexibility for organizations to choose between traditional edge-based routing and modern, host-level distributed networking. Why New Networking in VCF 9? Prior to VCF 9, NSX networking was largely edge-centric—requiring dedicated Edge clusters to handle north-south traffic. This created scaling and operational overheads. VCF 9 introduces a cloud-like networking abstraction with: Native VPCs (Virtual Private Clouds) for tenant isolation. Transit Gateways (TGW) to interconnect VPCs and external networks. Simplified bootstrapping – NSX VIBs are embedded in ESXi for easier enablement. Improved lifecycle and visibility via integrated VCF Operations. These enhancements align with VMware’s goal of offer...

High-Speed Networking for VMs: VMXNET3 Now Supports Up to 65 Gbps

The VMXNET3 adapter is VMware's high-performance virtual network adapter, designed for optimized performance in vSphere environments. It is the default and recommended adapter for most modern workloads on ESXi , delivering high throughput, low latency, and support for advanced networking features. With the release of vSphere ESXi 8.0 Update 2 , VMXNET3 introduces new capabilities that significantly enhance virtual network performance. What’s New in ESXi 8.0 Update 2 for VMXNET3? Traditionally, VMXNET3 reports a default link speed of 10 Gbps to the guest operating system. However, the actual achievable throughput often exceeds this value—thanks to technologies such as multi-queue support , Receive Side Scaling (RSS) , and Large Packet Offload . Starting with ESXi 8.0 Update 2 , this limitation can now be overridden , and VMXNET3 can be configured to support custom link speeds of up to 65 Gbps . This enhancement enables improved flexibility and throughput for high-performance w...