-->

Monday, May 25, 2020

Exchange - Adding A New Server To An Existing DAG

In my environment, we run Veeam to back up Exchange...this wasn't my decision (since we have a DAG and don't need backups) but I was overridden by management (who don't understand Exchange). 

The problems with Veeam are: 
- It uses VM snapshots, which we all know are not supported with Exchange, and it causes database failovers if it takes too long to commit the snap, which happens very, very often...very often, like, daily.
- If you use Network Backup Mode (in order to not cause failovers) the backup takes a long time...upwards of 70 hours in our shop.
- If you use hotadd mode (which is faster) it causes the aforementioned failovers.
In order to alleviate these issues, Veeam recommends a passive-only DAG node that the backup targets, and running hotadd mode on that passive node.

So, we're going to add a new Mailbox Server to our DAG, and since a web search turns up virtually no instructions on doing that, here ya go!

Install Exchange 2016 Pre-Reqs

On the server that is to be your new Exchange 2016 Mailbox Server, run the following (copy/paste the whole block) in an elevated Windows PowerShell:

Install-WindowsFeature AS-HTTP-Activation, Desktop-Experience, NET-Framework-45-Features, RPC-over-HTTP-proxy, RSAT-Clustering, RSAT-Clustering-CmdInterface, RSAT-Clustering-Mgmt, RSAT-Clustering-PowerShell, Web-Mgmt-Console, WAS-Process-Model, Web-Asp-Net45, Web-Basic-Auth, Web-Client-Auth, Web-Digest-Auth, Web-Dir-Browsing, Web-Dyn-Compression, Web-Http-Errors, Web-Http-Logging, Web-Http-Redirect, Web-Http-Tracing, Web-ISAPI-Ext, Web-ISAPI-Filter, Web-Lgcy-Mgmt-Console, Web-Metabase, Web-Mgmt-Console, Web-Mgmt-Service, Web-Net-Ext45, Web-Request-Monitor, Web-Server, Web-Stat-Compression, Web-Static-Content, Web-Windows-Auth, Web-WMI, Windows-Identity-Foundation

**Note** The Windows Server version needs to match your other DAG nodes, in my environment it’s Server 2012 R2

Install .Net Framework to match what’s installed on your other DAG nodes. At the time of this writing (and in my environment), CU16 is the current build, which requires .Net 4.8.

You can get it here:

https://support.microsoft.com/en-us/help/4503548/microsoft-net-framework-4-8-offline-installer-for-windows

Install MS Unified Communications API 4.0 Core Runtime from the following link:

https://www.microsoft.com/en-us/download/details.aspx?id=34992

Install Visual C++ 2013 from here:

https://www.microsoft.com/en-us/download/details.aspx?id=40784


Exchange Server Install

Uninstall any A/V client if currently installed.

Run Windows Update to patch the server up to the latest

Create your database storage volume; I use drive letter E: for Databases.

Label drive E: “Volume1”.

Mount the Exchange 2016 CU .ISO.

Right-click setup.exe and Run-as admin


1. Install Mailbox role
2. Leave malware scanning enabled
3. Wait a looooong time
4. Click OK to close Installer
5. Reboot the machine to complete Exchange setup
6. After the reboot, check that all Exchange Services are running
7. Install license key, by running the below cmdlet:
Set-ExchangeServer -Identity "server name" -ProductKey XXXX-XXXX-XXXX-XXXX
8. Restart Microsoft Exchange Information Store for the key to take effect.

Set the Exchange URLs

To avoid any Autodiscover/OWA issues, set the Virtual Directory URLs immediately.

**Note** If for some reason OWA and the EAC are broken after setting up the new server (usually this means DNS/Load Balancing isn’t configured properly), put the new server into maintenance mode, then set your URLs – that should fix it.

Use Paul Cunningham’s (ExchangeServerPro) awesome URL scripts to automatically set your namespaces in one shot.

Grab it here: https://practical365.com/exchange-server/powershell-script-configure-exchange-urls/

**Note** We’ll need the URLs to match those on the other DAG nodes.

Add New Server to Load Balancer

Next, we’ll add the new server to our Load Balancer; again, to avoid any service interrupts.

The following instructions are for Kemp Loadmaster, but it should be similar for other appliances as well.

We’ll be modifying the SubVSs for both our Internal and External Virtual Services.

Navigate to Virtual Services > View/Modify Services > Expand SubVSv > Click Modify

Expand Real Servers > Click Add New > Input the IP of the new server > Click Add this Real Server > Click OK

The new server will be listed along with your current ones.

Hit Back and repeat for all 9 SubVSs on both Internal and External Services.

To verify the new server was added and in service, navigate to Real Servers and it will be in the list with your other DAG nodes.

Delete Default Database

Next, we’ll delete the default database on the new server.

Run the following cmdlets in the EMS to delete the default Database:

Remove-mailbox -database "default database name"

**Note** You may receive a warning that health mailboxes were present. You can disregard the warning, as the database will still be deleted and the health mailbox isn’t needed, since our other production databases already have them.

Create DAG Mount Points

**Note** The following section is for reference only - your environment may differ. The directory structures are how I personally set up Exchange Mailbox Servers, which, if you set up Exchange following my "Installing Exchange in a Resource Forest" post, then feel free to create them this way.

Perform the following steps on the new Exchange server – it is very important that volumes and folders match your other DAG servers, exactly.

You should already have your E: Volume presented to your server as a drive.

1. On the C: drive, create a folder called EXVols – this folder will be used to mount our E: (Volume1).
2. Next, on the C: drive, create a folder called ExDBs – this folder will hold the Database mount points.

Creating the Volumes

3. Within the ExVols folder, create a new folder called Volume1.
4. Open Windows Disk Management to mount our volume to our ExVols folders.
5. Right-click E: and select Change Drive Letter and Paths…
6. Click Add and browse to the location of the Volume1 folder – C:\ExVols\Volume1
7. Click OK, twice

You should see the folder with a Disk icon meaning its now a Mount Point. 

Creating the Database Folders

Under the C:\ExDBs folder, create the new Database folders to match the DB’s you have on your other DAG nodes. 

In my case, we have 6, so we’ll create the following folders:

C:\ExDBs\DB01 
C:\ExDBs\DB02
C:\ExDBs\DB03
C:\ExDBs\DB04
C:\ExDBs\DB05
C:\ExDBs\DB06


After you have your folders set, open an Elevated command prompt, and run:

mountvol

This will list the available volumes for use.

In our case we know the one we want is \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\ because we can see the folder Volume1 is mounted to it

Run the following command to mount DB01:

Mountvol DB01 \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\

If you go to your C:\ExDBs, you’ll notice the folder icon for DB01 has changed to a mount point icon.

Now mount your other DB folders:

Mountvol DB02 \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\

Mountvol DB03 \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\

Mountvol DB04 \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\

Mountvol DB05 \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\

Mountvol DB06 \\?\Volume{03cf7f78-ed05-4bb7-a4f0-0914f9575bdd}\

If you run mountvol again, you’ll see all DB’s mounted under the Volume1 folder.

Create Database Directory Structure

Next, we’ll create database directory structure to match those on the other DAG nodes. Each folder will have 2 folders beneath it: one folder for the Database .edb file and one for the Logs.

You can create these folders directly from Volume1 (E:) or by going to C:\ExDBs\DB01 through DB06 (they will have the same folders).

In E:\ExDBs\DB01, create a new folder named DB01.db and new folder called DB01.log.

In E:\ExDBs\DB02, create a new folder named DB02.db and new folder called DB02.log.

In E:\ExDBs\DB03, create a new folder named DB03.db and new folder called DB03.log.

In E:\ExDBs\DB04, create a new folder named DB04.db and new folder called DB04.log.

In E:\ExDBs\DB05, create a new folder named DB05.db and new folder called DB05.log.

In E:\ExDBs\DB06, create a new folder named DB06.db and new folder called DB06.log.

Add New Mailbox Server to the DAG

Run the following in the EMS:

Add-DatabaseAvailabilityGroupServer -identity DAG01 –MailboxServer "EXCH-MBX-04"

**Note** change the –MailboxServer "EXCH-MBX-04" to the name of your new server

Import/Create New Certs

Next, we’ll Export your SAN certificate from one of the other DAG nodes.

In the EAC, navigate to Servers > Certificates

Select your SAN cert, and click the ellipses, then click Export.

Enter the UNC path where you want to store the .pfx and give it a password.

Click the ellipses again and click Import

Enter the UNC path where you exported the cert to, and the password you set.

Hit the Plus Sign and select your new Mailbox Server, and hit Finish.

Do the above process again if you have any Federation Certificates

If you run Unified Messaging, you most likely have a cert provided by your internal CA; you’ll need to request a new cert, because we’ll need to add the FQDN on the new server to that cert.

Follow my previous post https://www.exchangeitup.net/2017/11/exchangesfb-um-cross-forest-with_4.html to request the new cert.

**Note** If your certs status is "invalid" or “revocation check failed” after importing, it’s likely that the server can’t get to the internet because of a proxy; make sure to set web proxy settings on the new server by following my post here:

https://www.exchangeitup.net/2016/08/exchange-20132016-using-exchange.html


Re-Subscribe Edge Server

Next, we’ll need to re-subscribe our Edge Server so the new Mailbox server can participate in Mail Flow.

On the Edge server, in the EMS, run:

New-EdgeSubscription -FileName C:\Temp\EdgeSubscription.xml

Copy the EdgeSubscription.xml file to C:\Temp on any Mailbox server.

On the mailbox server you copied the .xml file to, in the EMS, run:

New-EdgeSubscription -FileData ([byte[]]$(Get-Content -Path "C:\Temp\ EdgeSubscription.xml" -Encoding Byte -ReadCount 0)) -Site "Default-First-Site-Name"

**Note** Change -Site "Default-First-Site-Name" to the name of your Exchange Organization Name. If you run a Resource Forest, you’ll need the full site UNC like so: "resourcedomain.com/Configuration/Sites/Default-First-Site-Name"

Add Database Copies

We will use Postpone Seeding to allow the copy creation to finish before seeding.

The copy creation will automatically set the Activation Preference to the next in-line preference according to what’s already set for your other DAG nodes.

Run the following cmdlets in the EMS to create the DB copies:

**Note** Run each cmdlet separate on each line.

Add-MailboxDatabaseCopy -Identity DB01 -MailboxServer EXCH-MBX-04 –SeedingPostponed

Add-MailboxDatabaseCopy -Identity DB02 -MailboxServer EXCH-MBX-04 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity DB03 -MailboxServer EXCH-MBX-04 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity DB04 -MailboxServer EXCH-MBX-04 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity DB05 -MailboxServer EXCH-MBX-04 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity DB06 -MailboxServer EXCH-MBX-04 –SeedingPostponed

Export Custom Tasks

 

If you run any custom tasks, like cleaning logs, which you can find in my previous post here https://www.exchangeitup.net/2016/11/exchange-20132016-cleaning-up-old.html, you’ll want to export the task from one of your other DAG nodes to the new server.

 

On another DAG node, In Task Scheduler, right-click the task > Export. Save the .xml file on a file share.

On your new server, in Task Scheduler, right-click Task Scheduler Library > Import Task and choose the .xml you exported.


Compress Logging Volume

 

I compress the Diagnostic Log Directory because Exchange doesn’t manage them very well, which leads to taking up huge amounts of space on the System Volume. Follow my previous post here: https://www.exchangeitup.net/2015/08/exchange-2013-compressing-diagnostic.html

 

Start Database Copy Seeding


Since we created our databases with seeding postponed, we'll go ahead and start that operation now.

If your DB’s are small (or if you’re feeling adventurous) you can seed them all once by running the following cmdlet:

Update-MailboxDatabaseCopy –Server EXCH-MBX-04

If you have huge DB’s, you might want to run a few at a time, specifying different source servers to more evenly distribute the load.

In the EMS, run the following cmdlets one by one:

Update-MailboxDatabaseCopy -Identity DB01\EXCH-MBX-04 –SourceServer EXCH-MBX-01 –DeleteExistingFiles

Update-MailboxDatabaseCopy -Identity DB02\EXCH-MBX-04 –SourceServer EXCH-MBX-02 –DeleteExistingFiles

Update-MailboxDatabaseCopy -Identity DB03\EXCH-MBX-04 –SourceServer EXCH-MBX-03 –DeleteExistingFiles

After the first 3 copies complete, run the next 3:

Update-MailboxDatabaseCopy -Identity DB04\EXCH-MBX-04 –SourceServer EXCH-MBX-01 –DeleteExistingFiles

Update-MailboxDatabaseCopy -Identity DB05\EXCH-MBX-04 –SourceServer EXCH-MBX-02 –DeleteExistingFiles

Update-MailboxDatabaseCopy -Identity DB06\EXCH-MBX-04 –SourceServer EXCH-MBX-03 –DeleteExistingFiles

**Note** Ensure that no backup is currently running during the seeding or else seeding will fail.

**Note** Depending on the size of your DB’s this operation can take a long time.

Check Database file Creation

Go to C:\ExDBs on your new server and get the Properties on the folder, it shouldn’t be taking up any space (should be 0 bytes) – this is because the data is actually stored on Volume1 not on C:\.

Now open C:\ExDBs\DB01\DB01.db and DB01.log and you should see the .edb file and the logs in their respective folders.

Other Install Notes/Cleanup

If your new node will be passive, run the following in the EMS to ensure that Databases won't mount on this server:

Set-MailboxServer -Identity EXCH-MBX-04 -DatabaseCopyAutoActivationPolicy blocked

Edit: Thanks to "gg" in the comments, if you have a need to enable DAC (Datacenter Activation Coordination) Mode you can do that any time. Since this was my fourth node, I already had it enabled and overlooked that step.

To enable it, run the following cmdlet:

Set-DatabaseAvailabilityGroup -Identity "DAGName" -DatacenterActivationMode DagOnly

**Note** Change "DAGName" to the name of your DAG

Check mailflow to ensure incoming and outgoing mail still works. This includes checking the mail queues on the new server to make sure they’re processing mail.

Check autodiscover to make sure clients can still connect to Exchange and nothing went awry with URLs/DNS.

If you run Anonymous Relay, you’ll want to set up the Receive Connector on the New Server to match the ones on your other nodes, and create any DNS entries needed for relay.

Congrats! You're done! Now target your backups to your new server and take a day off...I know you did this install and took up your entire Saturday ;)


4 comments:

  1. Hi and thanks for sharing your experience! What about enabling DAC mode? it is not enable by default when adding third node. thanks, georg

    ReplyDelete
    Replies
    1. Thanks for the note! I've added it in the "Other Install Notes" section

      Delete
  2. Thank you very much for your help :) I do have a question... Lets say that i have a problematic exchange mailbox server, after i follow your instructions how i can get rid of the problematic one?

    ReplyDelete
    Replies
    1. Hi George, it depends how bad the broken Mailbox Server is. If it's totally broken, you'd need to recover it to new hardware/VM and then proceed with removing it from the DAG.
      If it's still operational then you can remove it by removing the DB copies, then remove it from the DAG, then uninstall Exchange from the machine.

      Delete