-->

Wednesday, July 27, 2022

Exchange Set Forwarding On Mailboxes In Bulk From CSV

My company just sold off a couple of its business units and the sales agreement says offboarding users must keep their legacy email access for 6 months. But, they also need messages forwarded to their new (external) email addresses.

Normally, you could do this by OU (Organizational Unit) but in my case, not everyone in that OU is leaving and management didn't want to create a new OU since GPOs and ACLs are already linked to said OU.

So, we need to be able to import a CSV of those specific users and set the forwarding in one shot...I have about 450 users in my list, that would take a long time doing it manually :)

Create A New Send Connector

First, we need to create a dedicated Send Connector going out to the new external domain. This allows us to forward without creating External Contacts for all of the users - much easier:

In the EAC, navigate to Mail Flow, Send Connectors+

In the New Send Connector window, give it a name like "External Forward" and click Next

Create Send Connector

Leave MX record selected and click Next

Send Connector to MX

Click the + and under the FQDN type the domain name for the external contact, click Save, and Next

Send Connector Domain

For the Source Server, click the + and select your Edge server if you have one, or your Mailbox servers (all of them) and click Ok, then Finish

Send Connector Source Server

Enable Verbose Logging on the Connector:

You'll want full logging on the connector so you can check the SMTPSend protocol logs later to verify successful sending.

In the Exchange Management Shell (EMS), run the following:

Set-SendConnector "External Forward" -ProtocolLoggingLevel Verbose

Optional:

If the external domain requires it, you'll need to enable forced TLS, else messages will be dropped.

In the EMS run the following:

Get-SendConnector "External Forward" | Set-SendConnector -RequireTLS $true

Create New Remote Domain

Next, we need to create a new Remote Domain in Exchange - this will allow incoming messages to hit our Exchange servers and forward/relay on to the new, external domain.

Fire up the Exchange Management Shell (EMS) and run:

New-RemoteDomain -DomainName *.externaldomain.com -Name ExternalDomain

**Note** Change "*.externaldomain.com" to the SMTP domain name of the external domain (using the wildcard will ensure any subdomains are included) and change "ExternalDomain" to the name of the external domain

Next, we need to allow messages to be forwarded to the new remote domain, by running:

Set-RemoteDomain "ExternalDomain" -AutoForwardEnabled $True

**Note** Change "ExternalDomain" to the name you gave it in the earlier cmdlet

Set Forwarding

Now, we'll create our CSV with Email and Forward as the headers and the users' current email and new external email address under like so:

CSV Format




Lastly, we'll run the cmdlet to set the forwarding on the user in the CSV:

$users=Import-csv C:\Temp\emails.csv

ForEach($user in $users){Set-Mailbox $user.email -DeliverToMailboxAndForward $false -ForwardingSMTPAddress $user.forward}

**Note** This cmdlet sets the ForwardingSMTPAddress switch, NOT ForwardingAddress. You can refer to my previous post to see the difference between those, but in a nutshell ForwardingSMTPAddress is what you use when using a dedicated Send Connector and Remote Domain.

**Note** This will set the messages to be forwarded but not delivered to the current mailboxes, which is what we want since those people are leaving the company.

Now, all incoming messages to those users will be routed to their new external domain without landing in their current mailboxes!

Thursday, June 23, 2022

Skype For Business Showing Out of Office For User Who Is Not OOF

In my Exchange 2016/SFB 2015 environment, I have a user named Mary, where OOF is showing enabled/turned on when looking her up in SFB and Outlook:

SFB OOF Enabled


The problem is: she isn't Out Of Office, she's there, sitting at her desk.

In the Exchange Management Shell (and the ECP in "manage another user") automatic reply status shows as disabled:

EMS OOF Disabled

Well, we know the EMS doesn't lie, so it's gotta be client-related.

We tried turning off Outlook cached mode, creating a new local mail profile, and deleting Mary's SFB cache...nothing worked.

On a hunch, I checked Mary's calendar and one of her direct reports named Laura, sent her own OOF as an invite to Mary, which added it to Mary's calendar, therefore showing Mary as OOF:

OOF Invite


That's bad practice, and totally confusing for other users. 

To fix it, Mary needs to delete Laura's calendar event and it will remove Mary from looking like she's Out Of Office.

Saturday, March 26, 2022

Exchange Install Let's Encrypt On SpamTitan

If your company is anything like the one I work for, they really hate spending money on technology...it almost becomes a yearly fight when renewing things like certificates, even though certs are needed for almost everything regarding messaging.

One of those certs is the SSL cert on SpamTitan, which is used for the WebUI secure connections and more importantly, outbound TLS connections - meaning: external receiving servers can't verify TLS without a public cert.

Luckily, SpamTitan now supports free certs from Let's Encrypt. Now, you're prolly thinking "woah, Let's Encrypt? On production?" Let's Encrypt sometimes gets a bad rap because it expires in 90 days and depending on the system you have to jump through hoops to install/renew it. It's very easy on SpamTitan, it's free, and it will automatically request the certificate, install it, and set up TLS and HTTPS for you. Additionally, it will be automatically renewed before expiration, without ever touching it again...can't do that with a paid public cert, can ya? 😁

Here's how to install the Let's Encrypt certificate on your SpamTitan appliance.

Go to Settings > SSL

Provide all the same information as if you're requesting a cert from a public CA:

SpamTitan SSL

Common Name: This is the fully qualified domain name (FQDN) that will be used in the URL to access the SpamTitan UI. It must match the server name exactly, otherwise you will get a warning dialog every time you visit the site. An example, spamtitan.exchangeitup.com.

Organization: This is the name of your company or organization.

Organization Unit: Specify a specific department within your organization, like IT.

City: This is the name of the city or town where the organization is located

State/Province: This is the full name of the state or province where the organization is located.

Country: This is the two-letter country code of the location of the organization. e.g. US

**Note** For most CSR's the OU is optional....this is not so on Let's Encrypt. You must input an OU (I use "IT") or else you'll get the following error:

AcmePhp\Ssl\DistinguishedName::$organizationalUnitName expected a string. Got: ""

Notice that the "$organizationalUnitName expected a string" got an empty value.

If that happens fill out your CSR info again, with an OU this time and it will run successfully

Hit the "Run" button next to "Use Let's Encrypt to generate a Certificate"

SpamTitan Run Let's Encrypt

Important: the outbound cert validation needs to be accessible over port 80 at all times, for Let's Encrypt servers to renew/validate the cert. Blocking port 80 on the firewall if you run your own SpamTitan appliance, or disabling HTTP is not recommended or else Let's Encrypt may not be able to renew the certificate every 90 days. If your security requires it, you can disable HTTP after the validation, just set yourself a reminder to to temporarily enable it in 90 days for renewal.

If you get an error that port 80 is blocked, we'll need to enable HTTP.

Go to Settings > Access/Authentication under the Web Management Protocol section and hit the "Enable" button next to HTTP. It will then show as ON

SpamTitan HTTP

Now, run your Let's Encrypt again, it should verify successfully and be marked as active

SpamTitan Let's Encrypt Active


You'll also see that its installed in the "Installed Signed Certificates" section

SpamTitan Let's Encrypt Installed


SpamTitan Let's Encrypt Cert Info


And if you check the cert in your browser, you'll see the Let's Encrypt Issued By "R3" , which matches the Common Name in the above screenshot.

SpamTitan Let's Encrypt Cert

And you'll see that the Let's Encrypt cert has been set for the HTTPS connections in the Web Management Protocol settings

SpamTitan Let's Encrypt HTTPS


And you'll see that the Let's Encrypt cert has been enabled for TLS connections under Settings > TLS

SpamTitan Let's Encrypt TLS


And lastly, you'll need to make sure that the Let's Encrypt cert is selected for outbound TLS connections, by going to System Setup > Mail Relay...scroll to the bottom and select the Let's Encrypt cert in the drop-down and click Save:

SpamTitan Outbound TLS



There you have it! An easy, FREE cert for securing your SpamTitan management and outbound communications!

Monday, March 14, 2022

Exchange Safely Installing Security Updates Without Errors (Updated Post)

 As you probably know by now, MS is releasing out-of-band Security Updates for Exchange 2013/2016/2019 to combat regularly discovered exploits...it started last year about this time with the HAFNIUM exploit. It only affects on-prem Exchange, according to MS (though I don't know how since Exchange Online is Exchange, but that's beside the point).

I see on lots of forums where Exchange admins mess up their systems by installing the Security Updates wrong; this includes ECP being broken, services not starting, etc. So, I'll show you how to easily install them, error free.

For this, we will be using the command prompt. We WILL NOT use Windows Update and we WILL NOT run the .msp directly.

Hide/Disable The Update

First, you'll want to go to Windows Update and uncheck and hide the Security Update (right-click > hide update).

That means: DO NOT install it from Windows Update, and if you use SCCM/WSUS, stop pushing Exchange Security Updates altogether.

Download the Update

Next, you'll wanna make sure you manually download the correct update, because they are CU-dependent.

For instance, if you are currently up to date on Exchange 2016, which is CU22 as of this writing, you'll want this update:

Download Security Update For Exchange Server 2016 CU22 (KB5012698) from Official Microsoft Download Center

Install The Update

Once you have it downloaded, save it to your favorite directory on all of your Exchange Servers, like C:\Updates.

**Note** You will need to install the update on Mailbox Servers, CAS Servers (if you've spilt roles in Exchange 2013), Edge Servers, and Hybrid Servers. It's your choice if you want to install it on any Management Tools Servers, but it won't hurt to do so.

Put your server in Maintenance Mode.

**Note** I recommend Maintenance Mode even for single servers, because we don't want anything interfering with the install and since the installer will disable Exchange services, it's better to be safe.

Fire up an Elevated CMD (right-click CMD.exe > Run As Administrator).

Next, cd to the directory where you saved your install file like so:

cd C:\Updates

Next, in Windows Explorer navigate to the directory and copy the full path by selecting the .msp file and clicking "Copy Path" in the ribbon or SHIFT-Right-Click the file > Copy As Path:

Security Update Copy Path

In the CMD window, right-click to paste the path, including the double quotes:

Security Update Paste Path

Hit ENTER and the installer will ask if you want to run the file, hit OK and follow the prompts...accept the EULA, next, next a couple times and you're done.

**Note** Some machines will seem to take a loooooooong time during the "calculating space" phase; it could take an hour in some cases. For instance, one of my DAG nodes took 5 minutes, one took 30 minutes, and two took an hour. Let it do it's thing, don't mess with it...rest assured it will go on it's own.

Once the install is finished it may ask you to reboot the machine, go ahead and reboot. If it doesn't prompt you, go ahead and reboot.

Once the server comes back up, take it out of Maintenance Mode and proceed on your next server.

Updated 6/2022:

MS is now releasing SUs (Security Updates) in .EXE format instead of msp, which is supposed to negate running elevated as it automatically extracts to a temp directory and runs using UAC. 

They say you can double-click the su.exe file and it will run. Do I advise doing that? NO! It is still safer to run the .exe using the instructions above (in an elevated CMD) to make sure it runs with proper permissions...just copy the path of the exe instead of the msp.

Congrats, you're done! Now you should be patched up to date, with healthy servers...which means all services are running and ECP/OWA works!

Saturday, December 4, 2021

Exchange Removing Old/Expired Federation Certificates

I'll start off by saying Microsoft really dropped the ball on renewing an Exchange Federation Certificate. 

Following the MS instructions here, tells you how to renew it (sure, that's part of what we need) but they don't tell you how to remove the old one...which leads to the EAC screaming at you that certs are about to expire or already have. And there's no supported/built-in option like removing other self-signed certs.

If you even try to remove the old/expired Federation cert, it will throw the following error:

A special Rpc error occurs on server blah blah. Active certificates in use by Federation cannot be removed.

Well, that's weird...because we've already replaced the Fed Cert, right? Wrong...

When you renew a Federation Cert, it moves the old certificate to an ADSI attribute called "msExchFedOrgPrevPrivCertificate" and sets the new cert as the "msExchFedOrgPrivCertificate".

Notice the "PrevPriv" and "OrgPriv" in those names? 

**Note** These attributes can be found in ADSIEdit under CN=Configuration,DC=domain,DC=com > CN=Services > CN=Microsoft Exchange > CN=DOMAINNAME > CN=Federation Trusts > CN=Microsoft Federation Gateway > Properties

For some (stupid) reason, Exchange still sees the msExchFedOrgPrivCertificate as being used by Federation and MS should really fix that...I mean, do they NOT expect anyone to keep Exchange around longer than 5 years, which is when the certs expire?

In order to get around this, we need to clear that msExchFedOrgPrivCertificate attribute, after which, we can delete the cert from Exchange.

The Fix

I've come up with a PowerShell cmdlet to do just that.

Disclaimer: Modifying anything in ADSI can be destructive, so make sure you have a good AD backup or export the attribute using LDIFDE - you can find guides for that on the net.

Check The Attribute

In the Exchange Management Shell (EMS), run the following cmdlet to check our msExchFedOrgPrivCertificate value:

Get-ADObject -LDAPFilter "(objectclass=msexchfedtrust)" -SearchBase "CN=Microsoft Federation Gateway,CN=Federation Trusts,CN=EXCHANGEITUP,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchangeitup,DC=com" -Properties * | Select msExchFedOrgPrevPrivCertificate

**Note** Change "CN=EXCHANGEITUP" and "DC=exchangeitup,DC=com" to match your domain

It should output a hex value like so:

msExchFedOrgPrevPrivCertificate : {57, 52, 68, 48, 55, 52, 65, 53, 50, 69, 56, 51, 69, 49, 67, 56...}

Clear The Attribute

Next, we're going to clear the value, with the Set-ADObject, like so:

Set-ADObject "CN=Microsoft Federation Gateway,CN=Federation Trusts,CN=EXCHANGEITUP,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchangeitup,DC=com" -Clear msExchFedOrgPrevPrivCertificate 

What this does is search the ADSI Exchange Federation container for the "msExchFedOrgPrevPrivCertificate" attribute and uses the "-Clear" command to remove the entry.

**Note** Change "CN=EXCHANGEITUP" and "DC=exchangeitup,DC=com" to match your domain.

If you run your first cmdlet again, the value will be empty.

Delete The Certificate

Next, we'll get the cert thumbprint by running:

Get-ExchangeCertificate | Select Services,Thumbprint,NotAfter | ft -auto

You'll want to look at the Services column for "Federation" and the NotAfter column to find the old/expired cert and copy that Thumbprint.

Now, we'll remove that old certificate from Exchange by running the following for each of your Mailbox Servers:

Remove-ExchangeCertificate -Server MBX1 -Thumbprint 94D074A52E83E1C81528EE3C43FEE42266ED80A0 -Confirm:$false

**Note** Change "MBX1" to your server name(s), successively.

You can also do this in the EAC by navigating to Servers > Certificates > Select the Old Fed Cert and click the Trashcan icon

Do this on each Mailbox Server.

Now if you check the EAC, the old Federation Certificate will be gone from each of your servers, the alerts will be gone and Federation will still work!

Sunday, September 19, 2021

Renewing A Public Cert On A Totemomail Encryption Gateway

This isn't technically an Exchange article, but for those unfortunate admins that are unlucky enough to have been forced by management to run a Totemomail Encryption Gateway in their environment, this post will show you how to renew your public certificate.

The reason I'm writing this is because Totemo makes ZERO resources available to the public on how to manage this terrible product and they force you to open a ticket for everything...job security I guess.

Unlike most other appliances, web servers, etc. the cert request process on the Totemo Gateway doesn't include the private key, which is needed later for the new cert import. You'd think if it were required for the import, it would be included in the request, but as I said above, the product is garbage.

So you need to export the private key manually and then jump through some hoops to merge it with your new cert from your CA; that requires some OpenSSL work, which is a bit of a pain.

First, start your cert request. 

Expand Certificates > Own Server Certificates > hit the "New HTTPS cert" button.

Totemo Certs


Totemo CRL

Fill out your org info and download/save the req on your local machine.

Totemo CRL Info

Send the req to your public CA and you should get a cert.cer and intermediate.pem back.

Next, export the private key

**Note** The private key is stored in the Totemo database (either locally or in a standalone DB, depending on how you have it set up)

You can export the private key from your current HTTPS cert or from a current user.

To export the private key from the current HTTPS cert, Expand Certificates > Own Server Certificates

Click the cert icon > go to the Details tab > hit the "export with pr. key" button

Totemo Own Cert
Totemo Export Key

Enter your first and second "four eyes password"

Enter your password that will be set on the private key file, twice

Hit "Download"


Totemo Export Key Password

To export the private key from a user, expand User Management > Users.

Totemo Users

Click the certificate management (keys icon) for an active user > click the cert icon > Details tab > "export with pr. key" button

Totemo Users Key

Totemo Users Cert

Totemo Export Users Cert


Enter your first and second  "four eyes password"

Enter your password that will be set on the private key file, twice

Hit "Download"

A gotcha here: make sure you use a strong password on the export (capital, lowercase, number and special character, and at least 8 characters long). There is no "help" prompt on the Totemo for this, it just throws a stupid error that says "error connecting to database" which doesn't mean anything at all.

***Note*** In order for the "export with pr. key" button to be visible in the above steps, you first need to set your "Four Eyes Password" under the Four Eyes Password section...set both passwords; they can be anything, but make them strong. If they haven't been set yet, leave the box empty and hit the "Continue" button to set them.

Totemo Four Eyes

Totemo Four Eyes Password

Once you have your private key exported, fire up OpenSSL, which should be located in C:\Program Files\OpenSSL-Win64

If you don't have it installed you can grab it from here 

First we need to extract the key file. Copy your privatekey.p12 file to the C:\Program Files\OpenSSL-Win64\bin directory then run the following command:

openssl pkcs12 -in privatekey.p12 -out totemokey.key -nodes -nocerts

***Note*** Change privatekey.p12 to the filename of your p12

Next, copy your totemokey.key, cert.crt and intermediate.pem files to the C:\Program Files\OpenSSL-Win64\bin directory

Now we'll merge the files by running the following command:

openssl pkcs12 -export -out totemo.pfx -inkey totemokey.key -in cert.crt -certfile ntermediate.pem

***Note*** Change to totemo.pfx, totemokey.key, cert.crt, ntermediate.pem to the filenames of your certs

You will be prompted to create a password; this can be anything, but I suggest make it strong.

Once it completes, you will have a brand new pfx file, which you will import into the Totemo

Expand Certificates > Own Server Certificates

In the dropdown, select "Import certificate for HTTPS"

Hit the "choose file" button and select your new .pfx

Enter your password you set on the pfx in OpenSSL

Hit OK

Totemo Import Cert

If it goes correctly, it will list your new cert.

Totemo New Cert

You can now delete your old cert and the cert req and reboot the webmail service...that's required or else the old cert will still be assigned to the services (which is stupid).

Thursday, July 8, 2021

Exchange Finding Mismatched Email and Target Addresses

If you follow my blog, you know that I run an Exchange Resource Forest in the US, which syncs MailUsers from another Exchange environment in EMEA using MIM.

For the most part, it works...I say "most part" because it doesn't take into account user name changes as far as the TargetAddress attribute is concerned.

The problem is: if a user has a name change (which is very common when people get married or when HR misspells names - all too often), and their primary email address changes, but the TargetAddress doesn't, then your senders will get bounces from your mail server saying "recipient doesn't exist".

This is also a problem with some Hybrid Environments as well because mailboxes need to have the mail.onmicrosoft.com target in order to function...so this might help in that scenario too.

I came up with a quick PowerShell cmdlet to generate a list of all users with mismatched Email Addresses and Target Addresses.

Fire up the EMS (Exchange Management Shell) or the Active Directory Shell and  run:

Get-AdUser -Filter {Mail -like "*"} -Properties Mail,Targetaddress | Where-Object {($_.mail -and $_.targetAddress) -and ($_.mail -ne ($_.targetAddress -replace "SMTP:",""))} | select mail,@{Expression={$_.targetaddress -replace "SMTP:"};label="Target"} | Export-Csv MismatchedTargets.csv

**Note** If you want narrow down your search to a specific OU, you can add the -Searchbase switch after Get-AdUser

Now you can comb your list and correct the mismatches, and fix those bounces!

Saturday, April 10, 2021

Exchange Script Set Litigation, Description, and Deletion Protection

In most large organizations, the Active Directory (AD) team and Messaging Team are separate, and in most cases don't coordinate their account operations very much...meaning they don't talk to each other, until it's too late.

Case in point: A user's mailbox is under Litigation Hold. The Exchange admins know this, the AD admins don't. The AD admins delete the AD account from Active Directory Users and Computers (ADUC) effectively disabling the mailbox...and all content needed for legal hold goes with it.

An easy way to prevent this from happening is to set the AD Account description field with something like "Litigation Hold - Do Not Delete" and setting the "Prevent Accidental Deletion" flag to Enabled on the Object.

For that purpose, I've created the following script which will prompt you for a user, append the description, and enable prevent delete.

Instead of manually setting Litigation Hold in Exchange, run this script in the Exchange Management Shell (EMS), which will set everything for you. Then when the AD admins look at the account, they'll hopefully see the "Do Not Delete" description and even if they try, ADUC will throw an error that it's protected from deletion!

**Note** We run ServiceNow (SNOW) in our environment, which has the power to delete AD accounts. Through my testing, SNOW honors the Prevent Deletion flag as well and will throw an error, so I would assume most automation suites would too.

You can grab the script from my GDrive or copy/paste the following block into Notepad and save as "Set-Litigation-ADProtection.ps1":

#######################################################
# This script will prompt you for a mailbox, then
# set Litigation Hold, change AD description,
# and set AD deletion protection on account
#
# Run Set-Litigation-ADProtection.ps1 in the EMS
#
# Stacey Branham
#
# 2021
#######################################################
$name = Read-Host "Enter a username"

##Append legal hold disclaimer to AD Description Field

Get-ADUser "$name" -Properties Description | ForEach-Object {Set-ADUser $_ -Description "$($_.Description) - Do Not Delete - Legal Hold"}

##Enable accidental deletion protection on account

Get-ADUser "$name" | ForEach-Object {Set-ADObject -Identity $_ -protectedFromAccidentalDeletion $True}

##Enable Litigation Hold on Mailbox

Set-Mailbox "$name" -LitigationHoldEnabled $true

Start-Sleep -s 30

##Checking Our Work

Get-Mailbox "$name" | FL Name,LitigationHoldEnabled,@{N='Description';E={(Get-ADUser $_.Name -properties description).Description}},@{N='Protected';E={(Get-ADUser $_.Name -properties *).ProtectedFromAccidentalDeletion}}

Write-Host "All Done, check the above settings! `n`n" -ForegroundColor Green

**Note** You can change the "- Do Not Delete - Legal Hold" in the 3rd line to something of your choosing...that is what will be appended to the current Description Field.

**Note** If you have a large AD environment, with slow replication, you may need to change the "Start-Sleep -s 30" to 60 or higher to give it time for the "Set" cmdlets to finish.

Now, your users under Hold won't be deleted and you'll save yourself the headache of de-tombstoning the AD account and reconnecting the mailbox!

Wednesday, January 6, 2021

PSA: Kemp Patch 7.2.52.0 May Break Exchange Federation

In my Exchange 2016 environment, I run a Kemp LoadMaster VLM-2000 (which is awesome and their support is the best).

If you run a LoadMaster as well, and you're planning to update, please read my notes below:

I recently updated my LoadMaster to the latest build (7.2.52.0) and along with it, the cipher set gets replaced with their new "bestpractices cipher set". 

You can see the release notes here, which lays out what is removed and what is kept.

Basically, they removed unsecure ciphers according to SSL guidelines, but what I found (the hard way) is that it may break your Exchange Free/Busy Federation you have with other organizations.

In my organization, we run two Resource Forests, one in Europe and one in the US (which I am responsible for). After updating the Kemp, I got reports from the EU users that they can no longer view US users' Free/Busy. 

Interestingly, this only affected Outlook, not OWA; the Scheduling Assistant in OWA worked fine.

In order to fix the issue, I had to set the cipher set back to default. 

**Note** The better fix is to actually friggin' update Outlook to support new ciphers, but my client is woefully behind on client patching.

Here's how to set Kemp ciphers:

In the Kemp, navigate to Virtual Services > View/Modify Services > click Modify:

Kemp Modify VSS

**Note** You'll need to modify the VSSs for both internal and external.

Next, expand the SSL Properties section and in the "Cipher Set" dropdown choose either "default" or "Default_NoRc4". 

Kemp Ciphers

**Note** "Default_NoRc4" is more secure, but may not work with legacy clients, so "Default" can work also.

Once done, your Federated partners should be able to view Free/Busy.

If you want to keep the new "best practice" ciphers, you can build your own custom cipher set to include both the old ones (which worked previously) and the new updated ones. Just make sure to backup your LM first!

Under the SSL Properties, next to the Cipher Set dropdown, click "Modify Cipher Set".

You can then cross reference your current cipher set with the Kemp article linked above, to add/remove the ciphers and save your own list.

Saturday, October 24, 2020

Exchange Create New Databases And Delete the Old Ones

On rare occasions you may need to create new Mailbox Databases on your Exchange DAG. Maybe you need to scale out for a company merger, or re-balance your mailboxes evenly across your Mailbox servers. 

Or, as seen on many, many forums, people ask how to shrink databases safely, without downtime...which leads to: Like me, you planned for a year’s worth of mail to migrate from Notes to Exchange and the Quest guy (yes, when you pay for the Quest Migration Tool, an expert is required to run it) moved EVERYTHING going back 15 years, including when the tool failed on hundreds of mailboxes and created multiple subfolders of duplicate items. So, in my case I ended up with 6 Databases of ~1TB each...after deduping, remember that deletions leaves whitespace, they’re taking up a lot of disk space. MS technically supports 16TB DB’s but that gets pretty dicey if you have failovers or if you have to perform a restore.

When getting DB size under control (read that as "shrinking"), best practice is to create new DB’s, migrate mailboxes and delete the old DB’s…DO NOT perform an offline defrag, especially in a DAG because it will cause downtime and ruin your month if/when it goes wrong…you also need 1.5x the size of a DB to defrag, nope! Moving mailboxes causes no outage, and is nearly transparent to users. I say "nearly" because Outlook may do a small reconnect during the last 30 seconds of a mailbox move, but if you have your load balancer settings correct, users will never know.

In this article, I’ll show you how to set up those new databases and get rid of the giant old ones...strap in, it's a long one :)

Create DAG Mount Points

First, we’ll create the mount point needed for our new databases volume.

Perform the following steps on each Exchange Mailbox Server – it is very important that volumes and folders match exactly on each server.

If you’ve followed my previous Exchange setup articles before or you follow best practice, you may already have your Exchange Volume presented to your servers as a drive. 

**Note** In my environment, I was able to grow the disks in VMWare, so I created the DB structure on the same volume as the old DB’s. If you’re setting up new disks, then follow the next section.


1. On the C: drive, create a folder called EXVols – this folder will be used to mount our E: (Volume1).
2. Next, on the C: drive, create a folder called ExDBs – this folder will hold the Database mount points.

Creating the Volume

3. Within the ExVols folder, create a new folder called Volume1 

4. Open Windows Disk Management to mount our volume to our ExVols folders.
5. Right-click E: and select Change Drive Letter and Paths
6. Click Add and browse to the location of the Volume1 folder – C:\ExVols\Volume1
7. Click OK, twice

You should see the folder with a Disk icon meaning it’s now a Mount Point. 

Creating the Database Folders

Under the C:\ExDBs folder, create the new Database folders for as many DB’s as you plan to have. In my case we’ll have 12, so we’ll create the following folders:


C:\ExDBs\MDB01
C:\ExDBs\MDB02
C:\ExDBs\MDB03
C:\ExDBs\MDB04
C:\ExDBs\MDB05
C:\ExDBs\MDB06
C:\ExDBs\MDB07
C:\ExDBs\MDB08
C:\ExDBs\MDB09
C:\ExDBs\MDB10
C:\ExDBs\MDB11
C:\ExDBs\MDB12

**Note** We’re creating the DB folders with different names than your current DB folders. If you already databases name DB01-DB06, we’ll name the new ones MDB01-MDB12. You might wonder "why not continue with DB07 and up?". We’re replacing the old DB’s because we want to get all of that whitespace back, which we wouldn’t be able to achieve without moving all mailboxes and deleting the old DB’s.

After you have your folders set, open an Elevated command prompt, and run:

mountvol

This will list the available volumes for use.

In my case I know the one I want is \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\ because we can see the folder Volume1 is mounted to it

Run the following command to mount MDB01:

Mountvol MDB01 \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\

If you go to your C:\ExDBs folder, you’ll notice the folder icon for MDB01 has changed to a mount point icon.

Now mount your other DB’s:


Mountvol MDB02 \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\
Mountvol MDB03 \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\
Mountvol MDB04 \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\
Mountvol MDB05 \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\
Mountvol MDB06 \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\
Mountvol MDB07 \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\
Mountvol MDB08 \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\
Mountvol MDB09 \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\
Mountvol MDB10 \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\
Mountvol MDB11 \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\
Mountvol MDB12 \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\ 

If you run mountvol again, you’ll see all DB’s mounted under the Volume1 folder.

Creating the Database Directory Structure

Next, we’ll create the database directory structure; each folder will have 2 folders beneath it: one folder for the Database .edb file and one for the Logs.

**Note** In a DAG, it is best practice to keep database and log files on the same volume, as long as it is separate from the System Volume.

You can create these folders directly from Volume1 (E:) or by going to C:\ExDBs\MDB01 through MDB12 (they will have the same folders).

In E:\ExDBs\MDB01, create a new folder named MDB01.db and new folder called MDB01.log.

In E:\ExDBs\MDB02, create a new folder named MDB02.db and new folder called MDB02.log.

In E:\ExDBs\MDB03, create a new folder named MDB03.db and new folder called MDB03.log.

In E:\ExDBs\MDB04, create a new folder named MDB04.db and new folder called MDB04.log.

In E:\ExDBs\MDB05, create a new folder named MDB05.db and new folder called MDB05.log.

In E:\ExDBs\MDB06, create a new folder named MDB06.db and new folder called MDB06.log.

In E:\ExDBs\MDB07, create a new folder named MDB07.db and new folder called MDB07.log.

In E:\ExDBs\MDB08, create a new folder named MDB08.db and new folder called MDB08.log.

In E:\ExDBs\MDB09, create a new folder named MDB09.db and new folder called MDB09.log.

In E:\ExDBs\MDB10, create a new folder named MDB10.db and new folder called MDB10.log.

In E:\ExDBs\MDB11, create a new folder named MDB11.db and new folder called MDB11.log.

In E:\ExDBs\MDB12, create a new folder named MDB12.db and new folder called MDB12.log.

**Note** You will notice that each folder also contains the DB folders for the other DB’s. For instance if you go to C:\ExDBs\MDB02, you'll see all of the MDB folders listed. That’s normal because we’re using the same underlying storage presented to Windows.

Create Mailbox Databases

Next, we’ll be creating our twelve Databases, and evenly distributing them across our servers. 

MDB01, MDB04, MDB07, MDB10 on Server01

MDB02, MDB05, MDB08, MDB11 on Server02

MDB03, MDB06, MDB09, MDB12 on Server03

Create the Databases by running the following cmdlets in the Exchange Management Shell (EMS):


New-MailboxDatabase –Name MDB01 –Server EXCH-MBX-01 –LogFolderPath C:\ExDBs\MDB01\MDB01.log –EdbFilePath C:\ExDBs\MDB01\MDB01.db\MDB01.edb

New-MailboxDatabase –Name MDB02 –Server EXCH-MBX-02 –LogFolderPath C:\ExDBs\MDB02\MDB02.log –EdbFilePath C:\ExDBs\MDB02\MDB02.db\MDB02.edb

New-MailboxDatabase –Name MDB03 –Server EXCH-MBX-03 –LogFolderPath C:\ExDBs\MDB03\MDB03.log –EdbFilePath C:\ExDBs\MDB03\MDB03.db\MDB03.edb

New-MailboxDatabase –Name MDB04 –Server EXCH-MBX-01 –LogFolderPath C:\ExDBs\MDB04\MDB04.log –EdbFilePath C:\ExDBs\MDB04\MDB04.db\MDB04.edb

New-MailboxDatabase –Name MDB05 –Server EXCH-MBX-02 –LogFolderPath C:\ExDBs\MDB05\MDB05.log –EdbFilePath C:\ExDBs\MDB05\MDB05.db\MDB05.edb

New-MailboxDatabase –Name MDB06 –Server EXCH-MBX-03 –LogFolderPath C:\ExDBs\MDB06\MDB06.log –EdbFilePath C:\ExDBs\MDB06\MDB06.db\MDB06.edb

New-MailboxDatabase –Name MDB07 –Server EXCH-MBX-01 –LogFolderPath C:\ExDBs\MDB07\MDB07.log –EdbFilePath C:\ExDBs\MDB07\MDB07.db\MDB07.edb

New-MailboxDatabase –Name MDB08 –Server EXCH-MBX-02 –LogFolderPath C:\ExDBs\MDB08\MDB08.log –EdbFilePath C:\ExDBs\MDB08\MDB08.db\MDB08.edb

New-MailboxDatabase –Name MDB09 –Server EXCH-MBX-03 –LogFolderPath C:\ExDBs\MDB09\MDB09.log –EdbFilePath C:\ExDBs\MDB09\MDB09.db\MDB09.edb

New-MailboxDatabase –Name MDB10 –Server EXCH-MBX-01 –LogFolderPath C:\ExDBs\MDB10\MDB10.log –EdbFilePath C:\ExDBs\MDB10\MDB10.db\MDB10.edb

New-MailboxDatabase –Name MDB11 –Server EXCH-MBX-02 –LogFolderPath C:\ExDBs\MDB11\MDB11.log –EdbFilePath C:\ExDBs\MDB11\MDB11.db\MDB11.edb

New-MailboxDatabase –Name MDB12 –Server EXCH-MBX-03 –LogFolderPath C:\ExDBs\MDB12\MDB12.log –EdbFilePath C:\ExDBs\MDB12\MDB12.db\MDB12.edb

**Note** You will get a Warning that “The Information Store must be restarted after DB creation” - this is by design. Exchange 2013/2016 uses different memory management so that store.exe does not use all available RAM. MS suggests DB creation is during maintenance window, since restarting the store.exe service dismounts databases active on that server…even though it’s super annoying.

 

Add Database Copies

Next, we'll create our DB copies. Note the Activation Preference (AP), which mounts the copy according to server:


DB

EXCH-MBX-01 -AP

EXCH-MBX-02 -AP

EXCH-MBX-03 -AP

MDB01

1

2

3

MDB02

3

1

2

MDB03

3

2

1

MDB04

1

2

3

MDB05

3

1

2

MDB06

3

2

1

MDB07

1

2

3

MDB08

3

1

2

MDB09

3

2

1

MDB10

1

2

3

MDB11

3

1

2

MDB12

3

2

1


Run the following cmdlets in the EMS to create the DB copies according to activation preference, separately on each line (yes, there's a bunch):

Add-MailboxDatabaseCopy -Identity MDB01 -MailboxServer EXCH-MBX-02 -ActivationPreference 2
Add-MailboxDatabaseCopy -Identity MDB01 -MailboxServer EXCH-MBX-03 -ActivationPreference 3
Add-MailboxDatabaseCopy -Identity MDB02 -MailboxServer EXCH-MBX-03 -ActivationPreference 2
Add-MailboxDatabaseCopy -Identity MDB02 -MailboxServer EXCH-MBX-01 -ActivationPreference 3
Add-MailboxDatabaseCopy -Identity MDB03 -MailboxServer EXCH-MBX-02 -ActivationPreference 2
Add-MailboxDatabaseCopy -Identity MDB03 -MailboxServer EXCH-MBX-01 -ActivationPreference 3
Add-MailboxDatabaseCopy -Identity MDB04 -MailboxServer EXCH-MBX-02 -ActivationPreference 2
Add-MailboxDatabaseCopy -Identity MDB04 -MailboxServer EXCH-MBX-03 -ActivationPreference 3
Add-MailboxDatabaseCopy -Identity MDB05 -MailboxServer EXCH-MBX-03 -ActivationPreference 2
Add-MailboxDatabaseCopy -Identity MDB05 -MailboxServer EXCH-MBX-01 -ActivationPreference 3
Add-MailboxDatabaseCopy -Identity MDB06 -MailboxServer EXCH-MBX-02 -ActivationPreference 2
Add-MailboxDatabaseCopy -Identity MDB06 -MailboxServer EXCH-MBX-01 -ActivationPreference 3
Add-MailboxDatabaseCopy -Identity MDB07 -MailboxServer EXCH-MBX-02 -ActivationPreference 2
Add-MailboxDatabaseCopy -Identity MDB07 -MailboxServer EXCH-MBX-03 -ActivationPreference 3
Add-MailboxDatabaseCopy -Identity MDB08 -MailboxServer EXCH-MBX-03 -ActivationPreference 2
Add-MailboxDatabaseCopy -Identity MDB08 -MailboxServer EXCH-MBX-01 -ActivationPreference 3
Add-MailboxDatabaseCopy -Identity MDB09 -MailboxServer EXCH-MBX-02 -ActivationPreference 2
Add-MailboxDatabaseCopy -Identity MDB09 -MailboxServer EXCH-MBX-01 -ActivationPreference 3
Add-MailboxDatabaseCopy -Identity MDB10 -MailboxServer EXCH-MBX-02 -ActivationPreference 2
Add-MailboxDatabaseCopy -Identity MDB10 -MailboxServer EXCH-MBX-03 -ActivationPreference 3
Add-MailboxDatabaseCopy -Identity MDB11 -MailboxServer EXCH-MBX-03 -ActivationPreference 2
Add-MailboxDatabaseCopy -Identity MDB11 -MailboxServer EXCH-MBX-01 -ActivationPreference 3
Add-MailboxDatabaseCopy -Identity MDB12 -MailboxServer EXCH-MBX-02 -ActivationPreference 2
Add-MailboxDatabaseCopy -Identity MDB12 -MailboxServer EXCH-MBX-01 -ActivationPreference 3

Next, we need to exclude the old databases from mailbox autoprovisioning, so new mailboxes will only be created on the new DB’s during our migration…we don’t want to finish our moves and later have to go back and move new mailboxes too!

Run the following cmdlets on each line:

Set-MailboxDatabase "DB01" -IsExcludedFromProvisioning $true
Set-MailboxDatabase "DB02" -IsExcludedFromProvisioning $true
Set-MailboxDatabase "DB03" -IsExcludedFromProvisioning $true
Set-MailboxDatabase "DB04" -IsExcludedFromProvisioning $true
Set-MailboxDatabase "DB05" -IsExcludedFromProvisioning $true
Set-MailboxDatabase "DB06" -IsExcludedFromProvisioning $true

**Note** Change the database names to match your old DB’s

Mailbox Migration

In this section, I’ll show you how I personally move mailboxes to evenly spread them across the new Databases. We’ll be moving mailboxes in chunks, so we don’t overload our mailbox servers, by creating a set of CSV’s for each OldDB-to-NewDB.

**Note** A lot of Exchange admins like "group" mailbox type, department or an office location for specific databases; that goes against best practice. The reason is: if you lose that database or it becomes corrupt, you risk losing an entire group of mailboxes and if you had one for say "sales" all of those sales people would be down. Instead, we want to spread them around the entire org.

First I get a list of all mailboxes in the organization, by running the following cmdlet:

Get-Mailbox –Resultsize Unlimited | Select PrimarySMTPAddress | Export-CSV C:\Temp\All_Mailboxes.csv

Now, open the CSV in Excel and add a column where we will number them 1-12, down the column, which will correspond to our new databases…it will look like this:

 

New DB

PrimarySMTPAddress

1

JohnLennon@exchangeitup.com

2

GeorgeHarrison@exchangeitup.com

3

PaulMcCartnet@exchangeitup.com

4

RingoStarr@exchangeitup.com

5

RobertPlant@exchangeitup.com

6

JimmyPage@exchangeitup.com

7

JohnPaulJones@exchangeitup.com

8

JohnBonham@exchangeitup.com

9

JimiHendrix@exchangeitup.com

10

PeterYarrow@exchangeitup.com

11

PaulStookey@exchangeitup.com

12

MaryTravers@exchangeitup.com

1

DonHenley@exchangeitup.com

2

JoeWalsh@exchangeitup.com

3

TimSchmit@exchangeitup.com

4

DeaconFrey@exchangeitup.com

5

VinceGill@exchangeitup.com

6

NikkiSixx@exchangeitup.com

7

MickMars@exchangeitup.com

8

VinceNeil@exchangeitup.com

9

TommyLee@exchangeitup.com

10

GingerBaker@exchangeitup.com

11

JackBruce@exchangeitup.com

12

EricClapton@exchangeitup.com

1

DimebagDarrell@exchangeitup.com

2

VinniePaul@exchangeitup.com

3

TerryGlaze@exchangeitup.com

4

RexBrown@exchangeitup.com

5

PhilAnselmo@exchangeitup.com

6

RobZombie@exchangeitup.com

7

SeanYseult@exchangeitup.com

8

PhilBuerstatte@exchangeitup.com

9

JohnTempesta@exchangeitup.com

10

TimJeffs@exchangeitup.com

11

TomFive@exchangeitup.com

12

JYuenger@exchangeitup.com

**Note** 5 bonus points if you can name those bands without looking them up :)

Now that we have our Mailbox Master list, we’ll group the users by number. All 1’s will be grouped, all 2’s, all 3’s etc – like so:


New DB

Email Address

1

JohnLennon@exchangeitup.com

1

DonHenley@exchangeitup.com

1

DimebagDarrell@exchangeitup.com

2

GeorgeHarrison@exchangeitup.com

2

JoeWalsh@exchangeitup.com

2

VinniePaul@exchangeitup.com

3

PaulMcCartnet@exchangeitup.com

3

TimSchmit@exchangeitup.com

3

TerryGlaze@exchangeitup.com

4

RingoStarr@exchangeitup.com

4

DeaconFrey@exchangeitup.com

4

RexBrown@exchangeitup.com

5

RobertPlan@exchangeitup.com

5

VinceGill@exchangeitup.com

5

PhilAnselmo@exchangeitup.com

6

JimmyPage@exchangeitup.com

6

NikkiSixx@exchangeitup.com

6

RobZombie@exchangeitup.com

7

JohnPaulJones@exchangeitup.com

7

MickMars@exchangeitup.com

7

SeanYseult@exchangeitup.com

8

JohnBonham@exchangeitup.com

8

VinceNeil@exchangeitup.com

8

PhilBuerstatte@exchangeitup.com

9

JimiHendrix@exchangeitup.com

9

TommyLee@exchangeitup.com

9

JohnTempesta@exchangeitup.com

10

PeterYarrow@exchangeitup.com

10

GingerBaker@exchangeitup.com

10

TimJeffs@exchangeitup.com

11

PaulStookey@exchangeitup.com

11

JackBruce@exchangeitup.com

11

TomFive@exchangeitup.com

12

MaryTravers@exchangeitup.com

12

EricClapton@exchangeitup.com

12

JYuenger@exchangeitup.com

Now, we’ll split our number groups into separate CSV’s with “EmailAddress” as the heading, and we’ll name them something like DB01-MDB01.csv; DB01-MDB02.csv; DB01-MDB03.csv; DB01-MDB04.csv; DB01-MDB05.csv; DB01-MDB06.csv; DB01-MDB07.csv; DB01-MDB08.csv; DB01-MDB09.csv; DB01-MDB10.csv; DB01-MDB11.csv; DB01-MDB12.csv; DB02-MDB01.csv; DB02-MDB02.csv; etc…do this for each old database-to-new database (yes it will take a while, but it will pay off when you start migrating).

Your end results will look like:

DB01-MDB01.csv :

EmailAddress

JohnLennon@exchangeitup.com

DonHenley@exchangeitup.com

DimebagDarrell@exchangeitup.com

DB01-MDB02.csv:

EmailAddress

GeorgeHarrison@exchangeitup.com

JoeWalsh@exchangeitup.com

VinniePaul@exchangeitup.com

DB01-MDB03:

EmailAddress

PaulMcCartnet@exchangeitup.com

TimSchmit@exchangeitup.com

TerryGlaze@exchangeitup.com

Now, we’re ready to migrate mailboxes in bulk from our CSV’s.

In the EMS, run the following:

Import-CSV DB01-MDB01.csv | % {New-MoveRequest -Identity $_.EmailAddress -TargetDatabase MDB01 -BadItemLimit 1000 -AcceptLargeDataLoss}

**Note** Change the CSV name, and the TargetDatabase corresponding to our CSV.

**Note** I always use the BadItemLimit, which requires the AcceptLargeDataLoss switch, because some users who have giant mailboxes, which are very likely to have corrupted items, and we want to skip those and complete the move without errors. You will receive a warning in the Shell, that you may lose data; you can ignore that.

You may be asking “Wait! Why aren’t you using the EAC to migrate batches?!” Well, the EAC uses the uses the arbitration mailbox to manage moves, which generates TONS of transaction logs, which may fill up your disks very quickly! If that happens, you’d need to either enable circular logging or complete backups between each batch…ain’t nobody got time for that! Using the New-MoveRequest is a straight move, plus PowerShell is more fun ;)

To check your move status, run:

Get-MoveRequest | Get-MoveRequestStatistics

When the moves show as “100% completed”, run the following cmdlet to clear the moves (you must do this or else it will cause issue later if you ever needed to move them again or delete a database):

Get-MoveRequest –MoveStatus completed | Remove-MoveRequest –Confirm:$False

Deleting Old Databases

Ok, now that you’ve moved all mailboxes, before we can delete the old databases, we’ll make sure there are no lingering system mailboxes on them.

Run the following cmdlets to check each old database:

Get-Mailbox -Database DB01
Get-Mailbox -Database DB01 -archive
Get-Mailbox -Database DB01 -arbitration
Get-Mailbox -Database DB01 -publicfolder
Get-Mailbox -Database DB01 -monitoring
Get-Mailbox -Database DB01 –auditlog


**Note** Change “DB01” to your old database name(s)

If any of the above cmdlets result in hits, you’ll need to move those mailboxes to the new databases. For instance, to move Monitoring mailboxes, run:

Get-Mailbox -Database DB01 –monitoring | New-MoveRequest –TargetDatabase MDB01

**Note** I always move these mailboxes from the equivalent old DB to the new DB, for instance DB01-to-MDB01, DB02-to-MDB02 and so forth.

Run the above Get-MoveRequestStatistics to check status and Remove-Request cmdlets to clear these moves as well.

Once, you have all system mailboxes moved, we can delete the old database copies, then remove the DB, and delete the .ebd and log files:

In the EMS, run the following:

Remove-MailboxDatabaseCopy –Identity DB01\EXCH-MBX-02 –Confirm:$False
Remove-MailboxDatabaseCopy –Identity DB01\EXCH-MBX-03 –Confirm:$False

Do the above for each old database copy on the server it’s not hosted on. For instance, in the above example, DB01 is hosted on EXCH-MBX-01, so we’re removing the copies from EXCH-MBX-02/03.

You can also use the EAC, by going to Servers > Databases > Click the Remove link under passive copies:

EAC Remove DB Copy


Next, delete the databases, by running:

Remove-MailboxDatabase DB01 –Confirm:$False
Remove-MailboxDatabase DB02 –Confirm:$False
Remove-MailboxDatabase DB03 –Confirm:$False
Remove-MailboxDatabase DB04 –Confirm:$False
Remove-MailboxDatabase DB05 –Confirm:$False
Remove-MailboxDatabase DB06 –Confirm:$False

You’ll receive a warning/confirmation that the database has been removed, but you will need to manually delete the database and logs files.

To do that, browse to the path listed in the warning for both the .edb and log files, and CTRL-A, SHIFT-DEL. (SHIFT-DEL will bypass the Recycle Bin)

**Note** You may receive a warning that files are in use by noderunner or another service. If that happens, just give it a few minutes and try again...they'll go away eventually.

Once you have all database copies, databases and associated files deleted, we can delete the folders and mount points.

Navigate to E:\ (or whatever Volume you have your old DB’s stored on) and delete the DB01.db and DB01.log folders. Do this for each old database folder.

For the mount points, open Computer Management > Disk Management > Right-Click your Exchange Volume > Select your old DB mount point(s) and click Remove:


Disk management

Do the above for each old database mount point.

Congrats, you’re done! You now have shiny new databases, that are much smaller than the originals, which makes it easier to manage, easier to backup (if you’re one of those unlucky people whose management require backups even though a DAG doesn’t need it) and much less time for logs to ship in case of a failover.