-->

Saturday, June 20, 2020

Skype For Business Mobile On Android 10 Can't Connect To Exchange

We recently had several users get new phones with Android 10 installed, or update their existing handsets to Android 10. When trying to view or join meetings from the Skype For Business (SFB) mobile app, it would fail when trying to connect to Exchange.

Some users get prompted for creds over and over, some users get an error "we can't connect to Exchange right now", some users connect but just get a message saying "your meetings will appear here".

Apparently Android 10 includes a change that forces TLS 1.3, but doesn't fall back to older TLS versions. It then gets stuck on the Exchange auth part, which you can see in the client logs: it makes the initial connection to SFB, and then throws obscure "ID_4" errors when it switches over to the Exchange EWS service.

The fix is to disable TLS 1.3 on your load balancer.

I run a Kemp Loadmaster so I'll use that as an example:

Navigate to Virtual Services and select View/Modify Services.

Next, click Modify to the right of your Exchange 2016 VSs.

**Note** You'll want to modify both Internal and External services:

Kemp Modify Internal/External Service

Next, Expand SSL Properties and for Supported Protocols, uncheck the box TLS 1.3:

Kemp Disable TLS 1.3

The change will take effect immediately.

Now hop on to an Android 10 device, and you should be able to open your meetings tab!

Monday, May 25, 2020

Exchange Adding A New Server To An Existing DAG

In my environment, we run Veeam to back up Exchange...this wasn't my decision (since we have a DAG and don't need backups) but I was overridden by management (who don't understand Exchange). 

The problems with Veeam are: 
- It uses VM snapshots, which we all know are not supported with Exchange, and it causes database failovers if it takes too long to commit the snap, which happens very, very often...very often, like, daily.
- If you use Network Backup Mode (in order to not cause failovers) the backup takes a long time' upwards of 70 hours in our shop.
- If you use hotadd mode (which is faster) it causes the aforementioned failovers.
In order to alleviate these issues, Veeam recommends a passive-only DAG node that the backup targets, and running hotadd mode on that passive node.

So, we're going to add a new Mailbox Server to our DAG, and since a web search turns up virtually no instructions on doing that, here ya go!

Install Exchange 2016 Pre-Reqs

On the server that is to be your new Exchange 2016 Mailbox Server, run the following (copy/paste the whole block) in an elevated Windows PowerShell:

Install-WindowsFeature AS-HTTP-Activation, Desktop-Experience, NET-Framework-45-Features, RPC-over-HTTP-proxy, RSAT-Clustering, RSAT-Clustering-CmdInterface, RSAT-Clustering-Mgmt, RSAT-Clustering-PowerShell, Web-Mgmt-Console, WAS-Process-Model, Web-Asp-Net45, Web-Basic-Auth, Web-Client-Auth, Web-Digest-Auth, Web-Dir-Browsing, Web-Dyn-Compression, Web-Http-Errors, Web-Http-Logging, Web-Http-Redirect, Web-Http-Tracing, Web-ISAPI-Ext, Web-ISAPI-Filter, Web-Lgcy-Mgmt-Console, Web-Metabase, Web-Mgmt-Console, Web-Mgmt-Service, Web-Net-Ext45, Web-Request-Monitor, Web-Server, Web-Stat-Compression, Web-Static-Content, Web-Windows-Auth, Web-WMI, Windows-Identity-Foundation

**Note** The Windows Server version needs to match your other DAG nodes, in my environment it’s Server 2012 R2

Install .Net Framework to match what’s installed on your other DAG nodes. At the time of this writing (and in my environment), CU16 is the current build, which requires .Net 4.8.

You can get it here:

https://support.microsoft.com/en-us/help/4503548/microsoft-net-framework-4-8-offline-installer-for-windows

Install MS Unified Communications API 4.0 Core Runtime from the following link:

https://www.microsoft.com/en-us/download/details.aspx?id=34992

Install Visual C++ 2013 from here:

https://www.microsoft.com/en-us/download/details.aspx?id=40784


Exchange Server Install

Uninstall any A/V client if currently installed.

Run Windows Update to patch the server up to the latest

Create your database storage volume; I use drive letter E: for Databases.

Label drive E: “Volume1”.

Mount the Exchange 2016 CU .ISO.

Right-click setup.exe and Run-as admin


1. Install Mailbox role
2. Leave malware scanning enabled
3. Wait a looooong time
4. Click OK to close Installer
5. Reboot the machine to complete Exchange setup
6. After the reboot, check that all Exchange Services are running
7. Install license key, by running the below cmdlet:
Set-ExchangeServer -Identity "server name" -ProductKey XXXX-XXXX-XXXX-XXXX
8. Restart Microsoft Exchange Information Store for the key to take effect.

Set the Exchange URLs

To avoid any Autodiscover/OWA issues, set the Virtual Directory URLs immediately.

**Note** If for some reason OWA and the EAC are broken after setting up the new server (usually this means DNS/Load Balancing isn’t configured properly), put the new server into maintenance mode, then set your URLs – that should fix it.

Use Paul Cunningham’s (ExchangeServerPro) awesome URL scripts to automatically set your namespaces in one shot.

Grab it here: https://practical365.com/exchange-server/powershell-script-configure-exchange-urls/

**Note** We’ll need the URLs to match those on the other DAG nodes.

Add New Server to Load Balancer

Next, we’ll add the new server to our Load Balancer; again, to avoid any service interrupts.

The following instructions are for Kemp Loadmaster, but it should be similar for other appliances as well.

We’ll be modifying the SubVSs for both our Internal and External Virtual Services.

Navigate to Virtual Services > View/Modify Services > Expand SubVSv > Click Modify

Expand Real Servers > Click Add New > Input the IP of the new server > Click Add this Real Server > Click OK

The new server will be listed along with your current ones.

Hit Back and repeat for all 9 SubVSs on both Internal and External Services.

To verify the new server was added and in service, navigate to Real Servers and it will be in the list with your other DAG nodes.

Delete Default Database

Next, we’ll delete the default database on the new server.

Run the following cmdlets in the EMS to delete the default Database:

Remove-mailbox -database "default database name"

**Note** You may receive a warning that health mailboxes were present. You can disregard the warning, as the database will still be deleted and the health mailbox isn’t needed, since our other production databases already have them.

Create DAG Mount Points

**Note** The following section is for reference only - your environment may differ. The directory structures are how I personally set up Exchange Mailbox Servers, which, if you set up Exchange following my "Installing Exchange in a Resource Forest" post, then feel free to create them this way.

Perform the following steps on the new Exchange server – it is very important that volumes and folders match your other DAG servers, exactly.

You should already have your E: Volume presented to your server as a drive.

1. On the C: drive, create a folder called EXVols – this folder will be used to mount our E: (Volume1).
2. Next, on the C: drive, create a folder called ExDBs – this folder will hold the Database mount points.

Creating the Volumes

3. Within the ExVols folder, create a new folder called Volume1.
4. Open Windows Disk Management to mount our volume to our ExVols folders.
5. Right-click E: and select Change Drive Letter and Paths…
6. Click Add and browse to the location of the Volume1 folder – C:\ExVols\Volume1
7. Click OK, twice

You should see the folder with a Disk icon meaning its now a Mount Point. 

Creating the Database Folders

Under the C:\ExDBs folder, create the new Database folders to match the DB’s you have on your other DAG nodes. 

In my case, we have 6, so we’ll create the following folders:

C:\ExDBs\DB01 
C:\ExDBs\DB02
C:\ExDBs\DB03
C:\ExDBs\DB04
C:\ExDBs\DB05
C:\ExDBs\DB06


After you have your folders set, open an Elevated command prompt, and run:

mountvol

This will list the available volumes for use.

In our case we know the one we want is \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\ because we can see the folder Volume1 is mounted to it

Run the following command to mount DB01:

Mountvol DB01 \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\

If you go to your C:\ExDBs, you’ll notice the folder icon for DB01 has changed to a mount point icon.

Now mount your other DB folders:

Mountvol DB02 \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\

Mountvol DB03 \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\

Mountvol DB04 \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\

Mountvol DB05 \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\

Mountvol DB06 \\?\Volume{03cf7f78-ed05-4bb7-a4f0-0914f9575bdd}\

If you run mountvol again, you’ll see all DB’s mounted under the Volume1 folder.

Create Database Directory Structure

Next, we’ll create database directory structure to match those on the other DAG nodes. Each folder will have 2 folders beneath it: one folder for the Database .edb file and one for the Logs.

You can create these folders directly from Volume1 (E:) or by going to C:\ExDBs\DB01 through DB06 (they will have the same folders).

In E:\ExDBs\DB01, create a new folder named DB01.db and new folder called DB01.log.

In E:\ExDBs\DB02, create a new folder named DB02.db and new folder called DB02.log.

In E:\ExDBs\DB03, create a new folder named DB03.db and new folder called DB03.log.

In E:\ExDBs\DB04, create a new folder named DB04.db and new folder called DB04.log.

In E:\ExDBs\DB05, create a new folder named DB05.db and new folder called DB05.log.

In E:\ExDBs\DB06, create a new folder named DB06.db and new folder called DB06.log.

Add New Mailbox Server to the DAG

Run the following in the EMS:

Add-DatabaseAvailabilityGroupServer -identity DAG01 –MailboxServer "EXCH-MBX-04"

**Note** change the –MailboxServer "EXCH-MBX-04" to the name of your new server

Import/Create New Certs

Next, we’ll Export your SAN certificate from one of the other DAG nodes.

In the EAC, navigate to Servers > Certificates

Select your SAN cert, and click the ellipses, then click Export.

Enter the UNC path where you want to store the .pfx and give it a password.

Click the ellipses again and click Import

Enter the UNC path where you exported the cert to, and the password you set.

Hit the Plus Sign and select your new Mailbox Server, and hit Finish.

Do the above process again if you have any Federation Certificates

If you run Unified Messaging, you most likely have a cert provided by your internal CA; you’ll need to request a new cert, because we’ll need to add the FQDN on the new server to that cert.

Follow my previous post https://www.exchangeitup.net/2017/11/exchangesfb-um-cross-forest-with_4.html to request the new cert.

**Note** If your certs status is "invalid" or “revocation check failed” after importing, it’s likely that the server can’t get to the internet because of a proxy; make sure to set web proxy settings on the new server by following my post here:

https://www.exchangeitup.net/2016/08/exchange-20132016-using-exchange.html


Re-Subscribe Edge Server

Next, we’ll need to re-subscribe our Edge Server so the new Mailbox server can participate in Mail Flow.

On the Edge server, in the EMS, run:

New-EdgeSubscription -FileName C:\Temp\EdgeSubscription.xml

Copy the EdgeSubscription.xml file to C:\Temp on any Mailbox server.

On the mailbox server you copied the .xml file to, in the EMS, run:

New-EdgeSubscription -FileData ([byte[]]$(Get-Content -Path "C:\Temp\ EdgeSubscription.xml" -Encoding Byte -ReadCount 0)) -Site "Default-First-Site-Name"

**Note** Change -Site "Default-First-Site-Name" to the name of your Exchange Organization Name. If you run a Resource Forest, you’ll need the full site UNC like so: "resourcedomain.com/Configuration/Sites/Default-First-Site-Name"

Add Database Copies

We will use Postpone Seeding to allow the copy creation to finish before seeding.

The copy creation will automatically set the Activation Preference to the next in-line preference according to what’s already set for your other DAG nodes.

Run the following cmdlets in the EMS to create the DB copies:

**Note** Run each cmdlet separate on each line.

Add-MailboxDatabaseCopy -Identity DB01 -MailboxServer EXCH-MBX-04 –SeedingPostponed

Add-MailboxDatabaseCopy -Identity DB02 -MailboxServer EXCH-MBX-04 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity DB03 -MailboxServer EXCH-MBX-04 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity DB04 -MailboxServer EXCH-MBX-04 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity DB05 -MailboxServer EXCH-MBX-04 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity DB06 -MailboxServer EXCH-MBX-04 –SeedingPostponed

Export Custom Tasks

 

If you run any custom tasks, like cleaning logs, which you can find in my previous post here https://www.exchangeitup.net/2016/11/exchange-20132016-cleaning-up-old.html, you’ll want to export the task from one of your other DAG nodes to the new server.

 

On another DAG node, In Task Scheduler, right-click the task > Export. Save the .xml file on a file share.

On your new server, in Task Scheduler, right-click Task Scheduler Library > Import Task and choose the .xml you exported.


Compress Logging Volume

 

I compress the Diagnostic Log Directory because Exchange doesn’t manage them very well, which leads to taking up huge amounts of space on the System Volume. Follow my previous post here: https://www.exchangeitup.net/2015/08/exchange-2013-compressing-diagnostic.html

 

Start Database Copy Seeding


Since we created our databases with seeding postponed, we'll go ahead and start that operation now.

If your DB’s are small (or if you’re feeling adventurous) you can seed them all once by running the following cmdlet:

Update-MailboxDatabaseCopy –Server EXCH-MBX-04

If you have huge DB’s, you might want to run a few at a time, specifying different source servers to more evenly distribute the load.

In the EMS, run the following cmdlets one by one:

Update-MailboxDatabaseCopy -Identity DB01\EXCH-MBX-04 –SourceServer EXCH-MBX-01 –DeleteExistingFiles

Update-MailboxDatabaseCopy -Identity DB02\EXCH-MBX-04 –SourceServer EXCH-MBX-02 –DeleteExistingFiles

Update-MailboxDatabaseCopy -Identity DB03\EXCH-MBX-04 –SourceServer EXCH-MBX-03 –DeleteExistingFiles

After the first 3 copies complete, run the next 3:

Update-MailboxDatabaseCopy -Identity DB04\EXCH-MBX-04 –SourceServer EXCH-MBX-01 –DeleteExistingFiles

Update-MailboxDatabaseCopy -Identity DB05\EXCH-MBX-04 –SourceServer EXCH-MBX-02 –DeleteExistingFiles

Update-MailboxDatabaseCopy -Identity DB06\EXCH-MBX-04 –SourceServer EXCH-MBX-03 –DeleteExistingFiles

**Note** Ensure that no back is currently running during the seeding or else seeding will fail.

**Note** Depending on the size of your DB’s this operation can take a long time.

Check Database file Creation

Go to C:\ExDBs on your new server and get the Properties on the folder, it shouldn’t be taking up any space (should be 0 bytes) – this is because the data is actually stored on Volume1 not on C:\.

Now open C:\ExDBs\DB01\DB01.db and DB01.log and you should see the .edb file and the logs in their respective folders.

Other Install Notes/Cleanup

If new node will be passive, run the following in the EMS:

Set-MailboxServer -Identity EXCH-MBX-04 -DatabaseCopyAutoActivationPolicy blocked

Check mailflow to ensure incoming and outgoing mail still works. This includes checking the mail queues on the new server to make sure they’re processing mail.

Check autodiscover to make sure clients can still connect to Exchange and nothing went awry with URLs/DNS.

If you run Anonymous Relay, you’ll want to set up the Receive Connector on the New Server to match the ones on your other nodes, and create any DNS entries needed for relay.

Congrats! You're done! Now target your backups to your new server and take a day off...I know you did this install and took up your entire Saturday ;)


Saturday, February 8, 2020

Exchange Set OOF/AutoReply Template For Specific Mailboxes

I recently got a request from one of our location managers to implement a standard Out Of Office (OOF or Automatic Reply in Exchange lingo) because it seems users weren't using appropriate verbiage when creating their own.
Not a bad idea really...the issue I have is: we run a Resource Forest, and we use Custom AD Attributes for location codes that get sync'd from the Accounts Forest.
In order to set the OOF template by attribute, I had to do some fancy footwork in PowerShell to ensure only those specific users got the template.

In my example, we'll be using two AD attributes (CustomAttribute15 and ExtensionCsutomAttribute1) because that's what we set in my environment. You might be thinking "my organization is NORMAL, how can I set them by using something like the Office attribute?" That's easily doable, so I'll go over that later.

First, we'll want to check who has AutoReply currently enabled, because we don't want to overwrite those, as it will be sending a non-descriptive reply with no user-specific info and might confuse recipients.

To export a list of who has OOF set, run the following cmdlet in the Exchange Management Shell:

Get-Mailbox -Resultsize Unlimited | where {($_.CustomAttribute15 -eq 'USNY') -or ($_.ExtensionCustomAttribute1 -eq 'USNY')} | Get-MailboxAutoReplyConfiguration | where {$_.AutoReplyState -eq "disabled"}| Select Identity,StartTime,EndTime,AutoReplyState | Export-Csv NY_oof.csv

In the above cmdlet, we're checking for the auto reply in a "disabled" state for users who are located in New York (the USNY attribute). The reason we used AutoReplyState -eq "disabled" and not enabled is because the state could have also been "scheduled" so even though disabled will prolly give a lot of results, it's more fail-safe.

Next, we'll build our template into a variable using HTML formatting by running:

$reply = "<html><head></head><body><p>Thank you for your email.  I am out of the office from [DAY], [MONTH/DATE] through [DAY], [MONTH/DATE] and with limited access to email OR do not have access to email.  I will respond to your email as soon as possible.</br></br>For urgent topics, please text me on my cell at [PHONE], or if you need immediate assistance, please contact, [NAME] at [EMAIL] or [PHONE].</p></body></html>"

**Note** Feel free to change the text between the first and last "< p >" tags to fit your organization.

Setting Template By Attributes:

Now, we'll implement the template on the mailboxes that we checked earlier, with:

Get-Mailbox -Resultsize Unlimited | where {($_.CustomAttribute15 -eq 'USNY') -or ($_.ExtensionCustomAttribute1 -eq 'USNY')} | Get-MailboxAutoReplyConfiguration | where {$_.AutoReplyState -eq "disabled"} | Set-MailboxAutoReplyConfiguration –InternalMessage $reply -ExternalMessage $reply

The above cmdlet will set the HTML reply for both internal and external AutoReplies, for the users in NY, who don't have OOF enabled. It will not enable OOF, it will only set the text in the reply for when users go to enable it themselves.

Setting Template By Office:

Earlier I said we can do it using more "standard" attributes, here's how: To set the reply template according to the "Office" attribute, we'll do the following:

Get-Mailbox -Resultsize Unlimited -Filter "Office -like 'New York'" | Get-MailboxAutoReplyConfiguration | where {$_.AutoReplyState -eq "disabled"} | Set-MailboxAutoReplyConfiguration –InternalMessage $reply -ExternalMessage $reply

Setting Template For All:

What if we wanted to set it on all mailboxes? Even easier, and frankly what I would do if management would let me. To do that, run:

Get-Mailbox -Resultsize Unlimited | where {$_.AutoReplyState -eq "disabled"} | Set-MailboxAutoReplyConfiguration –InternalMessage $reply -ExternalMessage $reply

Now, we can spot check and see that the template has been set on a couple users from our original CSV export.

In the Shell, run the following:

Get-Mailbox "stacey branham" | Get-MailboxAutoReplyConfiguration

Or, you can check in the EAC, by clicking your name in the upper right corner, select "another user", find the user in the GAL, and select "Set up an automatic reply message" in the right pane.

The results will be shown in the internal and external message boxes.

Now your users will have a standard AutoReply!

Saturday, January 18, 2020

Skype For Business Planning Diagram: Hybrid Traffic Flow

By now, all of my readers should know that I like to draw pretty pictures for planning projects. Recently, I completed a Skype For Business Hybrid build out and I created a diagram in order for the client's firewall team to open ports for the Hybrid traffic flow.

SFB 2015 Hybrid Traffic Flow Overview

In the drawing below, we have the SFB Hybrid already set up, and the ports on the firewall are set to any/any - meaning, everything is wide open and traffic is flowing great. 

But, we need to lock it down to only the ports needed and we need to visualize to/from "what server/service" to/from what "O365 service".

I've also included a table in the drawing, of the on-prem services, the port, protocol and direction. This way our firewall team can easily cross-check with the diagram.

**Note** You'll see in the diagram, the ports required for Exchange Unified Messaging. This particular client was still using ExUM, before they transition to cloud voicemail...yes, you can still configure Exchange Online UM, but it requires SFB Hybrid.


SFB Hybrid Traffic Flow

To edit the diagram to fit your organization, you'll need to download the most excellent SFB stencils by PaulB

Once you have those downloaded, move them to C:\Users\Your-User\Documents\My Shapes

Then hop over to my Google Drive and grab the SFB-Office 365 Ports Traffic Flow.vsdx

Feel free to edit the drawing to fit your needs. Happy drawing!

Saturday, January 4, 2020

SFB Error: "Domain Not Ready" When Publishing Topology

Our Skype For Business environment has been running great for years and recently we went to add a SIP Trunk and the Topology Builder threw a bunch of errors about the domain not being ready.
This was very odd, since prepping the domain is the first step when installing SFB, and as I said, was done years ago.

From the looks of it, someone (another admin with too much power and little SFB knowledge) deleted a bunch of permissions...most likely the permissions for the various RTC and CS groups that are required on several AD OU's.

The check the full errors, click view logs in the topology builder results, or navigate to C:\Users\"yourusername"\AppData\Local\Temp\2\TopologyBuilder\"date-of-last-publish"

Drill down to the "Get Domain State" section and you'll get the following warnings:

Warning: Access control entry (ACE) Exchangeitup\RTCUniversalServerReadOnlyGroup; Allow; GenericRead; None; None
Warning: The access control entries (ACEs) on the object "users container" are not ready.
Warning: The access control entries (ACEs) on the domain "exchangeitup.com" are not ready.
Result: The domain is not ready.

Next, check the domain state in the SFB Management Shell, by running:

Get-CsAdDomain -Domain domain.com -Verbose

**Note** Change "domain.com" to your domain name.

You'll get the same errors, that the domain isn't ready:

Get-CsAdDomain -Domain exchangeitup.com -Verbose
VERBOSE: Creating new log file
"C:\Users\admin\AppData\Local\Temp\Get-CsAdDomain-10608f49-a1ea-477d-9f01-99fb610a450a.xml".
WARNING: Access control entry (ACE) Exchangeitup\RTCUniversalServerReadOnlyGroup; Allow; GenericRead; None; None
WARNING: The access control entries (ACEs) on the object "users container" are not ready.
WARNING: The access control entries (ACEs) on the domain "exchangeitup.com" are not ready.
WARNING: The domain is not ready.
LC_DOMAINSETTINGS_STATE_DISCOVERED, LC_DOMAINSETTINGS_STATE_ACCOUNTS_READY, LC_DOMAINSETTINGS_STATE_DOMAIN_ROOT_ACES_READY
VERBOSE: Creating new log file
"C:\Users\admin\AppData\Local\Temp\Get-CsAdDomain-10608f49-a1ea-477d-9f01-99fb610a450a.html".
WARNING: "Get-CsAdDomain" processing has completed with warnings. "4" warnings were recorded during this run.
WARNING: Detailed results can be found at
"C:\Users\admin\AppData\Local\Temp\Get-CsAdDomain-10608f49-a1ea-477d-9f01-99fb610a450a.html".

You'll also notice that, if you run the Deployment Wizard, it will show "Partial" for the domain readiness state:

Deployment Wizard Domain Partial


To fix it, we need to run "Enable-CsAdDomain -Domain domain.com"

**Note** Your admin account must be a member of Schema Admins in order to run the Enable-CsAdDomain cmdlet.

After it finishes (which takes all of 5 seconds), verify by running the Get-CsAdDomain cmdlet again...you should be error free and able to publish the topology now.

Get-CsAdDomain -Domain exchangeitup.com -Verbose
VERBOSE: Creating new log file
"C:\Users\admin\AppData\Local\Temp\Get-CsAdDomain-57c6af1d-610c-46cf-b1fe-3159c28f1429.xml".
LC_DOMAINSETTINGS_STATE_READY
VERBOSE: Creating new log file
"C:\Users\admin\AppData\Local\Temp\Get-CsAdDomain-57c6af1d-610c-46cf-b1fe-3159c28f1429.html".
VERBOSE: "Get-CsAdDomain" processing has completed successfully.
VERBOSE: Detailed results can be found at
"C:\Users\admin\AppData\Local\Temp\Get-CsAdDomain-57c6af1d-610c-46cf-b1fe-3159c28f1429.html".

**Note** Running Domain Prep seems to scare a lot of admins (we had to submit a change request and wait a week for it to be approved), because of the "extend schema" part. 
If you have previously run domain prep, running it again will not do anything to the schema, as it's already been extended. To see it in action, you can check the Enable-CsAdDomain logs in C:\Users\"youruser"\AppData\Local\Temp\Get-CsAdDomain-10608f49-a1ea-477d-9f01-99fb610a450a.html" and you'll see that all it does is set the required permissions.

Domain Prep Results

Now, publish your topology in the Topology Builder (you can publish it with no changes to test) and it will come up clean with no errors, and run the Deployment Wizard and it will show the domain is ready:

Deployment Wizard Domain Ready

The last step is: tell whoever deleted the permissions to READ THE DESCRIPTION IN AD BEFORE DELETING THINGS! Yes, I'm yelling.

Friday, October 25, 2019

Exchange SpamTitan Dynamic Recipient Verification With Edge Server

You may be reading the title of this post and wonder "why would you have an Edge Server and a spam filter?". Funny story: Well, not so funny. Back during planning of my Exchange environment, it was suggested by a Lotus Notes admin (who knows nothing about Exchange) that we should have an Edge. I (the Exchange admin) said "no, that's overkill". Well, my advice wasn't taken and we installed the Edge. The "funny" part: 3 years later I was asked "can we remove the Edge?" Come on! Of course not, it's there, it's staying put!

SpamTitan will do Recipient Verification to automatically bounce incoming messages addressed to invalid recipients, which can cut down on the amount of spam quite a bit; it also uses Recipient Verification for licensing. 
That last piece is important if you only pay for say, 1000 users, but the appliance sees 1,500 recipients.

Dynamic Recipient Verification (DRV) is the easiest to manage - there are other options like LDAP and manual import, but no one wants to mess with that. 
To use DRV, we need to enable Recipient Filtering on our Exchange environment. SpamTitan will then check with Exchange to verify if a recipient is valid or not, and either pass it through or bounce it.

SpamTitan has the following guide to enable DRV, but it only applies to Mailbox Servers, it doesn't cover if you're running an Edge Server.

https://helpdesk.spamtitan.com/support/solutions/articles/4000003763-dynamic-recipient-verification-using-exchange-2013-and-2016

The guide requires your Mailbox Servers to be internet-facing and involves allowing anonymous authentication on the Default Hub Transport connector on port 2525.
Edge Servers don't have a Default Hub Transport Connector, so there's no way to allow connections on port 2525.

I'll show you how to enable Dynamic Recipient Verification when you have an Edge in play.

First, enable Recipient Filtering:

On your Edge Server, in the Exchange Management Shell (EMS), run the following cmdlet to check our Recipient Filtering settings:

Get-RecipientFilterConfig

This should show the default settings, if you've never enabled any recipient filtering. The main thing to look for will be:

Name                                      : RecipientFilterConfig
RecipientValidationEnabled  : False
Enabled                                  : True

We'll enable Recipient Filtering with the following cmdlet:

Set-RecipientFilterConfig -RecipientValidationEnabled $true

Now, run the first cmdlet, and you'll see it enabled and ready:

Name                                    : RecipientFilterConfig
RecipientValidationEnabled  : True
Enabled                                 : True

Next, we'll enable our Accepted Domain(s) for Address Book recipient lookup:

Run the following cmdlet to check if Address Book is enabled:

Get-AcceptedDomain | FL Name,AddressBookEnabled

If it isn't enabled, run the following for each Authoritative Accepted Domain:


Set-AcceptedDomain "name of accepted domain" -AddressBookEnabled $true
**Note** Change "name of accepted domain" to your Accepted Domain.

**Note** You only want to set this on Authoritative Domains, not Relay domains.

Next, configure Dynamic Recipient Verification on the SpamTitan:

Browse to Setup > Mail Relay > Domains.  

Edit your domain(s) by clicking the "Pencil" icon, and select Dynamic Recipient Verification from the drop-down menu. 

Enter your Edge Server IP or host name (this will usually match your Destination Server):

SpamTitan Edit Domain

Click Save, and you'll see that DRV is enabled for your domains:

SpamTitan DRV Enabled

And lastly, your license will reflect the correct recipient count:


SpamTitan License Count

**Note** When Recipient Verification is not enabled, you'll get a warning that you have exceeded your recipient count. If that happens, you can reset the count (up to 3 times) and then wait a day or so and DRV will show the correct count.

Now you can test by sending an external message to a bunk internal email address and watch it bounce:


SpamTitan Bounce









The response from the remote server was:

550 5.1.1 <stace@exchangeitup.com>: Recipient address rejected: undeliverable address: host 66.66.66.66[66.66.66.66] said: 550 5.1.10 RESOLVER.ADR.RecipientNotFound; Recipient not found by SMTP address lookup (in reply to RCPT TO command)

As you can, we got a bounce from SpamTitan, and an explanation from our Edge server saying the address is invalid.

**Note** You can test by using telnet, but a lot of organizations don't allow telnet to port 25, so we're using just a regular email test.

Now w're all set! License count looks good, and dirty spammers will be bounced when they try to guess recipients.

Saturday, August 24, 2019

Exchange Veeam Backup Error 0x800401fd: Unable To Truncate Exchange Logs

We use Veeam to back up our Exchange 2016 environment. It generally works ok; if I had my way (and upper management understood Exchange) I'd run backup-less since we have a DAG and all...but whatever.

Every so often Veeam throws errors, mostly dealing with VSS issues; see my old posts here and here.

Now, I just got a new one, where Veeam finishes the backup job, but couldn't truncate the Exchange Transaction logs, causing the log volumes to get pretty big.

Veeam Error:

Unable to truncate Microsoft Exchange transaction logs. Details: Failed to call RPC function 'Vss.FinishSnapshot': Error code: 0x800401fd. Failed to invoke func [FinishSnapshot]: Object is not connected to server.

Exchange Error:

On the Exchange DAG node that Veeam connects to, there's also the following Application Event 2034:

Log Name:      Application
 Source:        MSExchangeRepl
 Date:          8/16/2019 8:51:56 PM
 Event ID:      2034
 Task Category: Exchange VSS Writer
 Level:         Error
 Keywords:      Classic
 User:          N/A
 Computer:      MBX1.exchangeitup.com
 Description:
 The Microsoft Exchange Replication service VSS Writer (Instance c2f51bd6-b7b3-49e1-8e12-4ad329955268) failed with error FFFFFFFD when processing the backup completion event.


This means our VSS Writer on the Exchange server is malfunctioning.

To check for VSS errors, fire up the CMD on the Exchange server, and run:

vssadmin list writers

You'll most likely get the following:

Writer name: 'Microsoft Exchange Writer'
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {2e1494ca-6e06-47a8-af0e-f77213bfecaf}
State: [12] Failed
Last error: Retryable error


The Fix:

Luckily the fix is easy!

A lot of forums will say to restart the Exchange Store service. DO NOT DO THIS! That results in database dismounts, and client connectivity loss.

To "reset" the VSS writer properly, restart the Exchange Replication Service (You'll notice the above event references the Repl service). This will not cause any downtime, as its for the database replication across servers.

Open the EMS (Exchange Management Shell) on the Exchange server, and run the following cmdlet to stop the Replication Service:

Stop-Service MSExchangeRepl

Now start it with the following:

Start-Service MSExchangeRepl

Now, let's check the VSS Writers again, and you'll see the error cleared:

Writer name: 'Microsoft Exchange Writer'
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {12c870b7-b0b2-418c-88c0-3564025eb01c}
State: [1] Stable
Last error: No error


Re-run your backup, or let it run on the next schedule, and you should get a successful backup, with logs truncated!