Saturday, January 18, 2020

Skype For Business Planning Diagram: Hybrid Traffic Flow

By now, all of my readers should know that I like to draw pretty pictures for planning projects. Recently, I completed a Skype For Business Hybrid build out and I created a diagram in order for the client's firewall team to open ports for the Hybrid traffic flow.

SFB 2015 Hybrid Traffic Flow Overview

In the drawing below, we have the SFB Hybrid already set up, and the ports on the firewall are set to any/any - meaning, everything is wide open and traffic is flowing great. 

But, we need to lock it down to only the ports needed and we need to visualize to/from "what server/service" to/from what "O365 service".

I've also included a table in the drawing, of the on-prem services, the port, protocol and direction. This way our firewall team can easily cross-check with the diagram.

**Note** You'll see in the diagram, the ports required for Exchange Unified Messaging. This particular client was still using ExUM, before they transition to cloud voicemail...yes, you can still configure Exchange Online UM, but it requires SFB Hybrid.

SFB Hybrid Traffic Flow

To edit the diagram to fit your organization, you'll need to download the most excellent SFB stencils by PaulB

Once you have those downloaded, move them to C:\Users\Your-User\Documents\My Shapes

Then hop over to my Google Drive and grab the SFB-Office 365 Ports Traffic Flow.vsdx

Feel free to edit the drawing to fit your needs. Happy drawing!

Saturday, January 4, 2020

SFB Error: "Domain Not Ready" When Publishing Topology

Our Skype For Business environment has been running great for years and recently we went to add a SIP Trunk and the Topology Builder threw a bunch of errors about the domain not being ready.
This was very odd, since prepping the domain is the first step when installing SFB, and as I said, was done years ago.

From the looks of it, someone (another admin with too much power and little SFB knowledge) deleted a bunch of permissions...most likely the permissions for the various RTC and CS groups that are required on several AD OU's.

The check the full errors, click view logs in the topology builder results, or navigate to C:\Users\"yourusername"\AppData\Local\Temp\2\TopologyBuilder\"date-of-last-publish"

Drill down to the "Get Domain State" section and you'll get the following warnings:

Warning: Access control entry (ACE) Exchangeitup\RTCUniversalServerReadOnlyGroup; Allow; GenericRead; None; None
Warning: The access control entries (ACEs) on the object "users container" are not ready.
Warning: The access control entries (ACEs) on the domain "exchangeitup.com" are not ready.
Result: The domain is not ready.

Next, check the domain state in the SFB Management Shell, by running:

Get-CsAdDomain -Domain domain.com -Verbose

**Note** Change "domain.com" to your domain name.

You'll get the same errors, that the domain isn't ready:

Get-CsAdDomain -Domain exchangeitup.com -Verbose
VERBOSE: Creating new log file
WARNING: Access control entry (ACE) Exchangeitup\RTCUniversalServerReadOnlyGroup; Allow; GenericRead; None; None
WARNING: The access control entries (ACEs) on the object "users container" are not ready.
WARNING: The access control entries (ACEs) on the domain "exchangeitup.com" are not ready.
WARNING: The domain is not ready.
VERBOSE: Creating new log file
WARNING: "Get-CsAdDomain" processing has completed with warnings. "4" warnings were recorded during this run.
WARNING: Detailed results can be found at

You'll also notice that, if you run the Deployment Wizard, it will show "Partial" for the domain readiness state:

Deployment Wizard Domain Partial

To fix it, we need to run "Enable-CsAdDomain -Domain domain.com"

**Note** Your admin account must be a member of Schema Admins in order to run the Enable-CsAdDomain cmdlet.

After it finishes (which takes all of 5 seconds), verify by running the Get-CsAdDomain cmdlet again...you should be error free and able to publish the topology now.

Get-CsAdDomain -Domain exchangeitup.com -Verbose
VERBOSE: Creating new log file
VERBOSE: Creating new log file
VERBOSE: "Get-CsAdDomain" processing has completed successfully.
VERBOSE: Detailed results can be found at

**Note** Running Domain Prep seems to scare a lot of admins (we had to submit a change request and wait a week for it to be approved), because of the "extend schema" part. 
If you have previously run domain prep, running it again will not do anything to the schema, as it's already been extended. To see it in action, you can check the Enable-CsAdDomain logs in C:\Users\"youruser"\AppData\Local\Temp\Get-CsAdDomain-10608f49-a1ea-477d-9f01-99fb610a450a.html" and you'll see that all it does is set the required permissions.

Domain Prep Results

Now, publish your topology in the Topology Builder (you can publish it with no changes to test) and it will come up clean with no errors, and run the Deployment Wizard and it will show the domain is ready:

Deployment Wizard Domain Ready

The last step is: tell whoever deleted the permissions to READ THE DESCRIPTION IN AD BEFORE DELETING THINGS! Yes, I'm yelling.

Friday, October 25, 2019

Exchange SpamTitan Dynamic Recipient Verification With Edge Server

You may be reading the title of this post and wonder "why would you have an Edge Server and a spam filter?". Funny story: Well, not so funny. Back during planning of my Exchange environment, it was suggested by a Lotus Notes admin (who knows nothing about Exchange) that we should have an Edge. I (the Exchange admin) said "no, that's overkill". Well, my advice wasn't taken and we installed the Edge. The "funny" part: 3 years later I was asked "can we remove the Edge?" Come on! Of course not, it's there, it's staying put!

SpamTitan will do Recipient Verification to automatically bounce incoming messages addressed to invalid recipients, which can cut down on the amount of spam quite a bit; it also uses Recipient Verification for licensing. 
That last piece is important if you only pay for say, 1000 users, but the appliance sees 1,500 recipients.

Dynamic Recipient Verification (DRV) is the easiest to manage - there are other options like LDAP and manual import, but no one wants to mess with that. 
To use DRV, we need to enable Recipient Filtering on our Exchange environment. SpamTitan will then check with Exchange to verify if a recipient is valid or not, and either pass it through or bounce it.

SpamTitan has the following guide to enable DRV, but it only applies to Mailbox Servers, it doesn't cover if you're running an Edge Server.


The guide requires your Mailbox Servers to be internet-facing and involves allowing anonymous authentication on the Default Hub Transport connector on port 2525.
Edge Servers don't have a Default Hub Transport Connector, so there's no way to allow connections on port 2525.

I'll show you how to enable Dynamic Recipient Verification when you have an Edge in play.

First, enable Recipient Filtering:

On your Edge Server, in the Exchange Management Shell (EMS), run the following cmdlet to check our Recipient Filtering settings:


This should show the default settings, if you've never enabled any recipient filtering. The main thing to look for will be:

Name                                      : RecipientFilterConfig
RecipientValidationEnabled  : False
Enabled                                  : True

We'll enable Recipient Filtering with the following cmdlet:

Set-RecipientFilterConfig -RecipientValidationEnabled $true

Now, run the first cmdlet, and you'll see it enabled and ready:

Name                                    : RecipientFilterConfig
RecipientValidationEnabled  : True
Enabled                                 : True

Next, we'll enable our Accepted Domain(s) for Address Book recipient lookup:

Run the following cmdlet to check if Address Book is enabled:

Get-AcceptedDomain | FL Name,AddressBookEnabled

If it isn't enabled, run the following for each Authoritative Accepted Domain:

Set-AcceptedDomain "name of accepted domain" -AddressBookEnabled $true
**Note** Change "name of accepted domain" to your Accepted Domain.

**Note** You only want to set this on Authoritative Domains, not Relay domains.

Next, configure Dynamic Recipient Verification on the SpamTitan:

Browse to Setup > Mail Relay > Domains.  

Edit your domain(s) by clicking the "Pencil" icon, and select Dynamic Recipient Verification from the drop-down menu. 

Enter your Edge Server IP or host name (this will usually match your Destination Server):

SpamTitan Edit Domain

Click Save, and you'll see that DRV is enabled for your domains:

SpamTitan DRV Enabled

And lastly, your license will reflect the correct recipient count:

SpamTitan License Count

**Note** When Recipient Verification is not enabled, you'll get a warning that you have exceeded your recipient count. If that happens, you can reset the count (up to 3 times) and then wait a day or so and DRV will show the correct count.

Now you can test by sending an external message to a bunk internal email address and watch it bounce:

SpamTitan Bounce

The response from the remote server was:

550 5.1.1 <stace@exchangeitup.com>: Recipient address rejected: undeliverable address: host[] said: 550 5.1.10 RESOLVER.ADR.RecipientNotFound; Recipient not found by SMTP address lookup (in reply to RCPT TO command)

As you can, we got a bounce from SpamTitan, and an explanation from our Edge server saying the address is invalid.

**Note** You can test by using telnet, but a lot of organizations don't allow telnet to port 25, so we're using just a regular email test.

Now w're all set! License count looks good, and dirty spammers will be bounced when they try to guess recipients.

Saturday, August 24, 2019

Exchange Veeam Backup Error 0x800401fd: Unable To Truncate Exchange Logs

We use Veeam to back up our Exchange 2016 environment. It generally works ok; if I had my way (and upper management understood Exchange) I'd run backup-less since we have a DAG and all...but whatever.

Every so often Veeam throws errors, mostly dealing with VSS issues; see my old posts here and here.

Now, I just got a new one, where Veeam finishes the backup job, but couldn't truncate the Exchange Transaction logs, causing the log volumes to get pretty big.

Veeam Error:

Unable to truncate Microsoft Exchange transaction logs. Details: Failed to call RPC function 'Vss.FinishSnapshot': Error code: 0x800401fd. Failed to invoke func [FinishSnapshot]: Object is not connected to server.

Exchange Error:

On the Exchange DAG node that Veeam connects to, there's also the following Application Event 2034:

Log Name:      Application
 Source:        MSExchangeRepl
 Date:          8/16/2019 8:51:56 PM
 Event ID:      2034
 Task Category: Exchange VSS Writer
 Level:         Error
 Keywords:      Classic
 User:          N/A
 Computer:      MBX1.exchangeitup.com
 The Microsoft Exchange Replication service VSS Writer (Instance c2f51bd6-b7b3-49e1-8e12-4ad329955268) failed with error FFFFFFFD when processing the backup completion event.

This means our VSS Writer on the Exchange server is malfunctioning.

To check for VSS errors, fire up the CMD on the Exchange server, and run:

vssadmin list writers

You'll most likely get the following:

Writer name: 'Microsoft Exchange Writer'
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {2e1494ca-6e06-47a8-af0e-f77213bfecaf}
State: [12] Failed
Last error: Retryable error

The Fix:

Luckily the fix is easy!

A lot of forums will say to restart the Exchange Store service. DO NOT DO THIS! That results in database dismounts, and client connectivity loss.

To "reset" the VSS writer properly, restart the Exchange Replication Service (You'll notice the above event references the Repl service). This will not cause any downtime, as its for the database replication across servers.

Open the EMS (Exchange Management Shell) on the Exchange server, and run the following cmdlet to stop the Replication Service:

Stop-Service MSExchangeRepl

Now start it with the following:

Start-Service MSExchangeRepl

Now, let's check the VSS Writers again, and you'll see the error cleared:

Writer name: 'Microsoft Exchange Writer'
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {12c870b7-b0b2-418c-88c0-3564025eb01c}
State: [1] Stable
Last error: No error

Re-run your backup, or let it run on the next schedule, and you should get a successful backup, with logs truncated!

Saturday, August 17, 2019

Exchange 2016 EMS Error: Could Not Load Microsoft.Exchange.CabUtility.dll

A few months ago, I upgraded my Exchange 2016 Servers to CU12, which included a Management Tools box that I run so other "non-Exchange" admins don't connect directly my Exchange machines.

The CU12 install was successful, but I just noticed that the EMS (Exchange Management Shell) on the Management Tools server threw an error when remoting to the Mailbox Servers. It was complaining that it couldn't find some required files; namely "Microsoft.Exchange.CabUtility.dll".

The Error:

Exception calling "EnsureTargetTypesLoaded" with "2" argument(s): "Could not load file or assembly
 'Microsoft.Exchange.CabUtility.dll' or one of its dependencies. The specified module could not be found."
 At C:\Program Files\Microsoft\Exchange Server\V15\bin\RemoteExchange.ps1:79 char:1
 + $typeLoadResult = [Microsoft.Exchange.Configuration.Tasks.CmdletAssemblyHelper]: ...

    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
     + FullyQualifiedErrorId : FileNotFoundException

 C:\Program Files\Microsoft\Exchange Server\V15\bin\RemoteExchange.ps1 : The Exchange types file wasn't loaded because not all of the required files could be found.
 At line:1 char:1
 + . 'C:\Program Files\Microsoft\Exchange Server\V15\bin\RemoteExchange.ps1'; Conne …

After some digging around on the server, I noticed that Visual C++ Redistributable for Visual Studio 2012 wasn't installed - Exchange 2016 CU12 requires Visual Studio 2012 and 2013.

What's even stranger: Visual Studio 2012 was installed prior to installing Visual Studio 2013 during the CU12 install...does 2013 overwrite 2012? I dunno, but this very same thing (Visual Studio 2012 missing) happened on my Edge server, which is how I knew to look there.

Again, this was strange since the upgrade finished with no errors. You'd think if something was missing, the install would bomb out...shrug.

The Fix:

Close the EMS.

Install Visual C++ Redistributable for Visual Studio 2012 Update 4 from here.

Fire up the EMS...no more errors!

Sunday, June 23, 2019

Exchange Inbox Rule To Move Email To Folder Sent To Alias

Here's a scenario I've seen quite often:

You have a Shared Mailbox called Human Resources (hr@exchangeitup.com), and you've added an alias of training@exchangeitup.com.

One of the user's with Full Access to the HR Mailbox, wants to create an inbox rule that will move messages sent to the training address to a Training folder - but this won't work.

When messages are sent to an alias, Exchange Directory Services will "translate" the alias as the primary email address; Training = Human Resources, and you can see it action by typing the address in the "To" field in Outlook and watch it change to the primary name. Therefore, inbox rules won't work for sorting/moving those emails to the alias.

In order to get around that behavior, we'll need to create a Mail Contact and a Transport Rule:

1. Remove the alias from the Shared Mailbox by running the following cmdlet in the Exchange Management Shell (EMS):

Set-Mailbox "Human Resources" -EmailAddresses @{remove="training@exchangeitup.com"}

Give it a few for Exchange/AD to update.

2. Create a Mail Contact for "Training" by running the following cmdlet:

New-MailContact -Name "Training" -ExternalEmailAddress training@exchangeitup.com -OrganizationalUnit “Contacts” 

**Note** This is re-using the secondary email address taken away from shared mailbox in step 1. Change the Name, External Email Address, OU match your environment.

3. Create a Transport Rule to redirect all messages sent to the Training Contact over to the HR Shared Mailbox:

 In the Exchange Admin Center (EAC), navigate to mail flow > rules.

Create a new Rule with the following settings:

Name: Training Forward

*Apply this rule if...
The recipient is..."Training"

*Do the following...
Redirect the message to..."Human Resources"

Or  you can use the EMS to run the cmdlet:

New-TransportRule, parameters -SentTo ("Training") -RedirectMessageTo ("Human Resources") -Name "Training Forward" -StopRuleProcessing "False" -Mode "Enforce" -Comments "" -RuleErrorAction "Ignore" -SenderAddressLocation "Header"

4. Create an inbox rule for the HR Shared Mailbox to move emails sent to the Training Contact to another folder in the Shared Mailbox:

In OWA, click your person icon in the upper-right corner, then select Open Another Mailbox.

Search for and open the Shared Mailbox (Human Resources in this example).

Then click the Settings Gear in the upper-right and go to Options > Inbox and sweep rules > create a rule using the following settings

It was sent to..."Training"
Do all of the following:
Move the message to folder..."Training Folder"

Now test it out! Send a message to training@exchangeitup.com and verify that the message was moved directly to the Training Folder.

Saturday, May 25, 2019

Exchange Forced TLS With SpamTitan

The growing trend nowadays, especially when sending messages to banks, is to use Forced TLS (also called Mandatory TLS). On top of TLS, most banks are now also requiring that the sending host use a public certificate to verify "you are who you say you are" - this is generally done with SPF records, but security requirements are getting more stringent.

If you're sending directly from Exchange and not using a spam filter (which probably isn't very common these days) you would just set up a Partner Send Connector and be done - except for assigning a cert to the connector, which is pretty tricky.
Since most organizations do use some sort of outgoing appliance (like SpamTitan), this post will walk you through setting up Forced TLS from Exchange on out through that gateway.

In my environment, we receive messages through a SpamTitan filter (which is awesome), but we send out through a Totemo encryption gateway...ever heard of it? Didn't think so. My short review on that product: It's garbage.
I was forced by company higher-ups to use it, going against my recommendations because it's not good practice to send and receive through different hosts/IPs - Totemo doesn't receive mail and it doesn't work very well...it runs on Java, which should tell you all you need to know. But it uses the "encrypt" button in Outlook to make users feel better, which adds complexity and results in help tickets...daily.
So, I decided to use the SpamTitan for outgoing TLS connections and it works perfectly!

Exchange Send Connector

We'll need to create a special Send Connector on Exchange to use the SpamTitan as a Smart Host.

**Note** Even if you're already using something like SpamTitan as your smart host on your current Send Connectors, I find it cleaner to create a dedicated connector for TLS - this way you can manage the domains requiring TLS without affecting your regular mailflow.

In the EAC (Exchange Admin Center) navigate to Mail Flow > Send Connectors. Click the "+" to create a new one:

New Send Connector
Give the Send Connector a name like "Google - TLS" and choose "Custom":

Custom Send Conenctor

**Note** We don't need a partner connector in this case, because mail will flow through our smart host not directly to our recipients.

Choose "Route mail through smart hosts", click the "+" and add your SpamTitan public IP address. Then click Next:

Send Connector Smart Host

There will be no authentication, so leave "None" selected and click Next:

Send Connector No Auth

In the Address space, click the "+" and enter the domains that you will sending over TLS to. If you want to include subdomains (which you always should), put an asterisk in front of the domain name. For instance to send to all of google/gmail, you'll enter *.google.com. Click Next when you've entered the domains:

Send Connector Address Space

In the Source server, click the "+" and choose your Edge Server(s) if you have them, otherwise choose your internet-facing Mailbox Server(s) and click Finish:

Send Connector Source Server

Your Send Connector is now created. Double-click it and change logging to "Verbose" and click Save. This will allow us to diagnose any issues later on by checking protocol logs, should TLS not work correctly:

Send Connector Logging

SpamTitan SSL Settings

On the SpamTitan WebUI, we need generate a CSR (Certificate Signing Request) so we can assign a public cert to send and receive with.

On your SpamTitan appliance, navigate to Settings > SSL. In the "Generate CSR" section, fill out the company info according to values below:

Common Name: The FQDN of your SpamTitan host
Organization: Your company name
OU: Optional
City/State/Country: The city, state and country your company is based in.

Then, hit "Run" under the "Generate Certificate Signing Request (CSR)" section.
SpamTitan CSR

Once the cert request is generated, copy the entire code block including "BEGIN CERTIFICATE REQUEST--- to ---END CERTIFICATE REQUEST---" and give that to whoever requests your certs from a Public CA.

When you get your cert generated and sent back to you in .pem format, import it into the SpamTitan under the "Import Certificates" section.

Under "Import Certificate from PEM" click "Choose File". Select your cert.pem file, and click "Import":

SpamTitan Cert import

You can view your imported cert under "Installed Signed Certificates" by clicking the magnifying glass icon:

SpamTitan Cert Properties

You'll then be presented with the cert details to make sure it's correct:

SpamTitan Cert

SpamTitan TLS Settings

Now that we have the cert in place and SSL enabled, we need to enable and configure TLS on the SpamTitan.

Navigate to Settings > TLS and click "Enable" for TLS. Select your new public cert, and enable logging:

SpamTitan TLS

**Note** This will also enable TLS for receiving and assign that cert for outside senders who need to verify your mail system (some banks require certs on both sending and receiving ends).

Next, navigate to System Setup > Mail Relay > Outbound.

Under "Trusted Networks" we'll add our Exchange external IP address(es). In my case it's my Edge server, but this could also be your internet-facing Mailbox Server(s):

SpamTitan Trusted Network

Under "TLS Encryption" we'll enable TLS, and in the drop-down choose "Use TLS for specified domains and Opportunistic TLS for all other connections". We do this because we don't want to force TLS on hosts that don't support it, which will drop connections:

SpamTitan TLS Encryption

Next, we'll add our recipient domains (these will match the domains we added in our Exchange Send Connector) with the following settings:

Domain: the top-level domain name
Include sub-domains: Yes
Policy: Mandatory TLS Encryption (aka Forced)
Allowed SSL/TLS Protocols: TLSv1/TLSv1.2 (these are most commonly supported)
Comment: Optional

SpamTitan TLS Domain

The domain will now be listed:

SpamTitan TLS Domain Listed
Under "Send Client Certificate" choose "enable" and verify that your public cert is selected and click "Save":

SpamTitan TLS Cert

Now, send a message to your gmail address (if you have one) or send a message to your external domain contact and verify that TLS was enforced by checking the receiving email header. You'll see the TLS version used to send the message:

TLS Verification

All done! Now you'll be sending secure messages!

Saturday, May 11, 2019

Exchange Transport Rule Block Attachments While Allowing Specific Ones

In my Exchange environment we have an invoicing mailbox that auto-forwards to an external document processing email address. This mailbox can only accept PDF or TIFF attachments, since that's what the doc processing service will accept. So, I needed a way to only allow those two attachment types, while blocking all others.

Articles on the Web are spotty on how to accomplish this...most are all or nothing. MS TechNet has an article on blocking all attachments, but we need to allow the ones mentioned above to come through. In this post, I'll expand on that.

We'll be using the EAC (Exchange Admin Center) to create our Transport Rule (aka Mail Flow Rules) since it's a little bit easier to see all of our options.

In the EAC, navigate to Mail Flow > Rules.
Click the "+" and select "Create a new rule...":

EAC Create New Rule

In the rule creator, give it a name.

Under "*Apply this rule if..." select "the recipient is..." and then search for and select your recipient(s).

Click the "more options" link and then click "add condition".

Drop-down to "any attachment" then mouse over to "is greater than or equal to..."

Input "1" in the field - for 1KB.

**Note** The EAC will only accept 1KB as the lowest value, which will block all attachments in my experience. If you need to go lower, like 0KB, you'll need to use the Shell to set it later.

Under "*Do the following..." drop-down to "Reject the message with the explanation..." and enter your outgoing bounce reason. In my case I specify that only PDF and TIFF are allowed.

Under "Except if..." drop-down to "Any attachment's file extension matches..." and enter "tiff" and "pdf".

**Note** Don't enter a dot in front of the file extensions, as it will throw an error; only enter the extension name.

Set your priority and leave "enforced" selected.

Click "Save"

The results should look like so:

EAC Attachment Rule Settings

Basically, what we did is block all attachments, but since we set an exception, we allowed those two specific ones through.
This might seem counterintuitive at first, but think if you did it the other way around and had to manually block every file type...that would take forever!

**Note** In the above example I set the rule for one mailbox, but if you need to allow only these certain attachments to all mailboxes, under "Apply this rule if..." you would choose "the recipient is... located inside the organization". You can also set it to "a member of this group" if you have groups set up for different settings.

Now, test your rule by sending a blocked attachment like a JPEG, to the mailbox and then send an allowed attachment like a PDF.

The blocked attachment should generate a 5.7.1 reply like so:

Delivery has failed to these recipients or groups:

invoices@exchangeitup.com (invoices@exchangeitup.com)

This message was rejected because only PDF or TIFF attachment types are allowed.

Your message wasn't delivered because the email admin for the organization 'exchangeitup.com' created an email rule restriction. Please contact the email admin for that organization and ask them to remove or update the rule restriction.
For more information about this error, see DSN code 5.7.1 in Exchange Online - Office 365.

As you can see, the sender will get the 571 bounce, and it includes my custom outgoing message under the recipient email address: "This message was rejected because only PDF or TIFF attachment types are allowed."

Happy attachment blocking!