Saturday, December 29, 2018

Exchange 2016 Create Remote PowerShell Shortcut

A growing trend for organizations is to lock down direct remote access to servers, requiring the need for Remote PowerShell Sessions to manage things like Exchange. The problem is, it's a nuisance to create a remote session every time you need to connect to Exchange.

To save time (and my sanity) I decided to create a shortcut that will start the remote session automatically when you run it.

Copy the following information into Notepad:

$UserCredential = Get-Credential
$Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri http://mail.exchangeitup.com/PowerShell/ -Authentication Kerberos -Credential $UserCredential
Import-PSSession $Session -DisableNameChecking

**Note** Change -ConnectionURI to match your Exchange namespace or a server name (if you don't publish PowerShell).

Save it as a .ps1 with a name like Exchange-RemotePS.ps1 to somewhere like C:\Scripts

Next, create a PowerShell shortcut anywhere, like on your Desktop:

Right-click the Desktop > New > Shortcut

New Shortcut
In the location field, enter:
PowerShell Shortcut
Click Next
Give it a name like ExchangeRemotePS and click Finish.
ExchangeRemotePS Shortcut

Right-click the new ExchangeRemotePS shortcut, and go to Properties.
In the Target field, add the following to the end of the line:
-NoExit -File "C:\Scripts\Exchange-RemotePS.ps1"
It will look like so:
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoExit -File "C:\Scripts\Exchange-RemotePS.ps1"
Shortcut Target
Click OK.
Run your new shortcut, and you'll get a creds prompt for your Exchange organization.
Once it starts the remote session, you'll be able to run your Exchange cmdlets.
Remote PowerShell Session
Now, you can use that new shortcut instead of tiring out your fingers before even getting into Exchange!

Sunday, December 2, 2018

Exchange 2016 Planning Diagram: Hybrid

If you follow my blog, you know that I like to use Visio to draw pretty pictures to use when planning out Messaging Environments. Examples can be found here, here and here. The diagrams really come in handy when presenting a build to a new client, or to your higher-ups so they can see what the finished setup will look like.

I just happen to be planning out an Exchange 2016 Hybrid scenario, which will use the current on-premises Exchange environment and expand it out to Office 365. What a great chance to create a new diagram!

Feel free to grab the Visio diagram further down in the post and edit it to match your environment, in order to save time in creating one yourself.

Exchange 2016 Hybrid Planning Overview

In my example below, we're starting with a 3-node Exchange 2016 DAG with the accepted domain of "domain.com" and "mail.domain.com" as the namespace.

We have a two-arm Load Balancer (one NIC or "arm" in the DMZ, and one in the Internal LAN) which will server up mail.domain.com for our users that have mailboxes homed on-prem.

An on-premises Skype For Business Front End (SFB1.domain.com), with Unified Messaging going to the on-prem Exchange servers.

We're going to expand the on-prem Exchange out into Office 365, creating a Hybrid Environment.

This will require adding:

An Azure AD Connect Server for DirSync (AAD.domain.com)

An ADFS Server for authentication (ADFS.domain.com)

And installing the Hybrid Server on Exchange (hybrid.domain.com)

**Note** You'll want to edit the generic names of the domain to match your environment.

As you can see in the diagram, all the new servers are added, with the connection flow to/from Exchange and O365.

Exchange 2016 Hybrid Overview

To edit the diagram to fit your organization, you'll need to download the Exchange/Office 365 Visio stencils from Microsoft, so the shapes will render correctly.

Once you have those downloaded, move them to C:\Users\Your_User\Documents\My Shapes

Then, hop over to my Google Drive and grab the Exchange Hybrid Overview.vsdx

Happy diagramming :)

Sunday, October 28, 2018

SFB Invoke Failback Error: Cannot Open Service

We recently updated our Skype For Business environment to CU7 (July 2018 Update). Since we have such a large SFB organization (20+ servers) I and another admin tag-teamed the install so it wouldn't be an all day affair.
When it came time to run the Invoke-CSComputerFailback cmdlet, the other admin was hit with the following error in the Skype For Business Management Shell:

WARNING: Invoke-CSComputerFailback failed.
WARNING: Detailed results can be found at


Invoke-CSComputerFailback : Command execution failed: Cannot open RTCXMPPTGW service on computer "SFB-FE1.exchitup.com".
At line:1 char:1
+ Invoke-CSComputerFailback -Computer SFB-FE1.exchitup.com
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 + CatagoryInfo  : InvalidOperation: <:> [Invoke-CSComputerFailback], InvalidOperationException
 + FullyQualifiedErrorId : ProcessingFailed,Microsft.Rtc.Management.HADR.ComputerFailover.

If you browse to the location of the logfile in the error, the XML and HTML reports show the following, respectively:


<?xml version="1.0" encoding="UTF-8"?>
-<CsMgmtLog Name="Invoke-CsComputerFailBack">
<Info Time="2018-10-27 17:31:33Z" Title="Connection">Data Source=SFB-FE2.exchitup.com;Failover Partner=SFB-FE3.exchitup.com;Initial Catalog=xds;Integrated Security=True;Application Name=Microsoft.Rtc.Management</Info>
<Info Time="2018-10-27 17:31:37Z" Title="Host">SFB-FE1.exchitup.com</Info>
<Info Time="2018-10-27 17:31:37Z">Enabling services at SFB-FE1.exchitup.com...</Info>
<Error Time="2018-10-27 17:31:37Z" Title="Error">An error occurred: "System.InvalidOperationException" "Cannot open RTCXMPPTGW service on computer 'SFB-FE1.exchitup.com'."</Error>

-<Exception Time="2018-10-27 17:31:37Z" Message="Cannot open RTCXMPPTGW service on computer 'SFB-FE1.exchitup.com'." Type="InvalidOperationException">
<StackTrace Time="2018-10-27 17:31:37Z"> at System.ServiceProcess.ServiceController.GetServiceHandle(Int32 desiredAccess) at System.ServiceProcess.ServiceController.get_ServiceHandle() at Microsoft.Rtc.Management.Deployment.Core.NTService.set_Mode(StartMode value) at Microsoft.Rtc.Management.HADR.ComputerFailover.InvokeComputerFailBackCmdlet.EnableServices(DeploymentContext context) at Microsoft.Rtc.Management.HADR.ComputerFailover.InvokeComputerFailBackCmdlet.InternalProcessRecord() at Microsoft.Rtc.Management.Deployment.DeploymentCmdlet.CmdletProcessRecord()</StackTrace>

-<Exception Time="2018-10-27 17:31:37Z" Message="Access is denied" Type="Win32Exception">
<StackTrace Time="2018-10-27 17:31:37Z"/>


SFB HTML Error Log

The Fix:

Run the Skype For Business Management Shell as administrator, then  try the cmdlet again.

**Note** I always set the shortcut for the Shell to Run as Admin, so I don't have to do that every time. I wish it just would out of the box since I am, you know, an admin :)

Right-click shortcut, Properties > Advanced button > Run As Administrator > Ok

Now your cmdlets should run successfully and you can finish your updates!

Saturday, October 6, 2018

Exchange 2016 Publishing REST/OData For API With Kemp Load Balancer

Our company is designing an app for users to have an all-in-one-do-everything portal that will incorporate a mail client that connects to our on-prem Exchange. The app utilizes the Outlook Mail REST API to grab messages from the user's mailbox.

**Note** Office365 supports REST by default; it wasn't until Exchange 2016 CU3 that it was available for on-prem (go figure).

You can find the details here:


If you're running on-prem, you should be patched up way beyond CU3, right?

The issue is: REST API is enabled on Exchange 2016 CU3+ but if you run a load balancer, like Kemp's super-awesome Load Master, the API won't be published out of the box.

By default, the Kemp Exchange templates don't enable Exchange API to be published. As a workaround, you can enable the "default" content rule on your SubVSs, but some IT security departments won't like that, because it's a blanket rule.

What we'll do instead, is create a Custom Content Rule, and apply it to our OWA SubVSs ('Sub-Virtual Services), which will allow the API to be published.

Create The Rule:

In your Load Master WUI, go to Rules & Checking > Content Rules > Create New Rule:

Kemp Add New Rule 1

Give it a name like "REST_API"

Rule type: leave as default (Content Matching)

Match type: leave as default (Regular Expression)

Match string: type in the following:


Tick Ignore Case

Click Create Rule:

Kemp Add New Rule 2

Your new rule will be listed:

Kemp Rule Created

Adding The New Rule:

Next, we'll add our new content rule to our OWA SubVSs

Navigate to Virtual Services > View/Modify Services

Click Modify next to your Internal Service:

Kemp Modify VSs 1

**Note** You can modify your external service if you want; it would just depend on your security requirements. For instance, our IT Security Department doesn't understand Exchange and therefore won't allow me to publish OWA externally, so I can only publish the API internally as well.

Expand SubVSs, and click the Rules button next to OWA:

Kemp Modify VSs 2

In the drop-down, select our new "API_REST" rule and click Add:

Kemp Add Rule 1
Kemp Add Rule 2
You'll now see the API_REST rule listed for the OWA rule set:
Kemp Rule Added to VSs
Testing The New Rule:
In a web browser navigate to your Exchange namespace with the API URL at the end, like so:
You will be presented with a Save/Open JSON prompt:
Mailbox json prompt
Open in Notepad, and you'll see your messages in your mailbox:
Mailbox API Messages Content
Now your on-premises Exchange API REST/OData is published, and the app (or whatever needs that service) can hit it!

Saturday, September 1, 2018

Exchange 2016 Recover Deleted Items To Original Folder Script

I recently had a huge issue in my Exchange environment where a retention policy went haywire and wiped mail from all User Mailboxes in the organization.

Backstory: We migrated from Notes last year, and the legal department wanted to institute retention policies moving to Exchange. I (stupidly) let another admin tag those migrated messages with an EWS script (which BTW, I said from the beginning isn't the best way to do it, and of course no one listens me) but they did it anyway. Fast-forward a year later, and the tag from that script got removed because it was named "test" and Exchange never threw a warning saying that it was applied to any mailboxes, which un-tagged all mailboxes, which deleted years worth of mail. Yippee.

So I desperately needed a way to get those messages back; preferably to their original folders, in bulk, without manually going to each mailbox to recover items from the Recoverable Items folder (aka the dumpster).

Exchange on-prem doesn't have this capability (it should, Microsoft are you listening?) so I scoured the webs and finally found a solution! An EWS script that moves items from the Recoverable Items folder back to the original folder they were deleted from.

***Starting with Exchange 2016 CU1, items are stamped with a LastActiveParentEntryID (LAPEID) attribute, but it couldn't be utilized until CU6; and it can only be accessed from OWA, not from Outlook, not from PowerShell.

Credit goes to Scott at flobee.net, who wrote an AMAZING script that ports the Restore-RecoverableItems from O365 to on-prem (again, this should be available in the first place, not everyone wants to, or even can move to O365, Microsoft).

The original script can be found here

The problem is: it did not work all the way; running with more than one parameter, caused the script to result in no output.

What I mean is: if I ran the script with the "-FilterStartTime" and "-FilterEndTime" or "-FilterStartTime" and "-FilterItemType" it wouldn't do anything...I could only use one of those parameters.

So, I edited the script to use AQS (Advanced Query Syntax) and hardcoded a date range so it will run successfully.

Saving The Script:

Grab the RecoverableItemsAQS.psm1 from my Google Drive here.

**Note** It's a .psm1 file. That's a PowerShell module that you need to import into your PowerShell session, which I'll show you later.

Open the script in Notepad and search for AQS and change the date range in the following block; you'll need to do this in both the Get-MailboxRecoverableItems and Restore-RecoverableItems sections:

  $AQS = 'received:4/22/2015..6/7/2017'

Edit the date range to fit your needs.

Also, change the page size in both sections to 1000; the default is 50.

Search for:

$pageSize = 50

You will also need the EWS API 2.2 installed on the machine where you'll be running the script from. Best is to install this on a non-Exchange server as it can cause some serious performance issues on Exchange - see my previous post for those issues.

Running The Script:

After you've made the above changes, save the script somewhere like C:\Scripts

There are two functions of the script: Get Recoverable Items, and Restore Recoverable Items.

**Note** You cannot pipe the Get function into the Recover Function, it is only for reporting.

Get Recoverable Items:

**We're going to be using Impersonation to run against mailboxes. This is easier and more secure than setting Full Access on a bunch of mailboxes. You'll need to pass the credentials into the script for the account that has impersonate rights to run both Get and Recover.

To set up impersonation follow this TechNet guide.

In the EMS, run:

Import-Module "C:\Scripts\RecoverableItemsAQS.psm1"

Then to get a list of items in the dumpster output to a txt file, run:

Get-MailboxRecoverableItems -Identity staceybranham@exchangeitup.com -UseImpersonation -Credential (Get-Credential) | Out-File c:\temp\staceydeletions.txt

**Note** The -UseImpersonation and -Credential (Get-Credential) parameters are for your impersonation account you set up earlier. You'll be prompted with a Widows cred prompt.

Restore Recoverable Items For A Single User:

In the EMS, run:

Import-Module "C:\Scripts\RecoverableItemsAQS.psm1"

Next run the following to recover on a single mailbox:

Restore-MailboxRecoverableItems -Identity staceybranham@exchangeitup.com -UseImpersonation  -Credential (Get-Credential)

Restore Recoverable Items In Bulk:

To run the recovery on a multiple mailboxes, we'll pipe in a CSV list of email addresses, and run the script against each of those.

Create your CSV where the heading is SMTPAddress, then each user's email address on each row like so:


Save it as MailboxImport.csv somewhere like C:\Scripts.

Next in the EMS, run:

$cred = get-credential

**Note** These creds will be the account you're using that has Impersonation Rights

Then, run:

$mailboxes = Import-CSV "C:\Script\MailboxImport.csv"; $mailboxes | foreach {Restore-MailboxRecoverableItems -Identity $_.SMTPAddress -UseImpersonation -Credential $cred}

Use caution when running the script on a bunch of mailboxes at one time, because it generates transaction logs, just as regular mail-flow does.

Test with groups of mailboxes to see how big your log volumes are growing before you run the script on a large batch. In my experience, recovering ~9GB of deleted items generated 1 to 1.5GB of logs.

My advice is to split up your Import CSVs into 10-20 mailboxes per day (or per backup cycle) so the logs will truncate in between runs -or- stagger the groups of mailboxes which are homed on different DB's...say 2 or 3 mailboxes per DB in each list.

Saturday, August 25, 2018

Exchange 2016 Upgrade Pre-CU5 to CU9/CU10

I see quite a few questions on forums floating around about upgrading old versions of Exchange 2016 to the latest CUs. The cause for concern are the new requirements for .Net on Exchange.

For instance, Exchange CU9 requires either .Net Framework 4.6.2 or 4.7.1; Exchange CU10 requires .Net 4.7.1.

So here's the issue: how do you upgrade Exchange, when you're far out of date and update .Net without breaking your mail servers?

I've put together the following guide which will walk you through the process, and hopefully calm your fears; it's not as painful as you think!
I just completed this on a very non-standard Exchange environment, so it should work for you too.

1. In your monitoring solution, pause all Exchange servers and the Domain Controller holding the FMSO roles

a. Download .Net 4.7.1 from here:


b. Download Exchange 2016 CU9 from here:


**Note** Microsoft only hosts the current and N-1 CU's, so if CU9 is no longer available, leave a comment and I'll share it out with you.

b.1. If you're installing CU10, download it from here:


2. On The DC Holding FSMO

a. Install .Net 4.7.1

**Note** This *shouldn't* require a reboot.

b. In an Elevated CMD, run each of the following commands one-by-one:

setup.exe /PrepareSchema /IAcceptExchangeServerLicenseTerms

Setup.exe /PrepareAD /IAcceptExchangeServerLicenseTerms

setup.exe /PrepareDomain /IAcceptExchangeServerLicenseTerms

c. After Domain Preps, check replication to ensure all is well:

repadmin /showrepl

**Note** I normally run the AD prep commands from an Exchange server, but since CU9 requires .Net 4.6.2 -or- 4.7.1 and CU10 requires 4.7.1, it will halt the install saying that .Net 4.5 or higher is required. So, I chose to run it from the DC so as to not break Exchange (CU4, which is currently installed, doesn't support .Net 4.7.1 so this would cause conflicts).

3. On your First Mailbox Server

- Remove from Load Balancer Rotation

a. Start Exchange Server Maintenance Mode

These commands will stop Transport queues, stop DAG cluster, perform DB switchover, block DB activation, and enable maintenance mode

a.1. In the EMS, run each command separately:

$Computer = $ENV:ComputerName

Set-ServerComponentState $Computer -Component HubTransport –State Draining -Requester Maintenance

Redirect-Message -Server $Computer -Target MBX2.exchangeitup.com

Suspend-ClusterNode $Computer

Set-MailboxServer $Computer -DatabaseCopyActivationDisabledAndMoveNow $True

Set-MailboxServer $Computer -DatabaseCopyAutoActivationPolicy Blocked

Set-ServerComponentState $Computer -Component ServerWideOffline -State Inactive -Requester Maintenance

b. Install .Net 4.7.1

**Note** MS says to use newest .Net version supported by Exchange. A lot of the advice on forums will mislead you by saying to install 4.6.2 then install CU9, then install 4.7.1. Don't do that! This adds a whole lot of unnecessary steps; just go straight to the latest .Net.

b.1. **If you are installing CU10** Download and install Visual C++ (CU10 requires it) from here:


b.2. Reboot

c. Uninstall previous the CU Language Pack by running the following in an Elevated CMD:

cd C:\wherever-you-have-stored-your-language-pack

setup.exe /RemoveUMLanguagePack:de-DE

**Note** Change "RemoveUMLanguagePack:de-DE" to whatever language pack you have

d. Mount the CU9 or CU10 ISO and run setup.exe - Run As Administrator for good measure

**Note** This will take a long time, so plan for the whole day depending on how many servers you have.

**Note** During the CU9/CU10 readiness checks if the installer throws an error stating: setup can't continue with the upgrade because the mscorsvw (PID#) has open files, just give it some time. The problem is, the .Net compiler hasn't finished yet, which takes about 30 minutes. Open task manager and watch for .Net optimization; once it disappears, it's finished and you can hit Retry in the installer.

e. Install the newest CU Language Pack - You'll need to Run As Administrator

f. Stop Exchange Server Maintenance Mode

These commands will start Transport queues, start DAG cluster, unblock DB activation, disable maintenance mode, and restart Transport Services

f.1. In the EMS, run each command separately

$Computer = $ENV:ComputerName

Set-ServerComponentState $Computer -Component ServerWideOffline -State Active -Requester

Resume-ClusterNode $Computer

Set-MailboxServer $Computer -DatabaseCopyActivationDisabledAndMoveNow $False

Set-MailboxServer $Computer -DatabaseCopyAutoActivationPolicy Unrestricted

Set-ServerComponentState $Computer -Component HubTransport -State Active -Requester Maintenance

Restart-Service MSExchangeTransport

Restart-Service MSExchangeFrontEndTransport

Get-ServerComponentState MBX1 - ensure that all services are active

 **Note** If the Database copies say: Content index state:  HealthyAndUpgrading no need to worry, carry on with your upgrade - it could take days depending on the size of your DB's. It's still totally safe to mount or activate copies on the other DAG members during this operation.

4. Rinse and Repeat the above steps on each Mailbox Server

5. On Your First Edge Server

a. Start Exchange Server Maintenance Mode

These commands will stop Edge Transport queues, and enable maintenance mode

a.1. In the EMS, run each command separate

$Computer = $ENV:ComputerName

Set-ServerComponentState $Computer -Component ServerWideOffline -State Inactive -Requester Maintenance

b. Install .Net 4.7.1

b.1. Reboot

b.2. **If installing CU10** Install Visual C++

c. Run the Exchange CU setup from an Elevated CMD:

Setup.exe /mode:upgrade /IAcceptExchangeServerLicenseTerms

d. Stop Exchange Server Maintenance Mode

These commands will start Edge Transport queues, and disable maintenance mode

d.1. In the EMS, run each command separate

$Computer = $ENV:ComputerName

Set-ServerComponentState $Computer -Component ServerWideOffline -State Active -Requester Maintenance

6. Repeat for Each Edge

7. On your Management Tools Server

a. Install .net 4.7.1

a.1. **If installing CU10** Install Visual C++

b. Install the CU - the CU installer will upgrade the tools automatically - it detects the installed roles.

8. Cleanup/Checking our work

a. Check build numbers on Servers:

Get-ExchangeServer | fl name,edition,admindisplayversion

For CU10 you'll get: 15.01.1531

For CU9, you'll get: 15.01.1466

**Note** The Edge will not show the current build number when running the above cmdlet from a Mailbox Server; this is by design. If you want it to show the updated build, you would need to re-subscribe the EdgeSync.

b. Rebalance the DAG by running the following in the EMS:

cd exscripts

.\RedistributeActiveDatabases.ps1 -DagName DAG01 -BalanceDbsByActivationPreference -Confirm:$False

**Note** Change "DAG01" to the name of your DAG

Now your Exchange environment will be up to date and healthy. In my environment, I chose to go to CU9 instead of CU10 because I always stay N-1 on CU's; if you do follow the forums it seems that almost every other CU release causes something to break in Exchange...most have been riddled with bugs. I would personally rather be one step behind than have to open an MS ticket to fix something they broke ;)

Saturday, August 11, 2018

Exchange OWA TimeZoneException Error

Here's an easy one today: Users sometimes get the "Time Zone Exception" error below when logging in to OWA, and it all depends on the web browser their using:

X-OWA-Error: SDServerErr;Microsoft.Exchange.Services.Core.Types.TimeZoneException
X-OWA-Version: 15.1.1531.3
X-FEServer: MBX1
X-BEServer: mail.exchangeitup.com

I've never seen this error happen in IE, but very often in Chrome and Firefox. This is due to the way those two browsers implement adblock, which doesn't handle OWA very well.

We don't want to disable adblock completely, since it is useful, but we'll turn it off for OWA.

In Chrome:

Navigate to OWA, then click “Secure” to the left of the URL.

**Note** It might show an "i" info bubble instead of Secure, if you aren't redirecting OWA to SSL - which you really should do.

Find the “Ads” entry, then choose Allow in the menu.

Now, OWA won't be blocked, but other sites will still block ads as normal.

In Firefox:

Browse to OWA and click the shield icon next to the URL to bring up the Control Center.

In the Control Center, click Disable protection for this session.

Once Tracking Protection is disabled, a shield icon with red strikethrough (which means tracking protection is off) will appear in your address bar.

If you need to do this on a lot domain machines, there are GPO templates, but I haven't dove into them to see what you can control...there might be a way to set the adblock exclusions for OWA if you wanna check them out.

For Chrome, grab them here

For Firefox, grab them here

You could also just disallow those browsers and force IE, but that's up to you, and it of course wouldn't have any affect on non-domain computers.