Sunday, October 28, 2018

SFB Invoke Failback Error: Cannot Open Service

We recently updated our Skype For Business environment to CU7 (July 2018 Update). Since we have such a large SFB organization (20+ servers) I and another admin tag-teamed the install so it wouldn't be an all day affair.
When it came time to run the Invoke-CSComputerFailback cmdlet, the other admin was hit with the following error in the Skype For Business Management Shell:

WARNING: Invoke-CSComputerFailback failed.
WARNING: Detailed results can be found at


Invoke-CSComputerFailback : Command execution failed: Cannot open RTCXMPPTGW service on computer "SFB-FE1.exchitup.com".
At line:1 char:1
+ Invoke-CSComputerFailback -Computer SFB-FE1.exchitup.com
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 + CatagoryInfo  : InvalidOperation: <:> [Invoke-CSComputerFailback], InvalidOperationException
 + FullyQualifiedErrorId : ProcessingFailed,Microsft.Rtc.Management.HADR.ComputerFailover.

If you browse to the location of the logfile in the error, the XML and HTML reports show the following, respectively:


<?xml version="1.0" encoding="UTF-8"?>
-<CsMgmtLog Name="Invoke-CsComputerFailBack">
<Info Time="2018-10-27 17:31:33Z" Title="Connection">Data Source=SFB-FE2.exchitup.com;Failover Partner=SFB-FE3.exchitup.com;Initial Catalog=xds;Integrated Security=True;Application Name=Microsoft.Rtc.Management</Info>
<Info Time="2018-10-27 17:31:37Z" Title="Host">SFB-FE1.exchitup.com</Info>
<Info Time="2018-10-27 17:31:37Z">Enabling services at SFB-FE1.exchitup.com...</Info>
<Error Time="2018-10-27 17:31:37Z" Title="Error">An error occurred: "System.InvalidOperationException" "Cannot open RTCXMPPTGW service on computer 'SFB-FE1.exchitup.com'."</Error>

-<Exception Time="2018-10-27 17:31:37Z" Message="Cannot open RTCXMPPTGW service on computer 'SFB-FE1.exchitup.com'." Type="InvalidOperationException">
<StackTrace Time="2018-10-27 17:31:37Z"> at System.ServiceProcess.ServiceController.GetServiceHandle(Int32 desiredAccess) at System.ServiceProcess.ServiceController.get_ServiceHandle() at Microsoft.Rtc.Management.Deployment.Core.NTService.set_Mode(StartMode value) at Microsoft.Rtc.Management.HADR.ComputerFailover.InvokeComputerFailBackCmdlet.EnableServices(DeploymentContext context) at Microsoft.Rtc.Management.HADR.ComputerFailover.InvokeComputerFailBackCmdlet.InternalProcessRecord() at Microsoft.Rtc.Management.Deployment.DeploymentCmdlet.CmdletProcessRecord()</StackTrace>

-<Exception Time="2018-10-27 17:31:37Z" Message="Access is denied" Type="Win32Exception">
<StackTrace Time="2018-10-27 17:31:37Z"/>


SFB HTML Error Log

The Fix:

Run the Skype For Business Management Shell as administrator, then  try the cmdlet again.

**Note** I always set the shortcut for the Shell to Run as Admin, so I don't have to do that every time. I wish it just would out of the box since I am, you know, an admin :)

Right-click shortcut, Properties > Advanced button > Run As Administrator > Ok

Now your cmdlets should run successfully and you can finish your updates!

Saturday, October 6, 2018

Exchange 2016 Publishing REST/OData For API With Kemp Load Balancer

Our company is designing an app for users to have an all-in-one-do-everything portal that will incorporate a mail client that connects to our on-prem Exchange. The app utilizes the Outlook Mail REST API to grab messages from the user's mailbox.

**Note** Office365 supports REST by default; it wasn't until Exchange 2016 CU3 that it was available for on-prem (go figure).

You can find the details here:


If you're running on-prem, you should be patched up way beyond CU3, right?

The issue is: REST API is enabled on Exchange 2016 CU3+ but if you run a load balancer, like Kemp's super-awesome Load Master, the API won't be published out of the box.

By default, the Kemp Exchange templates don't enable Exchange API to be published. As a workaround, you can enable the "default" content rule on your SubVSs, but some IT security departments won't like that, because it's a blanket rule.

What we'll do instead, is create a Custom Content Rule, and apply it to our OWA SubVSs ('Sub-Virtual Services), which will allow the API to be published.

Create The Rule:

In your Load Master WUI, go to Rules & Checking > Content Rules > Create New Rule:

Kemp Add New Rule 1

Give it a name like "REST_API"

Rule type: leave as default (Content Matching)

Match type: leave as default (Regular Expression)

Match string: type in the following:


Tick Ignore Case

Click Create Rule:

Kemp Add New Rule 2

Your new rule will be listed:

Kemp Rule Created

Adding The New Rule:

Next, we'll add our new content rule to our OWA SubVSs

Navigate to Virtual Services > View/Modify Services

Click Modify next to your Internal Service:

Kemp Modify VSs 1

**Note** You can modify your external service if you want; it would just depend on your security requirements. For instance, our IT Security Department doesn't understand Exchange and therefore won't allow me to publish OWA externally, so I can only publish the API internally as well.

Expand SubVSs, and click the Rules button next to OWA:

Kemp Modify VSs 2

In the drop-down, select our new "API_REST" rule and click Add:

Kemp Add Rule 1
Kemp Add Rule 2
You'll now see the API_REST rule listed for the OWA rule set:
Kemp Rule Added to VSs
Testing The New Rule:
In a web browser navigate to your Exchange namespace with the API URL at the end, like so:
You will be presented with a Save/Open JSON prompt:
Mailbox json prompt
Open in Notepad, and you'll see your messages in your mailbox:
Mailbox API Messages Content
Now your on-premises Exchange API REST/OData is published, and the app (or whatever needs that service) can hit it!

Saturday, September 1, 2018

Exchange 2016 Recover Deleted Items To Original Folder Script

I recently had a huge issue in my Exchange environment where a retention policy went haywire and wiped mail from all User Mailboxes in the organization.

Backstory: We migrated from Notes last year, and the legal department wanted to institute retention policies moving to Exchange. I (stupidly) let another admin tag those migrated messages with an EWS script (which BTW, I said from the beginning isn't the best way to do it, and of course no one listens me) but they did it anyway. Fast-forward a year later, and the tag from that script got removed because it was named "test" and Exchange never threw a warning saying that it was applied to any mailboxes, which un-tagged all mailboxes, which deleted years worth of mail. Yippee.

So I desperately needed a way to get those messages back; preferably to their original folders, in bulk, without manually going to each mailbox to recover items from the Recoverable Items folder (aka the dumpster).

Exchange on-prem doesn't have this capability (it should, Microsoft are you listening?) so I scoured the webs and finally found a solution! An EWS script that moves items from the Recoverable Items folder back to the original folder they were deleted from.

***Starting with Exchange 2016 CU1, items are stamped with a LastActiveParentEntryID (LAPEID) attribute, but it couldn't be utilized until CU6; and it can only be accessed from OWA, not from Outlook, not from PowerShell.

Credit goes to Scott at flobee.net, who wrote an AMAZING script that ports the Restore-RecoverableItems from O365 to on-prem (again, this should be available in the first place, not everyone wants to, or even can move to O365, Microsoft).

The original script can be found here

The problem is: it did not work all the way; running with more than one parameter, caused the script to result in no output.

What I mean is: if I ran the script with the "-FilterStartTime" and "-FilterEndTime" or "-FilterStartTime" and "-FilterItemType" it wouldn't do anything...I could only use one of those parameters.

So, I edited the script to use AQS (Advanced Query Syntax) and hardcoded a date range so it will run successfully.

Saving The Script:

Grab the RecoverableItemsAQS.psm1 from my Google Drive here.

**Note** It's a .psm1 file. That's a PowerShell module that you need to import into your PowerShell session, which I'll show you later.

Open the script in Notepad and search for AQS and change the date range in the following block; you'll need to do this in both the Get-MailboxRecoverableItems and Restore-RecoverableItems sections:

  $AQS = 'received:4/22/2015..6/7/2017'

Edit the date range to fit your needs.

Also, change the page size in both sections to 1000; the default is 50.

Search for:

$pageSize = 50

You will also need the EWS API 2.2 installed on the machine where you'll be running the script from. Best is to install this on a non-Exchange server as it can cause some serious performance issues on Exchange - see my previous post for those issues.

Running The Script:

After you've made the above changes, save the script somewhere like C:\Scripts

There are two functions of the script: Get Recoverable Items, and Restore Recoverable Items.

**Note** You cannot pipe the Get function into the Recover Function, it is only for reporting.

Get Recoverable Items:

**We're going to be using Impersonation to run against mailboxes. This is easier and more secure than setting Full Access on a bunch of mailboxes. You'll need to pass the credentials into the script for the account that has impersonate rights to run both Get and Recover.

To set up impersonation follow this TechNet guide.

In the EMS, run:

Import-Module "C:\Scripts\RecoverableItemsAQS.psm1"

Then to get a list of items in the dumpster output to a txt file, run:

Get-MailboxRecoverableItems -Identity staceybranham@exchangeitup.com -UseImpersonation -Credential (Get-Credential) | Out-File c:\temp\staceydeletions.txt

**Note** The -UseImpersonation and -Credential (Get-Credential) parameters are for your impersonation account you set up earlier. You'll be prompted with a Widows cred prompt.

Restore Recoverable Items For A Single User:

In the EMS, run:

Import-Module "C:\Scripts\RecoverableItemsAQS.psm1"

Next run the following to recover on a single mailbox:

Restore-MailboxRecoverableItems -Identity staceybranham@exchangeitup.com -UseImpersonation  -Credential (Get-Credential)

Restore Recoverable Items In Bulk:

To run the recovery on a multiple mailboxes, we'll pipe in a CSV list of email addresses, and run the script against each of those.

Create your CSV where the heading is SMTPAddress, then each user's email address on each row like so:


Save it as MailboxImport.csv somewhere like C:\Scripts.

Next in the EMS, run:

$cred = get-credential

**Note** These creds will be the account you're using that has Impersonation Rights

Then, run:

$mailboxes = Import-CSV "C:\Script\MailboxImport.csv"; $mailboxes | foreach {Restore-MailboxRecoverableItems -Identity $_.SMTPAddress -UseImpersonation -Credential $cred}

Use caution when running the script on a bunch of mailboxes at one time, because it generates transaction logs, just as regular mail-flow does.

Test with groups of mailboxes to see how big your log volumes are growing before you run the script on a large batch. In my experience, recovering ~9GB of deleted items generated 1 to 1.5GB of logs.

My advice is to split up your Import CSVs into 10-20 mailboxes per day (or per backup cycle) so the logs will truncate in between runs -or- stagger the groups of mailboxes which are homed on different DB's...say 2 or 3 mailboxes per DB in each list.

Saturday, August 25, 2018

Exchange 2016 Upgrade Pre-CU5 to CU9/CU10

I see quite a few questions on forums floating around about upgrading old versions of Exchange 2016 to the latest CUs. The cause for concern are the new requirements for .Net on Exchange.

For instance, Exchange CU9 requires either .Net Framework 4.6.2 or 4.7.1; Exchange CU10 requires .Net 4.7.1.

So here's the issue: how do you upgrade Exchange, when you're far out of date and update .Net without breaking your mail servers?

I've put together the following guide which will walk you through the process, and hopefully calm your fears; it's not as painful as you think!
I just completed this on a very non-standard Exchange environment, so it should work for you too.

1. In your monitoring solution, pause all Exchange servers and the Domain Controller holding the FMSO roles

a. Download .Net 4.7.1 from here:


b. Download Exchange 2016 CU9 from here:


**Note** Microsoft only hosts the current and N-1 CU's, so if CU9 is no longer available, leave a comment and I'll share it out with you.

b.1. If you're installing CU10, download it from here:


2. On The DC Holding FSMO

a. Install .Net 4.7.1

**Note** This *shouldn't* require a reboot.

b. In an Elevated CMD, run each of the following commands one-by-one:

setup.exe /PrepareSchema /IAcceptExchangeServerLicenseTerms

Setup.exe /PrepareAD /IAcceptExchangeServerLicenseTerms

setup.exe /PrepareDomain /IAcceptExchangeServerLicenseTerms

c. After Domain Preps, check replication to ensure all is well:

repadmin /showrepl

**Note** I normally run the AD prep commands from an Exchange server, but since CU9 requires .Net 4.6.2 -or- 4.7.1 and CU10 requires 4.7.1, it will halt the install saying that .Net 4.5 or higher is required. So, I chose to run it from the DC so as to not break Exchange (CU4, which is currently installed, doesn't support .Net 4.7.1 so this would cause conflicts).

3. On your First Mailbox Server

- Remove from Load Balancer Rotation

a. Start Exchange Server Maintenance Mode

These commands will stop Transport queues, stop DAG cluster, perform DB switchover, block DB activation, and enable maintenance mode

a.1. In the EMS, run each command separately:

$Computer = $ENV:ComputerName

Set-ServerComponentState $Computer -Component HubTransport –State Draining -Requester Maintenance

Redirect-Message -Server $Computer -Target MBX2.exchangeitup.com

Suspend-ClusterNode $Computer

Set-MailboxServer $Computer -DatabaseCopyActivationDisabledAndMoveNow $True

Set-MailboxServer $Computer -DatabaseCopyAutoActivationPolicy Blocked

Set-ServerComponentState $Computer -Component ServerWideOffline -State Inactive -Requester Maintenance

b. Install .Net 4.7.1

**Note** MS says to use newest .Net version supported by Exchange. A lot of the advice on forums will mislead you by saying to install 4.6.2 then install CU9, then install 4.7.1. Don't do that! This adds a whole lot of unnecessary steps; just go straight to the latest .Net.

b.1. **If you are installing CU10** Download and install Visual C++ (CU10 requires it) from here:


b.2. Reboot

c. Uninstall previous the CU Language Pack by running the following in an Elevated CMD:

cd C:\wherever-you-have-stored-your-language-pack

setup.exe /RemoveUMLanguagePack:de-DE

**Note** Change "RemoveUMLanguagePack:de-DE" to whatever language pack you have

d. Mount the CU9 or CU10 ISO and run setup.exe - Run As Administrator for good measure

**Note** This will take a long time, so plan for the whole day depending on how many servers you have.

**Note** During the CU9/CU10 readiness checks if the installer throws an error stating: setup can't continue with the upgrade because the mscorsvw (PID#) has open files, just give it some time. The problem is, the .Net compiler hasn't finished yet, which takes about 30 minutes. Open task manager and watch for .Net optimization; once it disappears, it's finished and you can hit Retry in the installer.

e. Install the newest CU Language Pack - You'll need to Run As Administrator

f. Stop Exchange Server Maintenance Mode

These commands will start Transport queues, start DAG cluster, unblock DB activation, disable maintenance mode, and restart Transport Services

f.1. In the EMS, run each command separately

$Computer = $ENV:ComputerName

Set-ServerComponentState $Computer -Component ServerWideOffline -State Active -Requester

Resume-ClusterNode $Computer

Set-MailboxServer $Computer -DatabaseCopyActivationDisabledAndMoveNow $False

Set-MailboxServer $Computer -DatabaseCopyAutoActivationPolicy Unrestricted

Set-ServerComponentState $Computer -Component HubTransport -State Active -Requester Maintenance

Restart-Service MSExchangeTransport

Restart-Service MSExchangeFrontEndTransport

Get-ServerComponentState MBX1 - ensure that all services are active

 **Note** If the Database copies say: Content index state:  HealthyAndUpgrading no need to worry, carry on with your upgrade - it could take days depending on the size of your DB's. It's still totally safe to mount or activate copies on the other DAG members during this operation.

4. Rinse and Repeat the above steps on each Mailbox Server

5. On Your First Edge Server

a. Start Exchange Server Maintenance Mode

These commands will stop Edge Transport queues, and enable maintenance mode

a.1. In the EMS, run each command separate

$Computer = $ENV:ComputerName

Set-ServerComponentState $Computer -Component ServerWideOffline -State Inactive -Requester Maintenance

b. Install .Net 4.7.1

b.1. Reboot

b.2. **If installing CU10** Install Visual C++

c. Run the Exchange CU setup from an Elevated CMD:

Setup.exe /mode:upgrade /IAcceptExchangeServerLicenseTerms

d. Stop Exchange Server Maintenance Mode

These commands will start Edge Transport queues, and disable maintenance mode

d.1. In the EMS, run each command separate

$Computer = $ENV:ComputerName

Set-ServerComponentState $Computer -Component ServerWideOffline -State Active -Requester Maintenance

6. Repeat for Each Edge

7. On your Management Tools Server

a. Install .net 4.7.1

a.1. **If installing CU10** Install Visual C++

b. Install the CU - the CU installer will upgrade the tools automatically - it detects the installed roles.

8. Cleanup/Checking our work

a. Check build numbers on Servers:

Get-ExchangeServer | fl name,edition,admindisplayversion

For CU10 you'll get: 15.01.1531

For CU9, you'll get: 15.01.1466

**Note** The Edge will not show the current build number when running the above cmdlet from a Mailbox Server; this is by design. If you want it to show the updated build, you would need to re-subscribe the EdgeSync.

b. Rebalance the DAG by running the following in the EMS:

cd exscripts

.\RedistributeActiveDatabases.ps1 -DagName DAG01 -BalanceDbsByActivationPreference -Confirm:$False

**Note** Change "DAG01" to the name of your DAG

Now your Exchange environment will be up to date and healthy. In my environment, I chose to go to CU9 instead of CU10 because I always stay N-1 on CU's; if you do follow the forums it seems that almost every other CU release causes something to break in Exchange...most have been riddled with bugs. I would personally rather be one step behind than have to open an MS ticket to fix something they broke ;)

Saturday, August 11, 2018

Exchange OWA TimeZoneException Error

Here's an easy one today: Users sometimes get the "Time Zone Exception" error below when logging in to OWA, and it all depends on the web browser their using:

X-OWA-Error: SDServerErr;Microsoft.Exchange.Services.Core.Types.TimeZoneException
X-OWA-Version: 15.1.1531.3
X-FEServer: MBX1
X-BEServer: mail.exchangeitup.com

I've never seen this error happen in IE, but very often in Chrome and Firefox. This is due to the way those two browsers implement adblock, which doesn't handle OWA very well.

We don't want to disable adblock completely, since it is useful, but we'll turn it off for OWA.

In Chrome:

Navigate to OWA, then click “Secure” to the left of the URL.

**Note** It might show an "i" info bubble instead of Secure, if you aren't redirecting OWA to SSL - which you really should do.

Find the “Ads” entry, then choose Allow in the menu.

Now, OWA won't be blocked, but other sites will still block ads as normal.

In Firefox:

Browse to OWA and click the shield icon next to the URL to bring up the Control Center.

In the Control Center, click Disable protection for this session.

Once Tracking Protection is disabled, a shield icon with red strikethrough (which means tracking protection is off) will appear in your address bar.

If you need to do this on a lot domain machines, there are GPO templates, but I haven't dove into them to see what you can control...there might be a way to set the adblock exclusions for OWA if you wanna check them out.

For Chrome, grab them here

For Firefox, grab them here

You could also just disallow those browsers and force IE, but that's up to you, and it of course wouldn't have any affect on non-domain computers.

Saturday, July 21, 2018

Exchange Export/Import EUM Addresses Cross Forest

In a previous post I showed you how to set up Exchange UM, Cross Forest with a Selective Trust in place.

Under the "Enable Users for Unified Messaging > Sync the EUM Proxy Addresses" section of that article, I outlined that in order for UM to function Cross Forest or when you have Exchange in a Resource Forest and your Skype For Business environment is in the Accounts Forest, you need to sync over the "EUM" proxy addresses, backwards to the Accounts Forest.

In my environment, we run MIM (Microsoft Identity Management) to sync our users from the Accounts to the Resource Forest. Unfortunately, I don't have direct control over what attributes get sync'd and the infrastructure "trolls" who do administer MIM have zero interest in adding the backwards sync of those EUM addresses...yeah don't listen to the Messaging Admin that this is something needed for actual functionality <rolleyes>

Since we can't (they won't) sync those EUM addresses, I came up with a way to export the EUM addresses that are assigned to mailboxes in the Resource Forest and output into a CSV. Then, we can take that CSV and use it to import them on the AD Users in the Accounts Forest.
This will at least save you time from manually copying/pasting them, especially if you have to do a bunch at one time (I just had to do 20 of them; no way was I gonna copy those by hand).

Export EUM Proxy Addresses to CSV

First, we'll need to export a list of all UM Enabled Mailboxes, and their corresponding EUM proxy addresses.

**Note** Using Exchange UM results in two EUM addresses per user - a 3rd party UM solution like Cisco Call Manager only uses one EUM address. (We run both types in my environment, until we cutover to purely Exchange UM)

Here's an example:

Exchange UM:


Cisco UM:

EUM:292;phone-context=Cisco CallManager Headquarters.exchangeitup.com

The following cmdlet will grab both types of EUM addresses and export them to a CSV separating them with a comma.
We're using a comma because the actual EUM address already contain semicolons as seen above, and that will come into play later when we import them.

Fire up the Exchange Management Shell (EMS) in your Resource Forest and run:

Get-Mailbox -Filter '((EmailAddresses -like "*EUM*"))' | select DisplayName, @{n="EUMAddress";e={($_.EmailAddresses | ? {$_ -like "EUM*"}) -join ","}} | Export-CSV "C:\Temp\EUM_Addresses.csv" -notypeinformation

This cmdlet will filter for only EUM addresses, leaving out any other type of proxy addresses, because we don't need them.

Import CSV And Add EUM Addresses

Once you have your CSV, copy it over a server in your Accounts Forest and open up the Active Directory PowerShell.

Run the following cmdlet, and it will import all EUM address according to DisplayName:

Import-Csv "C:\Temp\EUM_Addresses.csv" | foreach {Set-ADUser -Identity $_.DisplayName -Add @{Proxyaddresses=($_.EUMaddress -split ",")}}

**Note** There's no need to add any silently continue if the user already has the EUM in place, it won't throw any errors that it exists, and the cmdlet will run until it's finished.

**Note** See that comma that splits the EUM addresses in the last part of the cmdlet? If we had used a semicolon in the export, it would have imported the EUMs on several different lines, and that wouldn't be correct.

Depending on how many you had to import, give it a bit for the cmdlet to complete.

Once it's finished, go check a couple AD accounts from your list in your Accounts Forest, and you will see those EUM addresses populated in the proxyaddresses attribute!

Next, go tell the Infrastructure/AD guys that you don't need their help, you can do everything on your own ;)

Wednesday, July 4, 2018

Exchange Mailbox Litigation Hold Size Report

Most companies always have some sort of litigation going on, which requires putting users' mailboxes on hold to keep historical data. Sometimes I forget who's on litigation hold, and they stay in that state long after they should be - taking up storage space on Exchange.

I created a little script that will spit out an HTML report of mailboxes with litigation hold enabled and their total size.
This is also useful for when your higher-ups need that info for auditing and such, since it's easy to read in HTML format.

Grab the script from my Google Drive here.


Copy the following block and save as Litigation-Report.ps1

$style = "<style>BODY{font-family: Arial; font-size: 10pt;}"
$style = $style + "TABLE{border: 1px solid black; border-collapse: collapse;}"
$style = $style + "TH{border: 1px solid black; background: #dddddd; padding: 5px; }"
$style = $style + "TD{border: 1px solid black; padding: 5px; }"
$style = $style + "</style>"

Get-Mailbox -ResultSize Unlimited -Filter {LitigationHoldEnabled -eq $True} | Get-MailboxStatistics | Sort totalitemsize -desc | select displayname, totalitemsize | ConvertTo-HTML -head $style | out-file Litigation_Size.htm

Once you have the script downloaded or copied, fire up the EMS (Exchange Management Shell) and cd to the location where you saved it.

Next, run Litigation-Report.ps1

The script will save the "Litigation_Size.htm" in the same location as the .ps1 file.

Now you have a nice, pretty report...give it your legal department and tell them they're using too much space :)

Saturday, June 16, 2018

Exchange Get Distribution Groups That Contain Mail Contacts

A lot of organizations, including mine, add external contacts to Distribution Groups. In my case, we have a US and EU business units and users need to be included in groups. A lot of administrative housekeeping involves reporting which of those groups contain contacts.

There's no easy oneliner or a way to grab all the groups at once in the EAC or ADUC to get that info, so I threw together a quick little script that will list those DL's for us.

You can grab the script from my Google Drive here


Copy the following block into Notepad and save it as something like Get-DL-Contacts.ps1

foreach($dl in get-group -resultsize unlimited )
  foreach ($member in $dl.members)
   $recipient = ($member | get-recipient -erroraction silentlycontinue)
   if ($recipient.RecipientType -eq "MailContact")
    Write-Host "$($dl.Name) Contains Contacts"

Once you have the .ps1 file, fire up the Exchange Management Shell, cd to the location you saved it and run Get-DL-Contacts.ps1

**Note** If you need to report on Mail Users instead of Contacts, change the $recipient.RecipientType -eq "MailContact") to $recipient.RecipientType -eq "MailUser")

Depending on the amount of distros you have, it might take a while to run (I have 1500 and it took about 30 minutes) but when it completes, you'll get the list of lists in the PS window.

**Note** When I have time I'll update the script to list both the DL's and the actual contacts within each one...unless one of my readers wants to do that for me :)

Saturday, May 26, 2018

Exchange (One of Many Reasons) IIS Worker Process Using 99% Memory

Our Infrastructure Team had to bounce a domain controller in the middle of the day (yeah they didn't listen to me and did it anyway).
After which, one of my Exchange 2016 Mailbox Servers puked it's guts out every time I tried a PowerShell cmdlet; restarting the EMS resulted in WinRM errors as well. The machine was also running slow, so I checked task manager and sure enough, RAM was pegged at 99% from an IIS Worker Process.

On a lot of blogs and forums, in order to see what's consuming resources, people say to install procmon, find the PID, get the process name, kill it with an elevated CMD, etc.
I for one, really don't like installing extra software on my Exchange servers. Also, killing an entire IIS service will almost always bounce your client connections, because in Exchange 2013/2016 TONS of backend services run off of IIS. Luckily, there's a much easier way to find out which IIS pool is hogging your resources, and we can recycle that specific pool without killing your connections.

The Fix:

Open IIS Manager (not IIS 6.0)

Click your server name, and then Worker Processes
IIS Manager Worker Process

Click the Virtual Bytes heading twice to sort in descending order.

In my case its the MSExchangePowerShellAppPool consuming all that RAM:

IIS Manager RAM Consumption

**Note** You can get the PID from here without having to install something like procmon ;)

Make sure to close the EMS

Click Application Pools

Right-click MSExchangePowerShellAppPool or whichever pool is consuming your resources.

Hit Recycle:

IIS Manager Recycle Pool

Give it a minute and try the EMS again

That looks much better!

Task Manager Normal RAM

Successful EMS Session
Now, go yell at your infra guys to stop doing destructive things in the middle of a work day!