Exchange - 2019 CU15 Install Error mmc.exe and where.exe

I had a customer trying to upgrade their Exchange 2019 Servers from CU14 to CU15 and hit an error during the Management Tools step. The installer complained about the mmc.exe and where.exe files. 

For reference "where.exe" is part of PowerShell. Basically the installer is looking in two places for those two files and they're either missing or corrupt.

Here's the full error:

The following error was generated when "$error.Clear(); 
          if (Test-Path ($Env:SystemRoot + "\system32\mmc.exe")) {Copy-Item -Path ($RoleInstallPath+"Bin\mmc.exe.config") -Destination (Split-Path (where.exe mmc)) -Force}
        " was run: "System.Management.Automation.ParameterBindingException: Cannot convert 'System.Object[]' to the type 'System.String' required by parameter 'Destination'. Specified method is not supported. ---> System.NotSupportedException: Specified method is not supported.
   at System.Management.Automation.ParameterBinderBase.CoerceTypeAsNeeded(CommandParameterInternal argument, String parameterName, Type toType, ParameterCollectionTypeInformation collectionTypeInfo, Object currentValue)
   --- End of inner exception stack trace ---
   at System.Management.Automation.ParameterBinderBase.CoerceTypeAsNeeded(CommandParameterInternal argument, String parameterName, Type toType, ParameterCollectionTypeInformation collectionTypeInfo, Object currentValue)
   at System.Management.Automation.ParameterBinderBase.BindParameter(CommandParameterInternal parameter, CompiledCommandParameter parameterMetadata, ParameterBindingFlags flags)
   at System.Management.Automation.CmdletParameterBinderController.BindParameter(CommandParameterInternal argument, MergedCompiledCommandParameter parameter, ParameterBindingFlags flags)
   at System.Management.Automation.CmdletParameterBinderController.BindParameter(UInt32 parameterSets, CommandParameterInternal argument, MergedCompiledCommandParameter parameter, ParameterBindingFlags flags)
   at System.Management.Automation.CmdletParameterBinderController.BindParameters(UInt32 parameterSets, Collection`1 arguments)
   at System.Management.Automation.CmdletParameterBinderController.BindCommandLineParametersNoValidation(Collection`1 arguments)
   at System.Management.Automation.CmdletParameterBinderController.BindCommandLineParameters(Collection`1 arguments)
   at System.Management.Automation.CommandProcessor.BindCommandLineParameters()
   at System.Management.Automation.CommandProcessor.Prepare(IDictionary psDefaultParameterValues)
   at System.Management.Automation.CommandProcessorBase.DoPrepare(IDictionary psDefaultParameterValues)
   at System.Management.Automation.Internal.PipelineProcessor.Start(Boolean incomingStream)
   at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input)
   at System.Management.Automation.PipelineOps.InvokePipeline(Object input, Boolean ignoreInput, CommandParameterInternal[][] pipeElements, CommandBaseAst[] pipeElementAsts, CommandRedirection[][] commandRedirections, FunctionContext funcContext)
   at System.Management.Automation.Interpreter.ActionCallInstruction`6.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)".

**Note** This issue happens quite often when installing EXCH Management Tools on Windows 10/11 as well and the fix is the same. I've also seen it with Exchange 2016.

The Fix:

First we need to check if both the where.exe and mcc.exe files exist in the following directories:

\windows\system32

\windows\syswow64

If not, copy them from a working server and run the install again.

If that doesn't work, we'll need to rename the c:\windows\syswow64\mmc.exe to mmc.old by using the TrustedInstaller account.

You can run CMD as TrustedInstaller with psexec:

Grab PsTools - Sysinternals | Microsoft Learn

In an elevated command prompt run: 

psexec -I -s cmd.exe

A new cmd window will open.

Running a “whoami” in that window should show nt authority\system as the user.

Then rename the mmc file like so:

cd c:\windows\syswow64

rename mmc.exe mmc.old

Now run the CU15 install and it should complete!

Exchange - Migrating EXCH 2016 to 2019 Guide

Alot of my customers are scrambling to get ready for Exchange SE (Subscription Edition) which requires Exchange 2019 CU14/CU15 for the in-place upgrade. Seriously, I've led 7 migrations in the past 3 months. The reason they're in a hurry is because Exchange 2016/2019 goes End Of Life (EOL) October 14, 2025. Exchange SE will be released early Q3 of 2025 (July-ish). See how very little time there is to perform an upgrade before you're out of support?

Another issue is Microsoft throttles mail flow from out-of-date Exchange Hybrid servers...there's no word yet on when 2019 CU14/CU15 will be throttled, but since it will be EOL, I'd assume it would come pretty quickly after 10/2025.

I'll write a full post later on how to upgrade/migrate Exchange 2016 to 2019, but for now I'm providing a guide I wrote in a Word doc, which you can grab from my GDrive here.

The guide contains things like network and directory requirements, server req's, pre-req's and the steps to install 2019 from start to finish. It also includes migrating Edge Servers, if you have those.

There are also two embedded docs for virtual directory settings and exporting/importing certs...let me know if those don't work and I'll send you the doc to your email. Also let me know if it's missing anything you can think of.

A word on Edge Servers: if you still have them, and you're in a Hybrid, now would be a great time to get rid of them. Can they be useful? Yeah...sorta. Are they needed in a Hybrid? No! With email, less hops are better, so if you're building new Mailbox Servers, it would be beneficial to update inbound NAT for SMTP (TCP port 25) to point to the Exchange 2019 Mailbox server(s) and remove the unnecessary management and complexity of your environment.

Exchange: Don't Fear The Wizard...Re-running the HCW

A lot of my customers are in the process of renewing their Exchange certificates or upgrading to Exchange 2019 in preparation for Exchange SE (coming early 3rd quarter this year). Every one of them I've worked with are scared (terrified, actually) of re-running the Hybrid Configuration Wizard (HCW) thinking that it will overwrite their current settings like connectors and such because that's what they experienced in the past. As a result, some of them haven't updated their hybrid config in years - one of them hasn't touched it since setting up Exchange 2010.

Have no fear, the new HCW is leaps and bounds better than it used to be. You can pick and choose what you want to update, without touching other settings that would affect mail flow or custom setups.

Download the latest HCW from here:

https://aka.ms/hybridwizard

**Note** You can download and run it from any domain member, like a Management Tools server; you don't have to run it on an Exchange server, but it is recommended.

Once you have it installed, run the HCW and select Classic Hybrid and check the "Choose Exchange Hybrid Configuration option":

This will bring up a menu where you can choose what you wanna update.

If you just renewed your hybrid cert (your public SAN cert), or your OAuth cert, just clear all the checkboxes and select "Oauth, Intra Org Connector and Organization Relationship" option and "Update Secure Mail Certificate for connectors" option:

Now next, next, next through the steps and that's it! No other config needed, and your custom connectors are still in-tact. Easy-peasy!

**Note** Global Admin (GA) is no longer required to re-run the HCW, you can now run it with an Org Management account in Exchange and Exchange Online...makes things easier and you don't have call and wake up your Global Admin in the middle of the night.

Also...

When renewing the Exchange Auth Certificate (the self-signed one that lasts for 5 years), you only need to re-reun the HCW if you still have on-premises mailboxes that require free/busy or on-going migrations. Lots of articles, even from Microsoft, say it's required...it is not required. That being said, you can run it as many times as you like, since it won't break anything now.

Exchange PSA - Update Your Exchange and Load Balancer Certs At The Same Time

A lot of my customers are coming up on cert renewals for Exchange...must be that time of the year.

One of said customers, preemptively updated the certificate on their F5 Load Balancers without renewing and installing the Exchange certs. This resulted in Outlook credential prompts for the bulk of their users. 

That's because clients will authenticate against the LB/reverse proxy (depending on your SSL settings) but when the LB passes traffic to Exchange, there's a cert mismatch, resulting in prompts.

In the interim, they had to failover the databases to another DAG node that was homed on another F5, remove those servers from the LB rotation and then finally revert the cert to previous one.

So...don't update one cert and then the other - do them all at once. The easiest way is export the cert from an Exchange server and import it onto your Load Balancer (Kemp makes this easy, not sure about other providers).

Exchange - Regex Transport Rule To Block Recent Hotmail/Outlook.com Spam

I have a customer who's Exchange Online tenant is getting bombarded with spam from Hotmail and outlook.com free accounts. The spammers are creating "one-and-done" accounts with the following format:

firstname1234lastname5678@hotmail.com

firstname1234lastname5678@outlook.com

This particular tenant is receiving 1,000's of messages from these senders, so blocking them one-by-one or adding them to the tenant blocklist isn't feasible...neither is blocking the Hotmail/Outlook TLD because genuine senders could still be using those services.

So, I've come up with a regular expression to add to a Transport Rule (aka Mail Flow Rule) to block those by the text patterns.

Regex:

The patterns will look like so:

'^[a-zA-Z]+[0-9]+[a-zA-Z]+[0-9]+@outlook\.com$'

'^[a-zA-Z]+[0-9]+[a-zA-Z]+[0-9]+@hotmail\.com$'

Transport Rule:

We'll create a Transport Rule with the following settings:

Apply this rule if…

    The sender... is external

    The sender... address matches any of these text patterns:

        '^[a-zA-Z]+[0-9]+[a-zA-Z]+[0-9]+@outlook\.com$'

        '^[a-zA-Z]+[0-9]+[a-zA-Z]+[0-9]+@hotmail\.com$'

Do the following…

    Modify the message properties... set the SCL to 9 

**Note** Or you can just straight up block the messages, but I would do that after testing to ensure it’s catching these properly.

Now, you should see much less spam from Hotmail/Outlook.com until Microsoft can shut them down...they are starting to institute new throttling rules on new accounts, so hopefully that'll help.

M365 - SuccessFactors/SAP 554 5.6.0 Invalid Message Content Error

In my environment, we run SuccessFactors (part of SAP) and as most Exchange admins can attest, SAP is absolutely terrible when it comes to relaying through Exchange/M365. One of the many problems we have (weekly, it seems) was our timecard notification jobs every Monday started erroring out after about the 5th message. The error (below) would cause any further messages in the job to stop.

It would throw a 554 5.6.0 invalid message content, stopping all further messages:

The way we have our relaying set, is for SuccessFactors (SF) to connect to our on-premises Exchange Hybrid servers. So, the most obvious place to start was message tracking on there. Sure enough, tracking showed the following:

{[{LRT=};{LED=554 5.6.211 Invalid MIME Content: Single text value size (32782) exceeded
allowed maximum (32768) for the 'Replicon_time_entries' header.
[MN2PR06MB6623.namprd06.prod.outlook.com 2024-04-29T14:43:26.364Z 08DC665FAD5D7B03]
[DS7PR05CA0090.namprd05.prod.outlook.com 2024-04-29T14:43:26.372Z 08DC6833DB2077BC]
[DS3PEPF000099D3.namprd04.prod.outlook.com 2024-04-29T14:43:26.374Z
08DC667B5B6206E2]};{FQDN=};{IP=}], [{LRT=};{LED=554 5.6.211 Invalid MIME Content: Single
text value size (32782) exceeded allowed maximum (32768) for the 'Replicon_time_entries'
header. [MN2PR06MB6623.namprd06.prod.outlook.com 2024-04-29T14:43:26.364Z 08DC665FAD5D7B03]
[DS7PR05CA0090.namprd05.prod.outlook.com 2024-04-29T14:43:26.372Z 08DC6833DB2077BC]
[DS3PEPF000099D3.namprd04.prod.outlook.com 2024-04-29T14:43:26.374Z
08DC667B5B6206E2]};{FQDN=};{IP=}], [{LRT=};{LED=554 5.6.211 Invalid MIME Content: Single
text value size (32782) exceeded allowed maximum (32768) for the 'Replicon_time_entries'
header. [MN2PR06MB6623.namprd06.prod.outlook.com 2024-04-29T14:43:26.364Z 08DC665FAD5D7B03]
[DS7PR05CA0090.namprd05.prod.outlook.com 2024-04-29T14:43:26.372Z 08DC6833DB2077BC]
[DS3PEPF000099D3.namprd04.prod.outlook.com 2024-04-29T14:43:26.374Z
08DC667B5B6206E2]};{FQDN=};{IP=}]}

That means the header is too large for O365 to accept, so it bounces - it does not actually reach EXO, so tracing on that side won't show it.

If you look at the header of one of the working messages, SAP is adding a whole chunk of text, making the header way too big. Here's part of the header where you can see the extra junk:

Received: from BLAPR06MB6819.namprd06.prod.outlook.com (2603:10b6:208:29c::10)
 by SA1PR06MB7860.namprd06.prod.outlook.com with HTTPS; Mon, 29 Apr 2024
 14:43:30 +0000
Received: from CH2PR20CA0025.namprd20.prod.outlook.com (2603:10b6:610:58::35)
 by BLAPR06MB6819.namprd06.prod.outlook.com (2603:10b6:208:29c::10) with
Received: from vsa12723543.stl1.od.sap.biz (172.26.59.32) by
 EXHYB-01.exchangeitup.com (172.2.1.11) with Microsoft SMTP Server (TLS) id
 15.0.1497.2; Mon, 29 Apr 2024 09:38:24 -0500
Date: Mon, 29 Apr 2024 14:38:24 +0000
From: <sf_email@exchangeitup.com>
To: <raghi@exchangeitup.com>, <huang@exchangeitup.com>
Message-ID: <1453815201.500.1714401504559@webmail.exchangeitup.com
Subject: Replicon Clock Time Report for your Employees.
MIME-Version: 1.0
Content-Type: multipart/mixed;
boundary="----=_Part_499_2028040997.1714401503408"
breadcrumbId: ID-vsa12723543-1714116965992-360-6612
Replicon_time_entries:
 =?us-ascii?Q?Employee=5FId,User=5FFirst=5FName,User=5FLast=5FName,Punch?=
 =?us-ascii?Q?=5FDate,Actual=5FTime,Rounded=5FTime,Device=5FName20002644,M?=
 =?us-ascii?Q?ariah,Ak,04/22/2024,7:43:12,7:45:00,RepCC-4220002644,Mari?=
 =?us-ascii?Q?ah,Ak,04/22/2024,16:07:06,16:15:00,RepCC-4220008894,Kimbe?=
 =?us-ascii?Q?rly,Schex,04/22/2024,8:03:26,8:00:00,RepCC-4220008894,?=
 =?us-ascii?Q?Kimberly,Schex,04/22/2024,16:27:29,16:30:00,RepCC-4220?=
 =?us-ascii?Q?008894,Kimberly,Schex,04/23/2024,5:45:53,5:45:00,RepCC?=
 =?us-ascii?Q?-4220008894,Kimberly,Schex,04/23/2024,14:39:37,14:45:0?=
 =?us-ascii?Q?0,RepCC-4220008894,Kimberly,Schex,04/24/2024,5:51:44,5?=
 =?us-ascii?Q?:45:00,RepCC-4220008894,Kimberly,Schex,04/24/2024,14:3?=
 =?us-ascii?Q?1:30,14:30:00,RepCC-4220008894,Kimberly,Schex,04/25/20?=
 =?us-ascii?Q?24,5:49:15,5:45:00,RepCC-4220008894,Kimberly,Schex,04/?=
 =?us-ascii?Q?25/2024,15:50:32,15:45:00,RepCC-4220008894,Kimberly,Schex?=
 =?us-ascii?Q?der,04/26/2024,7:18:20,7:15:00,RepCC-4220008894,Kimberly,Sch?=
 =?us-ascii?Q?ex,04/26/2024,15:02:55,15:00:00,RepCC-4220009046,David?=
 =?us-ascii?Q?,Le,04/22/2024,8:52:00,9:00:00,RepCC-4220009046,David,L?=
 =?us-ascii?Q?e,04/22/2024,17:19:04,17:15:00,RepCC-4220009046,Davi=09?=
 =?us-ascii?Q?d,Le,04/23/2024,14:42:26,14:45:00,RepCC-4220009046,Davi?=
 =?us-ascii?Q?d,Le,04/23/2024,17:33:48,17:30:00,RepCC-4220009046,Davi?=
 =?us-ascii?Q?d,Le,04/24/2024,7:09:38,7:15:00,RepCC-4220009046,David,?=
 =?us-ascii?Q?Le,04/24/2024,11:33:42,11:30:00,RepCC-4220009046,David,?=
 =?us-ascii?Q?Le,04/24/2024,12:39:00,12:45:00,RepCC-4220009046,David,?=
 =?us-ascii?Q?Le,04/24/2024,17:07:04,17:15:00,RepCC-4220009046,David,?=
 =?us-ascii?Q?Le,04/25/2024,8:23:55,8:30:00,RepCC-4220009046,David,Le?=
 =?us-ascii?Q?,04/25/2024,12:38:36,12:45:00,RepCC-4220009046,David,Le?=
 =?us-ascii?Q?,04/25/2024,13:24:12,13:30:00,RepCC-4220009046,David,Le?=
 =?us-ascii?Q?,04/25/2024,18:24:30,18:30:00,RepCC-4220009046,David,Le?=
 =?us-ascii?Q?,04/26/2024,8:56:30,9:00:00,RepCC-4220009046,David,Leon?=
 =?us-ascii?Q?,04/26/2024,13:09:16,13:15:00,RepCC-4220009046,David,Leon?=
 =?us-ascii?Q?,04/26/2024,17:19:09,17:15:00,RepCC-42?=
SAP_MessageProcessingLogID: AGYvsBZt3pwAL7njbCl8KGJK3Otz
SAP_MplCorrelationId: AGYvsBZ7JcYHEU4lhPxb5wfRsIrh
SAP_PregeneratedMplId: AGYvsN9elo-RaXm3THlA0l_jfzPn
SapGroup: 1
SapSplitExpression: //RepliconPunches
Return-Path: sf_email@exchangeitup.com

See all that extra text? That made the header way past the 32KB size limit. So, we need to get our SAP guys to check their flow on the SAP server-side to stop it from editing the header.

Once you get that fixed it'll be just like any other mail header and send successfully!

Exchange/M365 - Who Moved Or Deleted A Folder In A Shared Mailbox?

I recently had an all-too-common request: "who deleted a folder from a Shared Mailbox?".

Technically you can audit these actions in M365 Purview/Exchange Admin Audit log, but it's cumbersome and takes forever...you know when you start a compliance search and hit refresh dozen times and it sits there in a "starting" state?

This is the method I've used for years before MS added the "audit 'move' for delegates" setting to mailboxes and it's still my go-to method over auditing. The reason being that auditing Shared Mailboxes can be hit or miss, and you can't easily tell where an entire folder was moved to; works fine for single items.

A faster, cleaner, way is to get the list of users who have Full Access into a CSV and pipe that in the Exchange Management Shell (EMS) to find the "missing" folder. 

The default way shared mailboxes function is: whoever deleted a folder/item, it goes into their Deleted Items folder, not the Shared Mailbox Deleted Items...unless you've done a registry hack to change it (which is only feasible if you have a handful of users).

In my environment, we use Security Groups to assign Full Access, so I'll show you how to grab that user list, but the method remains the same if you assign users one-by-one as well.

Create a CSV with "Identity" as the header and the email addresses below:

You can export the group members by running:

$DL = "DL"

Get-DistributionGroupMember -Identity $DL | Select primarysmtpaddress | Export-CSV "members.csv" -NoTypeInformation

**Note** Change "DL" to the distro you're working with.

Once you have your CSV exported, change the header to "Identity" like above.

Then, run:

import-csv filename.csv | foreach {Get-MailboxFolderStatistics -Identity $_.Identity | Where {$_.Name -eq "Margaret"} | Format-Table Identity,ItemsInFolderAndSubfolders,FolderAndSubfolderSize -AutoSize}

**Note** Change filename.csv to your CSV and "Margaret" to the name of the folder you're looking for.

It will spit out who has that folder in their mailbox:

Notice how it shows the folder under User\Inbox\Foldername path? The problem user inadvertently moved the folder into her Inbox. If they deleted it, it would show User\Deleted Items\Foldername

Caveat: If the user deleted the folder and emptied their Deleted Items folder, you're out of luck recovering the entire folder. You have do to some more PowerShell magic to recover those and that's beyond the scope of this article.

Now, you can contact that user and yell at them to MOVE THE FOLDER BACK!

M365/Proofpoint - Bypass Link Checking For MassMailers Like Mailchimp

One of our users sends monthly newsletter blasts to a couple thousand recipients through MailChimp. The problem is (and I've seen this issue floating around on various forums like reddit) spam filters that do URL checking skew the open and link-click rates in the MailChimp dashboard. So, when the user goes to check the reports it shows 100% clicks, which is not anywhere near accurate considering over 2,000 people received the email blast.

What's happening, is the URL defense service (in our case Proofpoint Targeted Attack Prevention and M365 Security SafeLinks) is simulating a "click" in order to scan the URLs in messages to check for any malware.

In my environment MailChimp is spoofing one of our email addresses. So setting a bypass for the actual MailChimp sender or IP or even sending host wouldn't work because bulk senders use several IPs/hosts and they use those ugly-looking bounce email addresses to send...you know those ones that are like bounce-1234567899876543221@your_own_domain_even_though_its_not_you_sending_it.com

Here's how I created a bypass on both Proofpoint and M365/Exchange Online. 

You might be asking why do we run both Proofpoint and EOP (Exchange Online Protection)? The answer is: we're stupid and gluttons for punishment...it's much more fun to chase down issues when mail takes way too many hops.

Having said that, Proofpoint support pointed fingers at MS and vice-versa...so I had to bypass both to be sure.

Proofpoint:

**Note** This the "real" full-blown Proofpoint, not Essentials.

Policy Route:

Under System > Policy Routes:

Create a policy route called something like DisableTAP

Add the sender email address (in our case the spoofed address). 

Since we're adding a spoofed address, it won't match the actual sender (i.e. envelope sender) so I had to use the "message header 'from' (address only) equals sender@exchangeitup.com" entry.

Save your changes

URL Defense:

Under Email Protection > Targeted Attack Prevention > URL Defense > Settings:

Check the box marked "Disable processing for selected policy routes..."

In the "available" list scroll till you find the DisableTAP rule you created earlier and click the >> button to add it to the "disable for any of" list.

Save your changes

TAP Dashboard:

Go to the Targeted Attack Prevention > Dashboard.

In the dashboard, click the settings gear in the left pane, then the Organization tab.

Scroll down to "Excluded Addresses".

Click "ADD EMAIL ADDRESS", input the address you used for the above Policy Route, and the drop-down to "Sender".

After all this is done, give it 15-30 minutes (sometimes Proofpoint is slow for settings take effect).

Exchange Online:

Under Mail flow > Rules:

Create a new rule called something like "Spammysender bypass SafeLinks"

Apply this if:

The Sender > address includes any of these words

input the sending address we used in the Proofpoint route

Do the following

modify the message properties > set a message header

input X-MS-Exchange-Organization-SkipSafeLinksProcessing with a value of 1

Give it a couple hours to take effect and test the inbound mail from the MailChimp sender. It should now show more accurate open and click rates.

M365 Planning Diagram: Mail Encryption Flow

If you follow my blog, you know that I like to use Visio to draw pretty pictures to use when planning out Messaging Environments, like this one here.

We're currently migrating our mail encryption from Proofpoint to M365 Purview. Like most organizations, we have steering committees filled with business leaders/stakeholders who need stuff in plain language and diagrams always help.

So, I created this encryption "flow" diagram to show what happens when one of our users encrypts an outbound message to when the recipient opens it - start to finish. This way the higher-ups can visualize how exactly mail encryption works.

M365 Mail Encryption Flow Overview

In my example below we have a user sending a message using the [secure] subjectline tag or encrypt button, which is an option that can be set in OME (Purview Message Encryption), or attaches some sensitive data containing PII/PHI (Personally Identifiable Information/Protected Health Information).

The message then flows through Purview where it detects the tag or encrypt button usage or sensitive data using DLP filters, then to the Encryption Portal where it's stored for the recipient.

A notification is sent to the recipient saying they have an encrypted message waiting.

The recipient clicks the link, which takes them to the Purview Portal to view the message.

A notification is sent to the sender that the recipient opened the message.

If the recipient replies, the message is sent back through Purview Encryption then on to the original sender.

Feel free to grab the Visio file from my Google Drive Mail Encryption Flow.vsdx to use for your environment!

Exchange - Assign Calendar Permissions In Bulk

In my current environment, we have a mix of Shared Mailbox Calendars and calendars shared out by users themselves. While I prefer to use Shared Mailboxes, some departments have high turnover depending on if they contain students and interns so managing calendar perms would be a never ending task for admins.

In this instance, a calendar managed by a user got corrupted (happens a lot on M365 for some reason) and we had to create a new calendar. The problem was, this particular calendar had over 70 users with varying permissions (editor and reviewer). So I needed a way to quickly assign those same perms to the new calendar, without the user having to do it by hand.

You may be asking, why not assign perms by group? Since this is a user-managed calendar, they have no access to AD/groups and with the turnover I mentioned, they add/remove users almost weekly so after the initial assignment, it will be done per user anyway.

As always, PowerShell to the rescue!

First, we'll build a CSV with "Name" as the heading and the list of users like so:

Or if you're copying perms from another calendar, you can export the names and access rights.

In the Exchange Management Shell (EMS) run the following:

Get-MailboxFolderPermission "Buddy Guy:\Calendar\Dept Calendar" | where {$_.accessrights -eq 'editor'} |select user,accessrights | Export-Csv C:\Scripts\caleditors.csv -NoTypeInformation

The above cmdlet will grab all users who have editor rights on the calendar.

Get-MailboxFolderPermission "Buddy Guy:\Calendar\Dept Calendar" | where {$_.accessrights -eq 'reviewer'} |select user,accessrights | Export-Csv C:\Scripts\calreviewers.csv -NoTypeInformation

The above cmdlet will grab all users who have reviewer rights on the calendar.

**Note** Change "Buddy Guy:\Calendar\Dept Calendar" to the user and calendar you're working with and the path and filename you prefer.

Then, format those CSVs like the example above, with just the names.

Next, run the following:

Import-Csv C:\Scripts\caleditors.csv | foreach {add-MailboxFolderPermission -Identity "Buddy Guy:\Calendar\NEW Dept Calendar" -User $_.name -AccessRights Editor}

This will assign editors on the new calendar.

Import-Csv C:\Scripts\calreviewers.csv | foreach {add-MailboxFolderPermission -Identity "Buddy Guy:\Calendar\NEW Dept Calendar" -User $_.name -AccessRights Reviewer}

This will assign reviewers on the new calendar.

**Note** Change "Buddy Guy:\Calendar\NEW Dept Calendar" to the user and new calendar you're working with and path and filename to the where you saved your CSVs.

Now, if you check permissions on the new calendar, you'll see that all of those editors and reviewers have been set. You can run Get-MailboxFolderPermission "Buddy Guy:\Calendar\NEW Dept Calendar" to grab all those perms.

A couple notes to finish up:

Users will still need to manually add the new Calendar to their Outlook.

If running on M365, it *may* take a while for the perms to take effect - I usually tell users to give it a day just to manage expectations - though it will usually work within a few minutes in OWA.

PS. I'll work on a script to copy and apply perms in one shot when I get a chance.

Teams - Get Team URL, Display Name and Owner

In my current environment, we do a lot of Litigation Holds...it's a medical org, so I mean A LOT of holds.

Not only do we have to do mailbox holds, but we have to place SharePoint, OneDrive and Teams on hold. SP and OneDrive are pretty easy as those URLs can be found in the Admin Center; not so for Teams URLs. We also need to find the owners of those Teams so they can be placed on hold as well.

I came up with a one-liner that will spit out all those URLs along with owners to make it easier to copy/paste them into the Hold in the eDiscovery Admin Center.

First, we'll need to make sure the results aren't truncated because some Teams can have several owners, and it will just show the "userna..." instead of the full name.

In M365 PowerShell, run:

$FormatEnumerationLimit=-1

Then, run:

Get-Team -User user@domain.com | foreach {Get-UnifiedGroup -Identity $_.groupid} |fl SharePointSiteUrl,managedby

**Note** user@domain.com is the account we're putting on hold and needing to find if they own any Teams. 

The output will look like this:

Once you have those links, you can paste them into your eDiscovery SharePoint/OneDrive/Teams holds instead of typing everything manually.

Happy holding!

M365 - Get-EXOMailbox : Error while querying REST service. HttpStatusCode=500

I previously wrote a post on how to create an M365 Remote PowerShell shortcut and would get errors when running EXOMailbox cmdlets. All other cmdlets worked fine, including Get-Mailbox, and connecting "manually" to Exchange Online worked fine. 

Btw, the reason you wanna use Get-EXOMailbox is it's a lot faster than Get-Mailbox...like, way faster.

The error I got was:

Get-EXOMailbox : Error while querying REST service. HttpStatusCode=500 ErrorMessage={"error":{"code":"InternalServerError","message":"The call to Microsoft Exchange Active Directory Topology service on server 'TopologyClientTcpEndpoint (localhost)' returned an error. Error details Active Directory server  is not available. Error message: Active directoryresponse: The LDAP server is unavailable..","innererror":{"message":"The call to Microsoft Exchange Active Directory Topology service on server 'TopologyClientTcpEndpoint (localhost)' returned an error. Error details Active Directory server  is not available. Error message: Active directory response: The LDAP server is unavailable..","type":"Microsoft.Exchange.Data.Directory.ADTransientException"}}}}

Apparently, this is a problem with the Exchange Online module 3 and above - I'm currently running 3.3.0. You'd think after 4 build updates MS would fix it but guess not.

The workaround is to connect to the Security & Compliance module before connecting to Exchange Online.

For instance, my PS Shortcut script would look like so:

$acctName="admin@domain.com"
$orgName="domain.com"
#Azure Active Directory
Connect-MsolService
#SharePoint Online
Connect-SPOService -Url https://exchangeitup-admin.sharepoint.com
#Security & Compliance Center
Import-Module ExchangeOnlineManagement
Connect-IPPSSession -UserPrincipalName $acctName
#Exchange Online
Connect-ExchangeOnline -UserPrincipalName $acctName -ShowProgress $true
#Teams and Skype for Business Online
Import-Module MicrosoftTeams
Connect-MicrosoftTeams

Now, when you try to run the EXOMailbox cmdlets, it'll fire off correctly!

M365 - Create Remote PowerShell Shortcut (updated)

In a previous post, I showed how to create a Remote PowerShell shortcut for Office 365 using Remote PowerShell (RPS), to save you from having to type in the remote session every time you connect.

Since Microsoft has deprecated RPS in favor of REST API, I've created a new script that will import the needed sessions and connect to the following M365 services in one shot:

Azure AD
SharePoint Online
Exchange Online
Security & Compliance Center
Teams

First, you'll need the latest EXO (Exchange Online) module - 3.3.0 at the time of this writing - in order to connect to Security & Compliance, which you can get here.

Next, set your PowerShell execution policy - I use Unrestricted but you can use RemoteSigned:

Set-ExecutionPolicy Unrestricted

Now, we'll grab the other required modules:

MSOL

PowerShell Gallery | MSOnline 1.1.183.66

SharePoint

PowerShell Gallery | Microsoft.Online.SharePoint.PowerShell 16.0.24009.12000

Teams

Install .NET 4.7.2 or higher

Run the two following cmdlets:

Install-Module -Name PowerShellGet -Force -AllowClobber

Install-Module -Name MicrosoftTeams -Force -AllowClobber

**Note** We're using MSOL because it's more comprehensive than AzureAD

**Note** Security & Compliance uses the EXO module, so nothing is needed for that.

Next, copy the following block and paste it into Notepad:

$acctName="admin@domain.com"
$orgName="domain.com"
#Azure Active Directory
Connect-MsolService
#SharePoint Online
Connect-SPOService -Url https://exchangeitup-admin.sharepoint.com
#Security & Compliance Center
Import-Module ExchangeOnlineManagement
Connect-IPPSSession -UserPrincipalName $acctName
#Exchange Online
Connect-ExchangeOnline -UserPrincipalName $acctName -ShowProgress $true
#Teams and Skype for Business Online
Import-Module MicrosoftTeams
Connect-MicrosoftTeams

**Note** Change $acctName="admin@domain.com" and $orgName="domain.com" to your admin account and domain name.

Save it as a .ps1 with a name like Connect-365.ps1 to somewhere like C:\Scripts

Next, create a PowerShell shortcut anywhere, like on your Desktop:

Right-click the Desktop > New > Shortcut

In the location field, enter:

Powershell.exe

Click Next

Give it a name like O365RemotePS and click Finish.

Right-click the new O365RemotePS shortcut, and go to Properties.

In the Target field, add the following to the end of the line:

-NoExit -File "C:\Scripts\Connect-365.ps1"

It will look like so:

C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoExit -File "C:\Scripts\Connect-365.ps1"

Click OK.

Run your new shortcut as your admin user, and you'll get a windows creds prompt and then the account picker for your Microsoft 365 organization.

**Note** You will get an account picker for each service depending on your security config in M365. 

**Note** Enter creds in the username@orgname.com format.

Once it starts the remote session, you'll be able to run your M365 cmdlets.

I Wrote A Book...

 ...a second book actually! My first book Exchange 2016 Resource Forest Setup can be found here. 

My new eBook Exchange N00b is live on Amazon Kindle. 

You can grab it here: Amazon.com: Exchange N00b eBook

This book is designed to guide fledgling (n00b) Exchange Administrators through the day-to-day management of an Exchange Environment.

The book applies to Exchange 2013/2016/2019, since not much has changed in terms of the GUI interface and basic PowerShell cmdlets and is broken down into sections so you can easily refer to the scenario you find yourself in.

Exchange - 2016 ObjectType '9b026da6-0d3c-465c-8bee-5199d7165cba' Can't Be Resolved

After installing a Server 2016 or Server 2019 Domain Controller into your existing Exchange environment, you'll get the following error prompt when trying to edit/view Mailbox Delegation or edit a Distribution Group in the EAC.

The object exchangeitup.com / Mailboxes / Shared Mailboxes / HR Shared Mailbox has been corrupted, and it's in an inconsistent state. The following validation errors happened:

The access control entry defines the ObjectType '9b026da6-0d3c-465c-8bee-5199d7165cba' that can't be resolved ..

This is because when promoting the server it extends the Schema, which breaks Exchange ACLs. 

**Note** It extends the Schema just by promoting the DC, even when leaving the Domain/Forest Functional Level as is.

Most online resources say restart IIS or restart the Exchange Server(s). Don't do that! Bouncing IIS will cause client disconnects, and restarting a whole server? Just...no.

A quicker and safer fix is to recycle the ECP pools on each Mailbox server by running the following cmdlet in the Exchange Management Shell:

Restart-WebAppPool -Name MSExchangeECPAppPool

Now refresh or close and reopen the EAC and it works!

Exchange - Dealing With Mailbox Retention In A Resource Forest

If you follow my blog, you know I run an Exchange Resource Forest (which really sucks). A quick rundown on the innerworkings of a Resource Forest involves syncing Active Directory accounts from the Accounts Forest (your main domain) to the Resource Forest (your Exchange domain) using a dirsync service like FIM/MIM. You can also see how it's set up in my post from a while back.

The problem we have in my environment is: government regulations require a certain retention time period on mailbox and mailbox items because of the type of business we do. This causes a big problem when removing AD accounts when employees leave the company. If you delete the AD account in the Accounts Forest, you'll end up deleting the AD account and mailbox along with it in the Resource Forest as soon as MIM completes its next scheduled sync.

So, I've come up with a way to be able to delete the user in the Accounts Forest, while keeping the mailbox intact for retention/historical data...this is also helpful when an employee leaves, and you've set Full Access on their mailbox for other users (like their previous manager) who need that data.

Move the AD Account In The Resource Forest

First, we'll create an Organizational Unit (OU) in the Resource Forest that isn't included in the MIM-sync, which will be outside the sync'd container(s); we'll call it something like "Disabled Mailboxes" - clever, eh?

Then, we'll move the soon-to-be-deleted user into this new OU. This will effectively "break" the sync, since MIM in the Accounts Forest will no longer "see" the AD account in the Resource Forest, because it's not in the sync'd OU any longer.

Convert The Linked Mailbox

Next, we'll convert the Linked Mailbox to a User Mailbox. We need to do this because deleting the AD account in the Accounts Forest (also called the "Linked Master Account") will also delete the mailbox along with it. So, converting to a User Mailbox will unlink the mailbox leaving it only connected to the Resource Forest Account.

To convert the Linked Mailbox to a User mailbox, fire up the Exchange Management Shell (EMS) and run:

Get-Mailbox "Account to be deleted name" | Set-User -LinkedMasterAccount $null

**Note** Change "Account to be deleted name" to the name of the account you're deleting.

The above cmdlet will wipe the Linked Master Account from the mailbox, and if you check the mailbox in the EAC or with the EMS, it will show "User Mailbox" as the Mailbox Type, now.

Delete The Accounts Forest AD Account

Now that we have moved our Resource Forest Account and converted the mailbox, nothing is tying it to the original/main account in the Accounts Forest. You can now delete the AD account from the Accounts Forest.

Once you've deleted it, depending on your MIM sync interval (usually 30 minutes) to verify that everything still works, go check the account/mailbox in the Resource Forest and you'll see that it's still there. 

**Note** The AD account in the Resource Forest will still be disabled, as is standard with Linked Mailboxes, so Mailbox Delegation will still work, but no one will be able to directly log in to it. If that needs to happen, you can just enable the AD account.

Now, you can keep those mailboxes around for however long you need them, while keeping a clean and secure Accounts Forest!

Exchange - Set Forwarding On Mailboxes In Bulk From CSV

My company just sold off a couple of its business units and the sales agreement says offboarding users must keep their legacy email access for 6 months. But, they also need messages forwarded to their new (external) email addresses.

Normally, you could do this by OU (Organizational Unit) but in my case, not everyone in that OU is leaving and management didn't want to create a new OU since GPOs and ACLs are already linked to said OU.

So, we need to be able to import a CSV of those specific users and set the forwarding in one shot...I have about 450 users in my list, that would take a long time doing it manually :)

Create A New Send Connector

First, we need to create a dedicated Send Connector going out to the new external domain. This allows us to forward without creating External Contacts for all of the users - much easier:

In the EAC, navigate to Mail Flow, Send Connectors, +

In the New Send Connector window, give it a name like "External Forward" and click Next

Leave MX record selected and click Next

Click the + and under the FQDN type the domain name for the external contact, click Save, and Next

For the Source Server, click the + and select your Edge server if you have one, or your Mailbox servers (all of them) and click Ok, then Finish

Enable Verbose Logging on the Connector:

You'll want full logging on the connector so you can check the SMTPSend protocol logs later to verify successful sending.

In the Exchange Management Shell (EMS), run the following:

Set-SendConnector "External Forward" -ProtocolLoggingLevel Verbose

Optional:

If the external domain requires it, you'll need to enable forced TLS, else messages will be dropped.

In the EMS run the following:

Get-SendConnector "External Forward" | Set-SendConnector -RequireTLS $true

Create New Remote Domain

Next, we need to create a new Remote Domain in Exchange - this will allow incoming messages to hit our Exchange servers and forward/relay on to the new, external domain.

Fire up the Exchange Management Shell (EMS) and run:

New-RemoteDomain -DomainName *.externaldomain.com -Name ExternalDomain

**Note** Change "*.externaldomain.com" to the SMTP domain name of the external domain (using the wildcard will ensure any subdomains are included) and change "ExternalDomain" to the name of the external domain

Next, we need to allow messages to be forwarded to the new remote domain, by running:

Set-RemoteDomain "ExternalDomain" -AutoForwardEnabled $True

**Note** Change "ExternalDomain" to the name you gave it in the earlier cmdlet

Set Forwarding

Now, we'll create our CSV with Email and Forward as the headers and the users' current email and new external email address under like so:

Lastly, we'll run the cmdlet to set the forwarding on the user in the CSV:

$users=Import-csv C:\Temp\emails.csv

ForEach($user in $users){Set-Mailbox $user.email -DeliverToMailboxAndForward $false -ForwardingSMTPAddress $user.forward}

**Note** This cmdlet sets the ForwardingSMTPAddress switch, NOT ForwardingAddress. You can refer to my previous post to see the difference between those, but in a nutshell ForwardingSMTPAddress is what you use when using a dedicated Send Connector and Remote Domain.

**Note** This will set the messages to be forwarded but not delivered to the current mailboxes, which is what we want since those people are leaving the company.

Now, all incoming messages to those users will be routed to their new external domain without landing in their current mailboxes!

Skype For Business - Showing Out of Office For User Who Is Not OOF

In my Exchange 2016/SFB 2015 environment, I have a user named Mary, where OOF is showing enabled/turned on when looking her up in SFB and Outlook:

The problem is: she isn't Out Of Office, she's there, sitting at her desk.

In the Exchange Management Shell (and the ECP in "manage another user") automatic reply status shows as disabled:

Well, we know the EMS doesn't lie, so it's gotta be client-related.

We tried turning off Outlook cached mode, creating a new local mail profile, and deleting Mary's SFB cache...nothing worked.

On a hunch, I checked Mary's calendar and one of her direct reports named Laura, sent her own OOF as an invite to Mary, which added it to Mary's calendar, therefore showing Mary as OOF:

That's bad practice, and totally confusing for other users. 

To fix it, Mary needs to delete Laura's calendar event and it will remove Mary from looking like she's Out Of Office.