Exchange 2013 Alert: "The A/V Edge service is misconfigured or isn't operating correctly"

After migrating from Lync to Skype For Business, we started getting SCOM Alerts for the A/V Edge Services on our Exchange servers.
Checking the Application Event Logs for UM events at that timestamp, I noticed Events ID's 1692 and 1438 for MSExchange Unified Messaging - UMCore.
It seems that Exchange was still communicating with an old Lync Edge rather than the new S4B Edge Pool.
Unfortunately the TechNet link in the SCOM alert doesn't tell you squat, except to check the Health Set. 
Luckily the Event 1438 tells you exactly what needs to be changed...I wish all events would be so kind :)

SCOM Alert Details:

Alert: {2}
Category: Custom
Time Raised: 3/25/2016 1:48:30 PM
State: New
Source: MBX-03 - UM.Protocol
Path: MBX-03.domain.com;MBX-03.domain.com
Last modified by: System
Last modified time: 3/25/2016 1:49:31 PM
Description: The A/V Edge service is misconfigured or isn't operating correctly
Knowledge: http://technet.microsoft.com/en-us/library/ms.exch.scom.UM.Protocol(EXCHG.150).aspx?v=15.0.1156.6

Event ID 1692:

Log Name:        Application
Source:              MSExchange Unified Messaging
Date:                  3/25/2016 1:48:30 PM
Event ID:           1692
Task Category:  UMCore
Level:                Warning
Keywords:         Classic
User:                  N/A
Computer:         MBX-03
Description:
The Microsoft Exchange Unified Messaging service couldn't establish a media channel with the current Lync Server Audio/Video Edge resources associated with LyncEdge.domain.com:5061. Diagnostic information: No diagnostic information is available.

Event ID 1438:

Log Name:       Application
Source:             MSExchange Unified Messaging
Date:                 3/25/2016 1:48:30 PM
Event ID:          1438
Task Category:  UMCore
Level:                Warning
Keywords:         Classic
User:                  N/A
Computer:         MBX-03.domain.com
Description:
The Microsoft Exchange Unified Messaging service on the Mailbox server has been configured to automatically use the Lync Server A/V Edge resources associated with LyncEdge.domain.com:5061'. Inbound and outbound calls involving remote users (located outside the enterprise) might be failing using the current Lync Server A/V Edge resources. To correct the issue, set the SIPAccessService property using the Set-UMService cmdlet. The Microsoft Exchange Unified Messaging  service will start using the Lync Server A/V Edge resources corresponding to the new value.

The Fix:

Fire up the Exchange Management Shell (EMS) and run:

 

Get-UMService | Select Name, Status, SIPAccessService | ft -AutoSize

Most likely your SIPAccessService field is blank, so we'll need add the new S4B Edge Pool in there.

Run the following cmdlet:

Get-UMService | Set-UMService -SIPAccessService Skype4BEdgePool.domain.com:5061​

**Note** Change Skype4BEdgePool.domain.com to your Skype For Business Edge Pool name.

 

Restart Unified Messing Service on each Exchange Mailbox Server.

 

No more annoying alerts!

Exchange 2010/2013 Event ID 1021 Warning "Receive connector rejected an incoming connection from IP address"

We started getting a few 1021 Warnings on our Exchange 2013 servers after ramping up our Anonymous Relay traffic. This means that you have a large number of connections from the same IP.

The default maximum concurrent connections from the same IP for Receive Connectors is 20...we needed that to be higher so it wouldn't reject messages!

**Note** Make sure you recognize the IP that being presented in the warning; it could be a spammer, and you wouldn't want to make the following changes unless you're sure that the volume of connections is normal.

The full warning is:

Log Name:        Application
Source:              MSExchangeFrontEndTransport
Date:                 3/17/2016 11:25:15 AM
Event ID:          1021
Task Category: SmtpReceive
Level:               Warning
Keywords:        Classic
User:                 N/A
Computer:         MBX1.exchangeitup.com
Description:
Receive connector Internal Anonymous Relay rejected an incoming connection from IP address x.x.x.x. The maximum number of connections per source (20) for this connector has been reached by this source IP address.

To check the current limits on Receive Connectors, fire up the EMS and run:

Get-ReceiveConnector | select name, *MaxInbound*

You'll get something like this:

As you can see, my "Internal Anonymous Relay" connectors are set to 20 for MaxInboundConnectionPerSource, and 2 for MaxInboundConnectionPercentagePerSource.

We want to bump those up for the "Internal Anonymous Relay" on all servers, so we'll run the following cmdlet:

Get-ReceiveConnector *\"Internal Anonymous Relay" | Set-ReceiveConnector -MaxInboundConnectionPerSource 500 -MaxInboundConnectionPercentagePerSource 100

**Note** You'll need to change "Internal Anonymous Relay" to your Receive Connector name.

This will set maximum number of inbound connections from a single source IP to 500 and maximum amount (in percentage) of connections remaining that can get used by a single source IP to 100.

**Note** You can set the max number of connections to a higher value or unlimited but you'll want to go in small increments to test for adverse effects.

Now run the Get-ReceiveConnector | select name, *MaxInbound* cmdlet again, and you'll get:

Now you'll stop getting warnings and connections won't be rejected.

Exchange 2013 Export Distribution Group Owners Report To CSV

I recently had a request to export the owners of a bunch of distribution groups so one of our Business Unit's managers could review them and see who should or shouldn't be group owners.

Pulling such a report is easy with PowerShell!

We have our distro groups for each Business Unit separated in OU's, so I'm including that part in the cmdlet.

Fire up the Exchange Management Shell (EMS) and run:

Get-DistributionGroup -OrganizationalUnit "domain.com/Distribution Lists/Business Unit" | Select-object Name,@{label="ManagedBy";expression={[string]($_.managedby | foreach {$_.tostring().split("/")[-1]})}},Primarysmtpaddress | Export-Csv "C:\BU_Dist_Owners.csv"

**Note** Change "domain.com/Distribution Lists/Business Unit" to the OU you're running it against, and change "C:\BU_Dist_Owners.csv" to the name and path of your choosing.

You'll get a nice report with the Group Name, Owner, and Primary SMTP Address.

To run a report for all Distribution Groups in your environment, just leave out the -OrganizationalUnit part, like so:

Get-DistributionGroup | Select-object Name,@{label="ManagedBy";expression={[string]($_.managedby | foreach {$_.tostring().split("/")[-1]})}},Primarysmtpaddress | Export-Csv "C:\All_Dist_Owners.csv"

**Note** Again, change "C:\All_Dist_Owners.csv" to your path.

Happy reporting!

Exchange 2013 Security Update for Exchange CU11 Error: RPC Over HTTP Autoconfig (Event ID 2002)

We recently installed Security update for Microsoft Exchange Server CU11 (KB3124557) on our multi-role Exchange 2013 servers, and after the rollout, one server started throwing MSExchange RPC Over HTTP Autoconfig (Event ID 2002) errors, and it happened every 15 minutes. To go along with the errors, some users reported that Outlook started reporting "Microsoft Exchange administrator has made a change that requires you quit and restart Outlook".

The problem (it seems) that the update itself is buggy; I've seen other reports on forums that it broke their Exchange 2013/2016 machines.

The full error is:

Log Name:      Application
Source:        MSExchange RPC Over HTTP Autoconfig
Date:          3/2/2016 4:49:02 PM
Event ID:      2002
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      MBX-03.domain.com
Description:
An unexpected failure has occurred. The problem will require administrator intervention. The service will retry in 00:15:00. Diagnostic information:
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.File.InternalCopy(String sourceFileName, String destFileName, Boolean overwrite, Boolean checkHost)
   at Microsoft.Exchange.Servicelets.RPCHTTP.RPCHTTPServicelet.CopyCustomRpcProxyFiles()
   at Microsoft.Exchange.Servicelets.RPCHTTP.RPCHTTPServicelet.ConfigureRpcForMailboxServer(Boolean isAlsoFrontEnd)
   at Microsoft.Exchange.Servicelets.RPCHTTP.RPCHTTPServicelet.Work()
The process cannot access the file 'C:\Windows\System32\RpcProxy\RpcProxyShim.dll' because it is being used by another process.

**Note** The server name has been changed to protect the innocent.

I was hesitant to remove the update, because I've seen in the past that doing so sometimes has terrible effects on Exchange, but I had no other choice.

I uninstalled the KB3124557 update, and the installer called for the original CU11 install files, so make sure you have those handy.

After uninstalling, it disabled and stopped all Exchange services - even on other Exchange servers, so you'll definitely want to do this in a maintenance window.
I thought no big deal it will re-enable them when we re-install the update. Re-installing the update did not fix the services.

So the only course of action was to entirely remove the update, and manually switch the services to automatic and start them.

**Note** We uninstalled the update on all of our Exchange servers, so they are all on matching patch levels.

These are the Exchange services you'll need to enable and start:

Windows Management Instrumentation
World Wide Web Publishing Service
Tracing Service for Search in Exchange
Remote Registry
Performance Logs & Alerts
IIS Admin Service
Application Identity
Microsoft Filtering Management Service
Microsoft Exchange Unified Messaging
Microsoft Exchange Transport Log Search
Microsoft Exchange Transport
Microsoft Exchange Throttling
Microsoft Exchange Service Host
Microsoft Exchange Search Host Controller
Microsoft Exchange RPC Client Access
Microsoft Exchange Replication
Microsoft Exchange POP3
Microsoft Exchange Mailbox Transport Submission
Microsoft Exchange Mailbox Replication
Microsoft Exchange Mailbox Assistants
Microsoft Exchange Information Store
Microsoft Exchange IMAP4
Microsoft Exchange Health Manager
Microsoft Exchange Frontend Transport
Microsoft Exchange Search
Microsoft Exchange EdgeSync
Microsoft Exchange Diagnostics
Microsoft Exchange Mailbox Transport Delivery
Microsoft Exchange DAG Management
Microsoft Exchange Anti-spam Update
Microsoft Exchange Active Directory Topology

We restarted the Exchange servers themselves for good measure, and everything is back to normal. I'm just gonna wait until Exchange 2013 CU12 drops and install that...to hell with this update :)