I've been running Exchange 2016 environment for a little while now, but I finally had to export a mailbox to PST.
I kept getting an error in both the EAC (Exchange Admin Console) and the Shell.
The error:
New-MailboxExportRequest -Mailbox "staceybranham" -FilePath \\server1\PSTs\stacey.pst
Couldn't find the Enterprise Organization container
This error is usually thrown when you haven't added your admin account to be allowed to import/export mailboxes.
The "funny" thing is: I had already run the following cmdlet to do so:
New-ManagementRoleAssignment -Role "Mailbox Import Export" -User Admin
But, every time I ran the export it would error out.
What I forgot to do, was close and reopen both the EMS (Exchange Management Shell) and the EAC.
After that, I was able to run the export without the error.
By the way this is a stupid error since the Enterprise Organization Container obviously exists, or there'd be no Exchange.
How about next time just have it say "you don't have a Management Role set for this account" or something to that effect? It would save a lot of digging around to ensure that the containers are set correctly...ugh
Microsoft Enterprise Messaging tips, tricks, and general help.
Wednesday, December 21, 2016
Saturday, December 10, 2016
Kemp Load Balancer: Change from One-Arm to a Two-Arm Configuration
During the planning and set up phase of my current Exchange 2016 project, management stipulated that no external Exchange access was to be allowed - less pain for me!
Several months later after the environment is in production, they now want to allow external access...go figure :)
There were several issues here: we have a one-arm Kemp virtual load balancer installed on the internal LAN; the firewall/security team didn't want to NAT anything internally (they will only NAT to the DMZ); NATing to the DMZ requires another load balancer (costing more money) or a reverse proxy solution, which would then have to redirect back to the internal Kemp...not a graceful solution.
I decided that it would save a ton of trouble just switching the existing Kemp from a one-arm config to a two-arm.
Meaning, it will have one NIC in the internal LAN, and one NIC in the DMZ thereby allowing both internal and external client connections.
The steps are pretty straightforward and some of the things will be specific to your environment.
**Disclaimer** You should not perform this on a production environment as it can result in loss of client connections...but if you have no choice, go for it!
**Note** You should take a back up the current config before starting the changes.
**Note** You will have to restart the load balancer during the changes, so it's best to do this outside business hours.
Changing a One-Arm Kemp LoadMaster to a Two-Arm Configuration:
First, you'll need to add a second NIC to your load balancer VM and assign it to another network, such as your DMZ.
In my case, I already had two NICs defined (eth0 and eth1) but eth1 wasn't being used, so I set the DMZ network on that one.
My initial setup was eth0 on the internal 10.x.69.x network, which allowed only internal connections.
Now, I've added the DMZ network to eth1, which is on the 10.x.68.x network.
In our network, the DMZ isn't routable back to internal, so we need to do some extra configuration on the Kemp itself.
We'll need to ensure that we don't have asymmetric back end routes, which occurs in two-arm setups, by enabling "Subnet Originating Requests". We'll also need to "Enable Non-local real Servers".
Both of the above settings are found in Systems Configuration > Miscellaneous Options > Network Options.
Put a checkmark for both Subnet Originating Requests and Enable Non-local real Servers
Next, we'll need to configure the WUI (Web User Interface) to be reachable from the internal NIC.
Navigate to Certificates > Security > Remote Access
Here, next to "Allow Web Admin Access" we'll use the drop-down box to select our Internal NIC, which on my system is eth0.
We'll then set the "Admin Default Gateway" to our Internal LAN Gateway on the 10.x.69.x subnet; this ensure we can still get to the WUI once we change our routes later.
**Note** Immediately after you change the Admin Default Gateway, you will lose WUI access, and the next step must be from the console.
Next we'll need to set the Default Gateway to the DMZ network, since our DMZ is not routable to the internal LAN.
**Note** This step requires console access because since we changed the admin gateway we lost WUI access. While in console, you will also need to reboot your device after changing the gateway.
**Note** The Default Gateway is a global setting, there isn't an option for setting it on each NIC.
In the console, login to your LoadMaster, and navigate to (4) Basic Setup.
Next, we'll choose option (4) Default Gateway Configuration.
Here, we'll set the GW to your DMZ Gateway, in my case this is my 10.x.68.x subnet then TAB down to OK and hit Enter when finished.
Next select (q) return to previous menu. Back at the main menu select (8) Reboot.
Now, the last step is creating our new Virtual Services (VS) and Virtual IP Address (VIP) on the DMZ interface. This is done exactly like you configured your internal VIP and VS, by using the Exchange templates and existing Exchange cert.
Go to Virtual Services > Add New
Assign your Virtual Address (VIP) which will be on your DMZ subnet.
Give it a Service Name - this is not optional.
I added the word "external" to the service name that auto-populates: when adding another VS using the same template, because it will automatically try to use the same service name as the already configured internal VS and they cannot be the same.
In the Use Template drop-down, select Exchange 2016 HTTPS Reencrypted...the port will magically change to 443.
Click "Add this Virtual Service"
Go to Virtual Services > View/Modify Services, you should now see your second VS on the DMZ VIP.
In my example, my DMZ Virtual Services are on the 10.28.68.135 VIP and my Internal VS are on the 10.28.69.167 VIP.
Now we need to configure the new DMZ SubVS just like you did for you internal Virtual Services, and then assign your Exchange Certificate.
If you need a refresher on how to configure those SubVS and certs, follow Gareth Gudger's (SuperTekBoy) awesome post here; you'll just be doing the same steps but this time it'll be on the newly created DMZ Virtual Services.
Once you finish those settings, you're ready to test!
From an external machine try to hit your OWA, set up Outlook, and ActiveSync on your device...if your NATing and firewall rules are set correctly you should have no problems with external access.
Now to get less sleep since you can check email from anywhere!
Several months later after the environment is in production, they now want to allow external access...go figure :)
There were several issues here: we have a one-arm Kemp virtual load balancer installed on the internal LAN; the firewall/security team didn't want to NAT anything internally (they will only NAT to the DMZ); NATing to the DMZ requires another load balancer (costing more money) or a reverse proxy solution, which would then have to redirect back to the internal Kemp...not a graceful solution.
I decided that it would save a ton of trouble just switching the existing Kemp from a one-arm config to a two-arm.
Meaning, it will have one NIC in the internal LAN, and one NIC in the DMZ thereby allowing both internal and external client connections.
The steps are pretty straightforward and some of the things will be specific to your environment.
**Disclaimer** You should not perform this on a production environment as it can result in loss of client connections...but if you have no choice, go for it!
**Note** You should take a back up the current config before starting the changes.
**Note** You will have to restart the load balancer during the changes, so it's best to do this outside business hours.
Changing a One-Arm Kemp LoadMaster to a Two-Arm Configuration:
First, you'll need to add a second NIC to your load balancer VM and assign it to another network, such as your DMZ.
In my case, I already had two NICs defined (eth0 and eth1) but eth1 wasn't being used, so I set the DMZ network on that one.
My initial setup was eth0 on the internal 10.x.69.x network, which allowed only internal connections.
Now, I've added the DMZ network to eth1, which is on the 10.x.68.x network.
In our network, the DMZ isn't routable back to internal, so we need to do some extra configuration on the Kemp itself.
We'll need to ensure that we don't have asymmetric back end routes, which occurs in two-arm setups, by enabling "Subnet Originating Requests". We'll also need to "Enable Non-local real Servers".
Both of the above settings are found in Systems Configuration > Miscellaneous Options > Network Options.
Put a checkmark for both Subnet Originating Requests and Enable Non-local real Servers
Next, we'll need to configure the WUI (Web User Interface) to be reachable from the internal NIC.
Navigate to Certificates > Security > Remote Access
Here, next to "Allow Web Admin Access" we'll use the drop-down box to select our Internal NIC, which on my system is eth0.
We'll then set the "Admin Default Gateway" to our Internal LAN Gateway on the 10.x.69.x subnet; this ensure we can still get to the WUI once we change our routes later.
**Note** Immediately after you change the Admin Default Gateway, you will lose WUI access, and the next step must be from the console.
Next we'll need to set the Default Gateway to the DMZ network, since our DMZ is not routable to the internal LAN.
**Note** This step requires console access because since we changed the admin gateway we lost WUI access. While in console, you will also need to reboot your device after changing the gateway.
**Note** The Default Gateway is a global setting, there isn't an option for setting it on each NIC.
In the console, login to your LoadMaster, and navigate to (4) Basic Setup.
Next, we'll choose option (4) Default Gateway Configuration.
Here, we'll set the GW to your DMZ Gateway, in my case this is my 10.x.68.x subnet then TAB down to OK and hit Enter when finished.
Next select (q) return to previous menu. Back at the main menu select (8) Reboot.
Now, the last step is creating our new Virtual Services (VS) and Virtual IP Address (VIP) on the DMZ interface. This is done exactly like you configured your internal VIP and VS, by using the Exchange templates and existing Exchange cert.
Go to Virtual Services > Add New
Assign your Virtual Address (VIP) which will be on your DMZ subnet.
Give it a Service Name - this is not optional.
I added the word "external" to the service name that auto-populates: when adding another VS using the same template, because it will automatically try to use the same service name as the already configured internal VS and they cannot be the same.
In the Use Template drop-down, select Exchange 2016 HTTPS Reencrypted...the port will magically change to 443.
Click "Add this Virtual Service"
Go to Virtual Services > View/Modify Services, you should now see your second VS on the DMZ VIP.
In my example, my DMZ Virtual Services are on the 10.28.68.135 VIP and my Internal VS are on the 10.28.69.167 VIP.
Now we need to configure the new DMZ SubVS just like you did for you internal Virtual Services, and then assign your Exchange Certificate.
If you need a refresher on how to configure those SubVS and certs, follow Gareth Gudger's (SuperTekBoy) awesome post here; you'll just be doing the same steps but this time it'll be on the newly created DMZ Virtual Services.
Once you finish those settings, you're ready to test!
From an external machine try to hit your OWA, set up Outlook, and ActiveSync on your device...if your NATing and firewall rules are set correctly you should have no problems with external access.
Now to get less sleep since you can check email from anywhere!
Saturday, December 3, 2016
Exchange 2016 Management Shell Won't Connect Error: The WinRM Shell client cannot process the request
The other day after patching and bouncing my Exchange 2016 servers, one of them came back up pretty grumpy.
All services were running, and mailflow was healthy, but the Exchange Management Shell (EMS) threw WinRm errors when opening.
The Application and System Event Logs were full of warnings and errors too.
Luckily the event logs are little more helpful, because the error thrown in the EMS didn't really tell me what was actually going on.
EMS Error:
Connecting to EXCHMBX01.exchangeitup.com.
New-PSSession : [EXCHMBX01.exchangeitup.com] Processing data from remote server EXCHMBX01.exchangeitup.com failed
with the following error message: The WinRM Shell client cannot process the request. The shell handle passed to the WSMan Shell function is not valid. The shell handle is valid only when WSManCreateShell function completes successfully. Change the request including a valid shell handle and try again. For more information, see the about_Remote_Troubleshooting Help topic.
As you can see, processing data failed...WinRM...ok what does that mean? After it throws the error, the Shell will remote session over to another Exchange server, so at least you're not dead in the water.
Event Logs:
Log Name: Application
Source: MSExchange Front End HTTP Proxy
Date: 11/28/2016 10:16:41 PM
Event ID: 1003
Task Category: Core
Level: Error
Keywords: Classic
User: N/A
Computer: EXCHMBX01
Description:
[PowerShell] An internal server error occurred. The unhandled exception was: System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.Exchange.HttpProxy.RemotePowerShellProxyRequestHandler.ExposeExceptionToClientResponse(Exception ex)
at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.CompleteWithError(Exception ex, String label)
at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.<>c__DisplayClass29.b__28()
at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Action tryDelegate, Func`2 filterDelegate, Action`1 catchDelegate)
Log Name: Application
Source: ASP.NET 4.0.30319.0
Date: 11/28/2016 10:16:40 PM
Event ID: 1309
Task Category: Web Event
Level: Warning
Keywords: Classic
User: N/A
Computer: EXCHMBX01
Description:
Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 11/28/2016 10:16:40 PM
Event time (UTC): 11/29/2016 3:16:40 AM
Event ID: 90a2cfed15c84e1ab33affbe5cb62d96
Event sequence: 2
Event occurrence: 1
Event detail code: 0
Application information:
Application domain: /LM/W3SVC/1/ROOT/PowerShell-1-131248629080807714
Trust level: Full
Application Virtual Path: /PowerShell
Application Path: C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\PowerShell\
Machine name: EXCHMBX01
Process information:
Process ID: 9212
Process name: w3wp.exe
Account name: NT AUTHORITY\SYSTEM
Exception information:
Exception type: NullReferenceException
Exception message: Object reference not set to an instance of an object.
at Microsoft.Exchange.HttpProxy.RemotePowerShellProxyRequestHandler.ExposeExceptionToClientResponse(Exception ex)
at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.CompleteWithError(Exception ex, String label)
at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.<>c__DisplayClass29.b__28()
at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Action tryDelegate, Func`2 filterDelegate, Action`1 catchDelegate)
at Microsoft.Exchange.HttpProxy.Diagnostics.SendWatsonReportOnUnhandledException(Action methodDelegate, LastChanceExceptionHandler exceptionHandler)
at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.CallThreadEntranceMethod(Action method)
Log Name: System
Source: Microsoft-Windows-HttpEvent
Date: 11/28/2016 10:25:39 PM
Event ID: 15021
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: EXCHMBX01
Description:
An error occurred while using SSL configuration for endpoint 0.0.0.0:444. The error status code is contained within the returned data.
From the above logs, we finally see that there's something wrong with the HTTP/SSL config on the PowerShell virtual directory.
The Fix:
Close the Exchange Management Shell.
Open IIS Manager, and drill down to Servername > Sites. Right-click Exchange Back End and select Edit Bindings.
In the Site Binding window, select the HTTPS binding, and click Edit.
Under SSL Certificate, it will say Not Selected - meaning there's no certificate assigned to the HTTPS service on port 444, which was referenced in the System log Event 15021.
Click the drop-down and select your applicable certificate - this will usually the self-signed cert, but you can check your other working servers to see which one is assigned.
Click OK
Fire up the EMS and you should have a successful connection!
All services were running, and mailflow was healthy, but the Exchange Management Shell (EMS) threw WinRm errors when opening.
The Application and System Event Logs were full of warnings and errors too.
Luckily the event logs are little more helpful, because the error thrown in the EMS didn't really tell me what was actually going on.
EMS Error:
Connecting to EXCHMBX01.exchangeitup.com.
New-PSSession : [EXCHMBX01.exchangeitup.com] Processing data from remote server EXCHMBX01.exchangeitup.com failed
with the following error message: The WinRM Shell client cannot process the request. The shell handle passed to the WSMan Shell function is not valid. The shell handle is valid only when WSManCreateShell function completes successfully. Change the request including a valid shell handle and try again. For more information, see the about_Remote_Troubleshooting Help topic.
As you can see, processing data failed...WinRM...ok what does that mean? After it throws the error, the Shell will remote session over to another Exchange server, so at least you're not dead in the water.
Event Logs:
Log Name: Application
Source: MSExchange Front End HTTP Proxy
Date: 11/28/2016 10:16:41 PM
Event ID: 1003
Task Category: Core
Level: Error
Keywords: Classic
User: N/A
Computer: EXCHMBX01
Description:
[PowerShell] An internal server error occurred. The unhandled exception was: System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.Exchange.HttpProxy.RemotePowerShellProxyRequestHandler.ExposeExceptionToClientResponse(Exception ex)
at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.CompleteWithError(Exception ex, String label)
at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.<>c__DisplayClass29.
at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Action tryDelegate, Func`2 filterDelegate, Action`1 catchDelegate)
Log Name: Application
Source: ASP.NET 4.0.30319.0
Date: 11/28/2016 10:16:40 PM
Event ID: 1309
Task Category: Web Event
Level: Warning
Keywords: Classic
User: N/A
Computer: EXCHMBX01
Description:
Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 11/28/2016 10:16:40 PM
Event time (UTC): 11/29/2016 3:16:40 AM
Event ID: 90a2cfed15c84e1ab33affbe5cb62d96
Event sequence: 2
Event occurrence: 1
Event detail code: 0
Application information:
Application domain: /LM/W3SVC/1/ROOT/PowerShell-1-131248629080807714
Trust level: Full
Application Virtual Path: /PowerShell
Application Path: C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\PowerShell\
Machine name: EXCHMBX01
Process information:
Process ID: 9212
Process name: w3wp.exe
Account name: NT AUTHORITY\SYSTEM
Exception information:
Exception type: NullReferenceException
Exception message: Object reference not set to an instance of an object.
at Microsoft.Exchange.HttpProxy.RemotePowerShellProxyRequestHandler.ExposeExceptionToClientResponse(Exception ex)
at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.CompleteWithError(Exception ex, String label)
at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.<>c__DisplayClass29.
at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Action tryDelegate, Func`2 filterDelegate, Action`1 catchDelegate)
at Microsoft.Exchange.HttpProxy.Diagnostics.SendWatsonReportOnUnhandledException(Action methodDelegate, LastChanceExceptionHandler exceptionHandler)
at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.CallThreadEntranceMethod(Action method)
Log Name: System
Source: Microsoft-Windows-HttpEvent
Date: 11/28/2016 10:25:39 PM
Event ID: 15021
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: EXCHMBX01
Description:
An error occurred while using SSL configuration for endpoint 0.0.0.0:444. The error status code is contained within the returned data.
From the above logs, we finally see that there's something wrong with the HTTP/SSL config on the PowerShell virtual directory.
The Fix:
Close the Exchange Management Shell.
Open IIS Manager, and drill down to Servername > Sites. Right-click Exchange Back End and select Edit Bindings.
In the Site Binding window, select the HTTPS binding, and click Edit.
Under SSL Certificate, it will say Not Selected - meaning there's no certificate assigned to the HTTPS service on port 444, which was referenced in the System log Event 15021.
Click the drop-down and select your applicable certificate - this will usually the self-signed cert, but you can check your other working servers to see which one is assigned.
Click OK
Fire up the EMS and you should have a successful connection!