System Center Operations Manager 2012 R2 - System Center Management Health Service Unloaded System Rule(s) - Windows Server 2003/2008 (non R2) missing PowerShell 2.0

I know its been awhile since the last post, so here it goes...

Issue:

This week we are dealing with a specific error that caused multiple agents to throw the alert monitor, System Center Management Health Service Unloaded System Rule(s) with the following description.

The System Center Management Health Service ????????-????-????-????-???????????? running on host <computername> and serving management group with id {????????-????-????-????-????????????} is not healthy. Some system rules failed to load.

This caused the agents to go into a "grey" state.  Review the Operations Manager log on an affected agent computer and look for errors with event id 4513 with the following description.

The callback method DeliverDataToModule failed with exception "Could not load file or assembly 'System.Management.Automation, Version=1.0.0.0, Culture=neutral, PublicKeyToken=????????????????' or one of its dependencies. The system cannot find the file specified." in the module Microsoft.EnterpriseManagement.Modules.PowerShell.PowerShellProbeActionModule.

Workflow: Microsoft.IntelligencePacks.Types.HealthServiceProxyConfiguration 

The common factor among the affected computer agents was all Windows Server 2003 and Windows Server 2008 (non R2) which were missing Powreshell 2.0.

Solution and workaround:

The solution is to override the Advisor Proxy Setting Rule or install Powershell 2.0.

It was not until after I had called Microsoft Support which were quick to find the cause and provide a solution to this issue, that I found this issue being discussed on the TechNet Forum - Grey Agents after New Management Pack with id:"Microsoft.IntelligencePacks.Types", version:"7.0.9781.0" received. - and the Microsoft Azure forum - Windows Server 2003 'grey' due to missing Powershell system rule Microsoft.IntelligencePacks.Types.HealthServiceProxyConfiguration.

System Center Operations Manager 2012 - Installing Certificates on an Agent

In order to monitor servers not in a trusted domain, System Center Operations Manager uses certificates to authenticate agent communication.  Therefore, you'll need a certificate authority server to submit the certificate request and then download the certificate. You can use either a domain Certificate Authority (CA) server or a standalone CA if your environment already has that in place.  I used a standalone.

Requirements
Certificate Authority Server
Port 5723 open from Server01 to the Management Server

1. Create the request *.inf file.
   
2. From the command prompt (server01) create the request file. *Must be done on the computer the certificate is for
   
   certreq -new -f server01.inf server01.req
3. Copy the request file to the CA server and submit request file
   
   certreq -submit -f server01.req
1. Click OK to the dialogue box that pops up.
2. Take note of the RequestId
4. On the CA Server - launch the Certificate Authority MMC (certsrv.msc) - Select Pending Requests, right-click the certificate request with the RequestId from the previous step select All Tasks - Issue
   
   
5. Retrieve the certificate (CA Server)
   
   certreq -retrieve -f 4 server01.cer
1. Click OK
2. Copy the certificate file (server01.cer) to the server you are installing the agent on (server01)
6. On Server01 - Install the certificate in the Local Computer - Personal Certificate store
7. On Server01 - Export the certificate you just imported with the Private Key (server01-exported.pfx)
8. Retrieve the CA certificate - This certificate will need to be installed in the Trusted Root Certification Authorities store on the Agent computer (server01)
1. On the CA Server navigate to http://localhost/certsrv/
2. Click on Download a CA certificate, certificate chain, or CRL
3. Click on Download CA certificate
4. Save this file (cacert.cer) and copy it over to the server the agent will be installed on (server01)
9. On Server01 - Install the CA Certificate in the Trusted Root Certification Authorities
1. Launch an MMC console and add the Certificates snap-in| for the local computer
2. Right-click the Certificates folder under Trusted Root Certification Authorities select All Tasks -> Import
3. Browse to the CA certificate saved from the previous step (cacert.cer)
4. Click on Next, Next, and Finish
10. Install the System Center Operations Manager Agent
1. The files can be found on the Management folder
   C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\AgentManagement\
   64 bit agent - amd64 folder
   32 bit agent - x86 folder
2. Run the MOMCertImport.exe to import the certificate from step 7
   
   MOMCertImport.exe server01-exported.pfx
3. Enter the password set from step 7
4. If you're updating the certificate run the remove command first
   
   MOMCertImport.exe /Remove
11. Restart the Microsoft Monitoring Agent service

System Center Operations Manager 2012 Daily Alert Management

Here's a quick little Operations Manager powershell command that I use pretty much on a daily basis.



Get-SCOMAlert | Where-Object { ($_.ismonitoralert -like "False") -and ($_.lastmodified -le (get-date).addhours(-24)) -and $_.resolutionstate -ne 255 } | Set-SCOMAlert -ResolutionState 255

So, this basically closes all the open alerts created by alert rules that are older than 24 hours.  Alerts created by alert monitors will close themselves once the condition that triggered them is resolved.

Installing Operations Manager 2012 R2 agent fails with error code 80070005

Today, I ran into an issue when trying to install the SCOM 2012 R2 agent on a domain controller running Windows Server 2012 R2.  I found the following technet article which proved useful as it had the error and error code I was getting, but the remediation steps didn't solve my problem.

Troubleshooting Issues When You Use the Discovery Wizard to Install an Agent

Upon further investigation, I could not access the SYSVOL and NETLOGON shares on that specific domain controller from the SCOM management servers running Windows Server 2012 R2.

I remember seeing this sort of issue when accessing shares on a NetApp that we have from a Windows 2012 Server.  So disabling the secure negotiate on the Windows 2012 Server solved that issue.  So I did the same on the SCOM Management Servers which ended up fixing this issue using the following powershell command.

Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters" RequireSecureNegotiate -Value 0 -Force

 http://support.microsoft.com/kb/2686098

You should also read this which explains Secure Negotiate which is new for SMB3.
http://blogs.msdn.com/b/openspecification/archive/2012/06/28/smb3-secure-dialect-negotiation.aspx