Service Manager 2012 SP1 secondary Management Server–Cannot set availability on a health service that doesn’t exist

I recently ran into this issue whilst performing a Service Manager 2012 SP1 deployment for a customer.

The customer didn’t have Service Manager installed previously, therefore this was a fresh installation of SP1 – not an upgrade.  The infrastructure consisted of the following:

Qty Role Configuration
1 Service Manager Database SQL Server 2008 R2 SP2 CU5 running on Windows Server 2008 R2 SP1 – Service Manager Databases.  Collation: Latin1_General_100_CI_AS
1 Data Warehouse Databases SQL Server 2008 R2 SP2 CU5 running on Windows Server 2008 R2 SP1 – Service Manager Databases.  Collation: Latin1_General_100_CI_AS
1 Management Server (Workflow) Service Manager 2012 SP1running on Windows Server 2008 R2 SP1
2 Management Server (Console Connections Service Manager 2012 SP1running on Windows Server 2008 R2 SP1
1 Data Warehouse Management Server Service Manager 2012 SP1running on Windows Server 2008 R2 SP1
2 Web Content Server Service Manager 2012 SP1running on Windows Server 2008 R2 SP1

I deployed all the roles as described above, and all appeared well.  However, whilst looking through the event logs I spotted some strange errors:

Log Name:      Operations Manager
Source:        DataAccessLayer
Date:          07/03/2013 13:35:50
Event ID:      33333
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      SERVER.DOMAIN.COM
Description:
Data Access Layer rejected retry on SqlError:
Request: UpdateAvailability — (BaseManagedEntityId=c059728e-ee41-1688-18de-b7528afa1b8a), (IsAvailable=True), (ReasonCode=0), (TimeGenerated=07/03/2013 13:35:50), (RETURN_VALUE=1)
Class: 16
Number: 777980050
Message: Cannot set availability on a health service that doesn’t exist.

Log Name:      Operations Manager
Source:        OpsMgr Root Connector
Date:          07/03/2013 13:35:50
Event ID:      28000
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      SERVER.DOMAIN.COM
Description:
The Root connector received an exception from the SDK Service while submitting task status:

Cannot set availability on a health service that doesn’t exist.

At first I thought that the Health Service State might have become corrupt in some way.  So I stopped the System Center services on the Console Management Server and the Workflow Management server.  This didn’t resolve the issue.  I even checked the %PROGRAMFILES%\Microsoft System Center 2012\Service Manager\Health Service State\Management Packs folder to see if any configuration had been received, and…….nothing!  I therefore turned my attention to the Service Manager database.  The error suggests that the HealthService ID specified by the Console Management Server at service start-up was not in the database.  I therefore used the following SQL query to find the Management Server within the database:

image

As you can see, this returned no results.  I therefore ran a similar query, this time looking for a partial match on the AuthenticationName column:

image

Bingo!  I got a result.  But hang on a moment……its has a different BaseManagedEntityID to that reported in the event log.  So I thought I would check this against some of the other Management Servers.  I just did a “select *” on the dbo.MT_HealthService table to get all the BaseManagedEntityIDs and checked these with the event logs on the Workflow Management Server and other Console Management Server.  I found that the Workflow Management Server’s BaseManagedEntityID matched its entry in the database, but the second Console Management Server’s BaseManagedEntityID did not.  The second Console Management Server also had the errors mentioned previously in the event log.  One thing I did notice in the dbo.MT_HealthService table was the the AuthenticationName and DisplayName of the Workflow Management Server was its FQDN, whilst for the two Console Management Servers these were set the the NetBIOS name.

At this point I was unsure as to why the Management Servers were using what appeared to be an invalid BaseManagedEntityID.  My initial thought was that the database collation was having an effect.  So, I decided to lab this up on a few VMs on my laptop.  I created the following:

  • 1 x  SQL Server 2008 R2 SP2 CU5 running Windows Server 2008 R2 SP1.  2 SQL Server instances; one with a collation of Latin1_General_100_CI_AS, and one with SQL_Latin1_General_CP1_CI_AS
  • 2 x  Service Manager 2012 running Windows Server 2008 R2 SP1

My plan was to perform the following installations:

  • 2 x Service Manager 2012 SP1 Management Servers using SQL Server Instance with collation of Latin1_General_100_CI_AS
  • 2 x Service Manager 2012 SP1 Management Servers using SQL Server Instance with collation of SQL_Latin1_General_CP1_CI_AS

At this point, to cut a long story short, the behaviour of both of these installations was the same.  The initial management server installed successfully, whilst the second Management Server was unable to communicate as the database had a different BaseManagedEntityID that the one the Management Server was using.  The following screenshots illustrate this:

image

image

image

image

image

I then thought that this might be an SP1 only issue.  So, i performed similar tests as follows:

  • 2 x Service Manager 2012 RTM Management Servers using SQL Server Instance with collation of Latin1_General_100_CI_AS
  • 2 x Service Manager 2012 RTM Management Servers using SQL Server Instance with collation of SQL_Latin1_General_CP1_CI_AS

This time, my results were very different.  The second Management Server connected successfully to the Management Group without error in both cases.  The following screenshots illustrate this:

image

image

image

image

As you can see here, there is a key difference.  The AuthenticationName and DisplayName for both the first and second Management Servers in the Management Group are the FQDN of the machine.

I had a Google/Bing to see how the BaseManagedEntityID is generated and came across the following article by Mike ‘SentryBoy’ – Sorry, couldn’t find a last name for him:

http://sentryboy.wordpress.com/2009/10/07/basemanagedentityid/

Although not directly applicable to Service Manager, it sheds some light onto the possible cause of this particular issue. This article describes in some detail how the BaseManagedEntityID is generated for Operations Manager (Which Service Manager is based upon) and that it is deterministic.   One of the properties that affects the BaseManagedEntityID is PrincipleName.  Mikes original queries were based around generating BaseManagedEntityIDs for the Microsoft.Windows.Computer class.  However, here we are looking to generate BaseManagedEntityIDs for a Health Service, which is the Microsoft.SystemCenter.HealthService class.  Using Mike’s work as a starting point I came up with the following for Service Manager

1. Get the ManagedTypeID for the Microsoft.SystemCenter.HealthService Class:

Select ManagedTypeId, TypeName from ManagedType where TypeName = ‘Microsoft.SystemCenter.HealthService’

2. Get the ManagedTypePropertyId for PrincipleName:

Select ManagedTypePropertyId, ManagedTypePropertyName From ManagedTypeProperty where ManagedTypePropertyName = ‘PrincipalName’

We can then plug the output of the above into Mikes’’ final query to generate a BaseManagedEntityID for the HealthService which is failing to communicate with the management group:

declare @hashstring nvarchar(max)
select @hashstring = ‘TypeId={AB4C891F-3359-3FB6-0704-075FBFE36710},{5C324096-D928-76DB-E9E7-E629DCC261B1}=SM3.CONTOSO.COM’
select convert(UniqueIdentifier, HASHBYTES(‘SHA1’,@hashstring ))

Running the query twice, once with the failing value of SM3 and once with the successful value of SM3.CONTOSO.COM, this yields the following results:

SM3: 5C324096-D928-76DB-E9E7-E629DCC261B1 – This is the value present in the database when the Management Server cannot connect to the Management Group

SM3.CONTOSO.COM: 715A87F2-75C0-177F-8787-D54C10521624 – This is the value that the Management Server uses to attempt to communicate with the management group in all cases, and the value that is present in the database when the Management Server successfully connects with the Management Group

So from this, we can see that somehow, an incorrect value is being entered in the database for any secondary Management Server.  The only culprit I can see for this at this time is that installation routine is somehow doing this, and as a result, incorrect BaseManagedEntityIDs are being generated.  The only work around I can see at the moment is to perform a Service Manager 2012 RTM installation first, as this appears to insert the correct AuthenticationName and DisplayName into the database, and then upgrade to SP1.  I have yet to try this, but will report on this when I have.  But for now, one for Connect I think………….

***UPDATE***  A work around is now available for this particular issue.  This is available from Microsoft, but you must raise a PSS case in order to obtain the work around.  Essentially, the work around is a new SM.msi file which has had the custom actions fixed which were inserting the wrong AuthenticationName into the Service Manager database.  Please note that you MUST obtain this before installing a second Management Server in a greenfield Service Manager 2012 SP1 environment.

10 thoughts on “Service Manager 2012 SP1 secondary Management Server–Cannot set availability on a health service that doesn’t exist

  1. Any word on this?

    1. Hi Herb.

      My sources tell me that Microsoft have confirmed this to be a bug, and will have a fix for this in the near future.

      Cheers

      Shaun

      1. Hi Shaun,

        It appears that the secondary server is still functioning from a console perspective. Can I leave it as is until they publish a fix or is it causing something detrimental to my environment?

        If I must fix it do I need to rebuild both the primary and secondary server? This will be a major undertaking as my primary has been up and configured for some time.

        Thanks,
        Herb

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

search previous next tag category expand menu location phone mail time cart zoom edit close