I recently wrote about My Experience Moving the Operations Database to New Hardware.
Something I noticed today – is that the application event log on the SQL server was full of 18054 events, such as below:
Log Name: Application
Source: MSSQL$I01
Date: 10/23/2010 5:40:14 PM
Event ID: 18054
Task Category: Server
Level: Error
Keywords: Classic
User: OPSMGR\msaa
Computer: SQLDB1.opsmgr.net
Description:
Error 777980007, severity 16, state 1 was raised, but no message with that error number was found in sys.messages. If error is larger than 50000, make sure the user-defined message is added using sp_addmessage.
You might also notice some truncated events in the OpsMgr event log, on your RMS or management servers:
Event Type: Warning
Event Source: DataAccessLayer
Event Category: None
Event ID: 33333
Date: 10/23/2010
Time: 5:40:13 PM
User: N/A
Computer: OMMS3
Description:
Data Access Layer rejected retry on SqlError:
Request: p_DiscoverySourceUpsert -- (DiscoverySourceId=f0c57af0-927a-335f-1f74-3a3f1f5ca7cd), (DiscoverySourceType=0), (DiscoverySourceObjectId=74fb2fa8-94e5-264d-5f7e-57839f40de0f), (IsSnapshot=True), (TimeGenerated=10/23/2010 10:37:36 PM), (BoundManagedEntityId=3304d59d-5af5-ba80-5ba7-d13a07ed21d4), (IsDiscoveryPackageStale=), (RETURN_VALUE=1)
Class: 16
Number: 18054
Message: Error 777980007, severity 16, state 1 was raised, but no message with that error number was found in sys.messages. If error is larger than 50000, make sure the user-defined message is added using sp_addmessage.
Event Type: Error
Event Source: Health Service Modules
Event Category: None
Event ID: 10801
Date: 10/23/2010
Time: 5:40:13 PM
User: N/A
Computer: OMMS3
Description:
Discovery data couldn't be inserted to the database. This could have happened because of one of the following reasons:- Discovery data is stale. The discovery data is generated by an MP recently deleted.
- Database connectivity problems or database running out of space.
- Discovery data received is not valid.The following details should help to further diagnose:
DiscoveryId: 74fb2fa8-94e5-264d-5f7e-57839f40de0f
HealthServiceId: bf43c6a9-8f4b-5d6d-5689-4e29d56fed88
Error 777980007, severity 16, state 1 was raised, but no message with that error number was found in sys.messages. If error is larger than 50000, make sure the user-defined message is added using sp_addmessage..
After a little research – apparently this is caused when following the guide to move the Operations Database to new hardware.
Marnix blogged about this issue http://thoughtsonopsmgr.blogspot.com/2009/02/moving-scom-database-to-another-server.html which references Matt Goedtel’s article http://blogs.technet.com/b/mgoedtel/archive/2007/08/06/update-to-moving-operationsmanager-database-steps.aspx
Because in this process – we simply restore the Operations Database ONLY, we do not carry over some of the modifications to the MASTER database that are performed when you run the Database Installation during setup to create the original operations database.
For some OpsMgr events, which stem from database activity, we get the event data from SQL. If these messages do not exist in SQL – you see the above issue.
What is bad about this – is that it will keep some event rules from actually alerting us to the condition! For instance – the rule “Discovery Data Submission Failure” which will alert when there is a failure to insert discovery data – will not trigger now, because it is looking for specific information in parameter 3 of the event, which is part of the missing data:
To resolve this – we need to add back the missing information into the MASTER database.
- IF you have moved your OperationsManager database to new hardware
AND:
- IF you are seeing event 18054 events in the application log of the OpsDB SQL instance server.
Then you are impacted. To resolve this – you should run the attached SQL script against the Master database of the SQL instance that hosts your OperationsManager Database. You should ONLY consider running this if you are 100% sure that you are impacted by this issue.
See attached: Fix_OpsMgrDB_ErrorMsgs.sql