Infrastructure

BizTalk DB Log File Explosion Due to Corrupt Index

Geordie — Wed, 17 Jun 2009 23:24:19 GMT

I have just encounted an unusual database problem. My dev BizTalk server suddenly ran out of space. Initially I thought it was the transaction logs. No luck there they were well within normal limits. I did check the size ot the
'Microsoft SQL Server' directory. It was over 8GB. The volume was traced to the LOG directory. SQL was pumping out 4 MB of error log every 10 seconds. After fighting with the server to free up some disk space, I checked out the error files. The parts appear to be:

  ***Stack Dump being sent to C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\LOG\SQLDump7243.txt
  * *******************************************************************************
  *
  * BEGIN STACK DUMP:
  *   06/17/09 15:52:15 spid 54
  *
  * CPerIndexMetaQS::ErrorAbort - Index corruption
  *
  * Input Buffer 510 bytes -
  *             EXECUTE msdb.dbo.sp_sqlagent_log_jobhistory @job_id = 0xBE810
  * 129F3D1574AA4EB72EE897A6CCC, @step_id = 0, @sql_message_id = 0, @sql_sev
  * erity = 0, @run_status = 1, @run_date = 20090617, @run_time = 155201, @r
  * un_duration = 3, @operator_id_emailed = 0, @operator_id_netsent = 0, @op
  * erator_id_paged = 0, @retries_attempted = 0, @session_id = 65, @message
  * = N'The job succeeded. The Job was invoked by Schedule 4 (Schedule). T
  * he last step to run was step 1 (Purge).'

And

  2009-06-17 15:52:14.55 spid56      Error: 8646, Severity: 21, State: 1.
  2009-06-17 15:52:14.55 spid56      Unable to find index entry in index ID 1, of table
  117575457, in database 'msdb'. The indicated index is corrupt or there is a
  problem with the current update plan. Run DBCC CHECKDB or DBCC CHECKTABLE. If
  the problem persists, contact product support.
  2009-06-17 15:52:15.24 spid54 Using 'dbghelp.dll' version '4.0.5'
  2009-06-17 15:52:15.24 spid54      **Dump thread - spid = 54, PSS = 0x0DDD32D0, EC = 0x0DDD32D8
  2009-06-17 15:52:15.24 spid54      ***Stack Dump being sent to C:\Program Files\Microsoft SQL
  Server\MSSQL.1\MSSQL\LOG\SQLDump7243.txt
  2009-06-17 15:52:15.24 spid54      * *******************************************************************************
  2009-06-17 15:52:15.24 spid54      *
  2009-06-17 15:52:15.24 spid54      * BEGIN STACK DUMP:
  2009-06-17 15:52:15.24 spid54      *   06/17/09 15:52:15 spid 54
  2009-06-17 15:52:15.24 spid54      * 2009-06-17 15:52:15.24 spid54      * CPerIndexMetaQS::ErrorAbort - Index corruption

Searching for 'CPerIndexMetaQS::ErrorAbort - Index corruption' online led me to a post that suggested running the following SQL script from SQL Server Manager on the local machine.

  DECLARE tempcursor
  CURSOR
  READ_ONLY
  FOR
        select spid From master..sysprocesses
        where dbid = (select database_id from sys.databases where name = 'msdb')

  DECLARE @name int
  OPEN tempcursor

  FETCH NEXT FROM tempcursor INTO @name
  WHILE (@@fetch_status <> -1)
  BEGIN
        exec('kill ' + @name)

        FETCH NEXT FROM tempcursor INTO @name
  END

  Use msdb
  Go
  sp_dboption 'msdb', 'single_user', 'true'
  GO
  DBCC CHECKDB('msdb', REPAIR_REBUILD)
  go
  sp_dboption 'msdb', 'single_user', 'false'

  CLOSE tempcursor
  DEALLOCATE tempcursor

The script found and repaired the corrupt index, as shown below.

  DBCC results for 'sysdownloadlist'.
  There are 0 rows in 0 pages for object "sysdownloadlist".
  DBCC results for 'sysjobhistory'.
  Repair: Successfully deleted row in index "dbo.sysjobhistory, nc1" in database "msdb".
  Msg 8952, Level 16, State 1, Line 1
  Table error: table 'sysjobhistory' (ID 117575457). Index row in index 'nc1' (ID 2) does not match any data row. Possible extra or invalid keys for:
          The error has been repaired.
  Msg 8956, Level 16, State 1, Line 1
  Index row (1:1141:114) with values (job_id = '89F8A96E-AF2B-4BE9-B5E4-FEF3B8C94EC5' and instance_id = 466459) pointing to the data row identified by (instance_id = 466459).
  There are 6640 rows in 459 pages for object "sysjobhistory".
  CHECKDB found 0 allocation errors and 1 consistency errors in table 'sysjobhistory' (object ID 117575457).
  CHECKDB fixed 0 allocation errors and 1 consistency errors in table 'sysjobhistory' (object ID 117575457).

The log file generation has stopped and the server appears to be back to normal.

Turning Off Data Execution Prevention (DEP)

Geordie — Wed, 03 Jun 2009 21:29:24 GMT

If you can find the control panel for DEP, you will find there is not an option to turn it off. This is not a terrible thing unless you are trying to troubleshoot a problem. Then you want to eliminate it as an issue, to confirm that the application will work in the environment as a whole. Then you can re enable it and try to configure DEP correctly to allow your application to function:

Access to DEP configuration panel-

1. Click Start, click Run, type sysdm.cpl, and then click OK.

2. On the Advanced tab, under Performance, click Settings.

3. On the Data Execution Prevention tab, use one of the following procedures:

a. Click Turn on DEP for essential Windows programs and services only to select the OptIn policy.

b. Click Turn on DEP for all programs and services except those I select to select the OptOut policy, and then click Add to add the programs that you do not want to use the DEP feature.

4. Click OK two times.

There are actually 4 global configurations for DEP-

Configuration	Description
OptIn	This setting is the default configuration. On systems with processors that can implement hardware-enforced DEP, DEP is enabled by default for limited system binaries and programs that "opt-in." With this option, only Windows system binaries are covered by DEP by default.
OptOut	DEP is enabled by default for all processes. You can manually create a list of specific programs that do not have DEP applied by using the System dialog box in Control Panel. Information technology (IT) professionals can use the Application Compatibility Toolkit to "opt-out" one or more programs from DEP protection. System compatibility fixes, or shims, for DEP do take effect.
AlwaysOn	This setting provides full DEP coverage for the whole system. All processes always run with DEP applied. The exceptions list to exempt specific programs from DEP protection is not available. System compatibility fixes for DEP do not take effect. Programs that have been opted-out by using the Application Compatibility Toolkit run with DEP applied.
AlwaysOff	This setting does not provide any DEP coverage for any part of the system, regardless of hardware DEP support. The processor does not run in PAE mode unless the /PAE option is present in the Boot.ini file.

To turn off DEP globally, we must edit the Boot.ini file. Here are some instructions and an example of a Boot.ini file.-

1. Click Start, right-click My Computer, and then click Properties.

2. Click the Advanced tab, and then click Settings under the Startup and Recovery field.

3. In the System startup field, click Edit. The Boot.ini file opens in Notepad.

4. In Notepad, click Find on the Edit menu.

5. In the Find what box, type /noexecute, and then click Find Next.

6. In the Find dialog box, click Cancel.

7. Replace policy_level with AlwaysOff

Example of a Boot.ini file with /NoExecute set to AlwaysOff-

[boot loader]
timeout=30
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional" /fastdetect /NoExecute=AlwaysOff

Source: http://support.microsoft.com/kb/875352

How to Turn Off 'Shut Down Tracker' on Server 2003.

Geordie — Mon, 19 Jan 2009 22:34:32 GMT

How to remove the shutdown tracker.

A: Start > Run MMC

File > Add/Remove Snap-in...

Add...

Group Policy Object Editor

Add

Finish

Local Computer Policy > Computer Configuration > Administrative Templates > System > Dbl-Click "Display Shutdown Event Tracker" and select Disabled.

File

Exit

To help differentiate between production and test I leave the Shut Down Tracker running on production servers and turn it of in development.

For Server 2008 try this post: http://www.win2008workstation.com/wordpress/2008/03/08/disabling-the-shutdown-event-tracker/

Geordie