T-SQL

Monitoring BizTalk Server Without SCCM, using SQL query

Kevin Shyr — Thu, 15 Sep 2011 03:37:04 GMT

For the environment that does not have SCCM to monitor BizTalk server health, the following SQL query can be incorporate into a SQL job and have BizTalk server at least on a "constant" check.

Note: be aware that the following was tested only against a single suspended instance with a send port. Since BizTalk SQL server does not allow select against the ServiceClass table, I couldn't use a single table join. With the data I currently have, I am not 100% sure the Receive Port, Pipeline, Orchestration parts work.

SELECT TOP 1000

_M.nvcName AS ApplicationName

, CASE

WHEN _BSP.nvcName IS NOT NULL

THEN 'Send Port: ' + _BSP.nvcName

WHEN _BSP.nvcName IS NOT NULL

THEN 'Receive Port: ' + _BSP.nvcName

WHEN _BPL.Name IS NOT NULL

THEN 'Pipeline: ' + _BPL.Name

WHEN _BO.nvcName IS NOT NULL

THEN 'Orchestration: ' + _BO.nvcName

ELSE 'Unknown'

END AS ServiceName

, _IS.dtCreated

, _IS.dtSuspendTimeStamp

, _IS.nvcAdapter

, _IS.nvcURI

, _IS.nvcErrorDescription

, _IS.nvcErrorProcessingServer

FROM [BizTalkMsgBoxDb].[dbo].[InstancesSuspended] AS _IS WITH(READPAST)

INNER JOIN [BizTalkMsgBoxDb].[dbo].[Services] AS _S WITH(READPAST)

ON _IS.[uidServiceID] = _S.uidServiceID

INNER JOIN [BizTalkMsgBoxDb].[dbo].[Modules] AS _M WITH(READPAST)

ON _S.nModuleID = _M.nModuleID

LEFT OUTER JOIN [BizTalkMgmtDb].[dbo].[bts_sendport] AS _BSP WITH(READPAST)

ON _IS.uidServiceID = _BSP.uidGUID

LEFT OUTER JOIN [BizTalkMgmtDb].[dbo].[bts_receiveport] AS _BRP WITH(READPAST)

ON _IS.uidServiceID = _BRP.uidGUID

LEFT OUTER JOIN [BizTalkMgmtDb].[dbo].[bts_pipeline] AS _BPL WITH(READPAST)

ON _IS.uidServiceID = _BPL.PipelineID

LEFT OUTER JOIN [BizTalkMgmtDb].[dbo].[bts_orchestration] AS _BO WITH(READPAST)

ON _IS.uidServiceID = _BO.uidGUID

WHERE

_M.nvcName = 'Your application name' -- change application name here to monitor different applications

Do NOT trust CHECKSUM or BINARY_CHECKSUM in SQL, use HASHBYTES instead

Kevin Shyr — Sat, 21 May 2011 01:50:01 GMT

We had a process ot using BizTalk to take in a reference file, then send data in via a table_type parameter in a stored procedure. The stored procedure uses a MERGE statement to insert and update; the update part of which does a CHECKSUM first to determine whether an update is even necessary.

Then it happened, we had a reference row that should have been updated by the latest file, but it wasn't done. We checked the incoming and outgoing BizTalk Message in and out of the pipeline; everything looked normal. Then we decided to take out the CHECKSUM part and the update succeeded.

In reading over the Microsoft documentation, I was absolutely shocked when I read this in the remarks section:

However, there is a small chance that the checksum will not change. For this reason, we do not recommend using CHECKSUM to detect whether values have changed, unless your application can tolerate occasionally missing a change

http://msdn.microsoft.com/en-us/library/ms189788%28v=SQL.100%29.aspx

Then I checked the page for BINARY_CHECKSUM, a similar remark exists. http://msdn.microsoft.com/en-us/library/ms173784%28v=SQL.100%29.aspx

According to these 2 articles, we should be using HASHBYTES(). http://msdn.microsoft.com/en-us/library/ms174415%28v=SQL.100%29.aspx

Personally, I was really shocked to find a function in any database that "most of the time" does its job. I really do not know any system that tolerates "occasional" mistakes, or maybe it's that I haven't worked in enough industries. I equate this to getting a notice on your bank statement saying "we might not have all your transaction listed, and we don't know for sure whether we have them all".

I love quantum physics, but not in my computer systems.

Frustration and rewards on using SSIS Pivot and Unpivot

Kevin Shyr — Fri, 20 May 2011 20:36:30 GMT

Granted, if you are just transforming data in the same SQL server, why would you do it in SSIS instead of just using Pivot and Unpivot in T-SQL query? But what about in the case where data comes from various sources?

What I had fun doing was the ability to skip using Linked server, pull in data from different sources, like XML over web service. Another point that made me happy was the ability to simulate ROW_NUMBER() function with the merged data by using a Script Component.

Anyway, I highly recommend everyone to try this out. It's very frustrating throughout the process, but the result is very rewarding.

Pivot: http://msdn.microsoft.com/en-us/library/ms140308.aspx

Unpivot: http://msdn.microsoft.com/en-us/library/ms141723.aspx

Instead of alter table column to turn IDENTITY on and off, turn IDENTITY_INSERT on and off

Kevin Shyr — Thu, 03 Mar 2011 05:24:34 GMT

First of all, I don't know which version of SQL this post (http://www.techonthenet.com/sql/tables/alter_table.php) is based on, but at least for Microsoft SQL Server 2008, the syntax is not:

ALTER TABLE [table_name]
MODIFY [column_name] [data_type] NOT NULL;

Instead, it should be:

ALTER TABLE [table_name]
ALTER COLUMN [column_name] [data_type] NOT NULL;

Then, as several posts point out, you can't use T-SQL to run an existing column into an IDENTITY column. Instead, use the IDENTITY_INSERT to copy data from other tables. http://msdn.microsoft.com/en-us/library/ms188059.aspx

SET IDENTITY_INSERT [table_name] ON
INSERT ....
SET IDENTITY_INSERT [table_name] OFF

http://www.sqlservercentral.com/Forums/Topic126147-8-1.aspx

http://www.sqlteam.com/forums/topic.asp?TOPIC_ID=65257

Change Data Capture to get a paricular set of change within a time span

Kevin Shyr — Wed, 10 Nov 2010 23:38:49 GMT

The "fn_cdc_map_time_to_lsn" is especially useful

DECLARE @begin_time datetime, @end_time datetime, @from_lsn binary(10), @to_lsn binary(10)

SET @begin_time = '11/10/2010 10:00 AM'

SET @end_time = '11/10/2010 8:00 PM'

SET @from_lsn = sys.fn_cdc_map_time_to_lsn('smallest greater than or equal', @begin_time);

SET @to_lsn = sys.fn_cdc_map_time_to_lsn('largest less than or equal', @end_time);

SELECT

_OU.unit_natural_id AS Order_Id

, CONVERT(DATETIME, LEFT(CONVERT(VARCHAR(80), _OR.return_body.query('

declare default element namespace ''urn:com:maritz:research:caf:returns:BIE:1:0'';

/Return/ReturnHeader/ReturnDate/text()

')), 10), 126) AS ReturnReceivedDate

FROM ODS.Unit AS _OU

LEFT OUTER JOIN ODS.[Return] AS _OR

ON _OU.unit_key = _OR.unit_key

WHERE

_OU.unit_guid IN (SELECT population_guid FROM cdc.fn_cdc_get_all_changes_Clean_Collection(@from_lsn, @to_lsn, 'all'))

ORDER BY

CONVERT(DATETIME, LEFT(CONVERT(VARCHAR(80), _OR.return_body.query('

declare default element namespace ''urn:com:maritz:research:caf:returns:BIE:1:0'';

/Return/ReturnHeader/ReturnDate/text()

')), 10), 126)

Execute SQL Job synchronously for BizTalk via a Stored Procedure call

Kevin Shyr — Mon, 01 Nov 2010 20:18:30 GMT

The async one was very easy to do. However, in the process automation task I was given, there was a need to execute SQL Job and waiting for the execution result. I couldn't find anything on the web to do what I wanted, so I make the following stored procedure.

In addition, this stored procedure is quasi-synchronous. I start the job and then check the status. You can change the timing of the status check, but for the example below, it is "synchronous within 30 seconds." I wish there is a better way to do this, but at my current knowledge base I only came up with this:

CREATE PROCEDURE [SQLJobExecSyncByName]
    @SQLJobName AS [varchar](128),
    @SQLJobStepName AS [varchar](128),
    @MinutesToWait AS [int],
    @JobExecStatus AS [bit] output
AS
BEGIN
    -- SET NOCOUNT ON added to prevent extra result sets from
    -- interfering with SELECT statements.
    SET NOCOUNT ON;

    DECLARE @JobStartDateTime AS DATETIME
    DECLARE @SPQuitDateTime AS DATETIME
    DECLARE @JobStartDateAsString AS VARCHAR(8)
    DECLARE @JobStartTimeAsString AS VARCHAR(6)
    DECLARE @SPOutcome AS BIT
    SET @JobStartDateTime = DATEADD(mi, -1, GETDATE())
    SET @SPQuitDateTime = DATEADD(mi, @MinutesToWait, GETDATE())
    SET @SPOutcome = 0
    SELECT @JobStartDateAsString = CONVERT(VARCHAR(8), @JobStartDateTime, 112)
    SELECT @JobStartTimeAsString = REPLACE(CONVERT(VARCHAR(8), @JobStartDateTime, 108), ':', '')

    -- Create a temp table to store result
    CREATE TABLE #Result_sp_help_jobhistory
    (
        instance_id [int]
        , job_id [uniqueidentifier]
        , job_name [sysname]
        , step_id [int]
        , step_name [sysname]
        , sql_message_id [int]
        , sql_severity [int]
        , [message] [nvarchar](1024)
        , run_status [int]
        , run_date [int]
        , run_time [int]
        , run_duration [int]
        , operator_emailed [nvarchar](20)
        , operator_netsent [nvarchar](20)
        , operator_paged [nvarchar](20)
        , retries_attempted [int]
        , [server] [nvarchar](30)
    )

    -- Start the job @SQLJobName at step @SQLJobStepName
    EXEC msdb.dbo.sp_start_job
        @job_name = @SQLJobName,
        @server_name = @@SERVERNAME,
        @step_name = @SQLJobStepName;

    WHILE (GETDATE() < @SPQuitDateTime) -- not time to quit yet
    BEGIN
        -- Check to see if the job has finished successfully
        INSERT INTO #Result_sp_help_jobhistory
            (instance_id
                , job_id
                , job_name
                , step_id
                , step_name
                , sql_message_id
                , sql_severity
                , [message]
                , run_status
                , run_date
                , run_time
                , run_duration
                , operator_emailed
                , operator_netsent
                , operator_paged
                , retries_attempted
                , [server]
            )
        EXEC msdb.dbo.sp_help_jobhistory
            @job_name = @SQLJobName,
            @start_run_date = @JobStartDateAsString,
            @start_run_time = @JobStartTimeAsString,
            --@run_status = 1, -- Success
            @mode = 'FULL'

        IF(EXISTS(SELECT *
                    FROM #Result_sp_help_jobhistory
                    WHERE job_name = @SQLJobName
                        AND step_name = '(Job outcome)'
                        AND run_status = 1
                        AND @JobStartDateTime < CONVERT(DATETIME, CONVERT(VARCHAR(8), run_date) + ' ' + SUBSTRING(CONVERT(VARCHAR(6), run_time), 1, 2) + ':' + SUBSTRING(CONVERT(VARCHAR(6), run_time), 3, 2) + ':' + SUBSTRING(CONVERT(VARCHAR(6), run_time), 5, 2), 112)
                )
            )
        BEGIN
            -- The job has succceeded at least once after it was called
            SET @SPOutcome = 1
            GOTO FinishingStoredProcedureCall;
        END

        IF(EXISTS(SELECT *
                    FROM #Result_sp_help_jobhistory
                    WHERE job_name = @SQLJobName
                        AND step_name = '(Job outcome)'
                        AND run_status = 0
                        AND @JobStartDateTime < CONVERT(DATETIME, CONVERT(VARCHAR(8), run_date) + ' ' + SUBSTRING(CONVERT(VARCHAR(6), run_time), 1, 2) + ':' + SUBSTRING(CONVERT(VARCHAR(6), run_time), 3, 2) + ':' + SUBSTRING(CONVERT(VARCHAR(6), run_time), 5, 2), 112)
                )
            )
        BEGIN
            -- The job has failed after it was called
            GOTO FinishingStoredProcedureCall;
        END

        -- Clean up for the next run
        TRUNCATE TABLE #Result_sp_help_jobhistory

        -- check every 30 seconds
        WAITFOR DELAY '00:00:30';
    END

    FinishingStoredProcedureCall:
        SET @JobExecStatus = @SPOutcome
        DROP TABLE #Result_sp_help_jobhistory;

END

This SP was made to return an output parameter as result as to maximize reusability of the logic. In order to use this from BizTalk, it was hooked into the following SP:

CREATE PROCEDURE [SQLJobExecSyncByNameForBizTalk]
    @SQLJobName AS [varchar](128),
    @SQLJobStepName AS [varchar](128),
    @MinutesToWait AS [int]
AS
BEGIN
    -- SET NOCOUNT ON added to prevent extra result sets from
    -- interfering with SELECT statements.
    SET NOCOUNT ON;

    DECLARE @JobExecStatus AS BIT

    EXEC [ODS].[SQLJobExecSyncByName]
        @SQLJobName = @SQLJobName,
        @SQLJobStepName = @SQLJobStepName,
        @MinutesToWait = @MinutesToWait,
        @JobExecStatus = @JobExecStatus OUTPUT;

    SELECT CONVERT(VARCHAR(8), @JobExecStatus)
        FOR XML PATH(N''), ROOT(N'JobExecStatus');

END

BizTalk UseAmbientTransaction and System.ObjectDisposedException

Kevin Shyr — Wed, 25 Aug 2010 15:46:05 GMT

[UPDATE: 2010-10-15]

This update is long overdue. Once we implemented all the changes, we were still getting that error from time to time. The sympton was that we would not get that error for hours, even days, but once we get it, the error keeps coming. After fiddling with some settings, we found that setting the "PollWhileDataFound" to False did the trick. We have not seen the error since.

[UPDATE: 2010-08-26]

We were able to fix our problem by setting the WCF-SQL receive location transaction isolation level. Go to your BizTalk WCF-SQL receive location --> Click Configure --> Go to Behavior tab --> Set sqlAdapterInboundTransactionBehavior to "ReadCommitted". We have now set the "UseAmbientTransaction" property to True and have not had any problems for 2 days.

[Original Post]

We are continuing to experience the "System.ObjectDisposedException" error with UseAmbientTransaction set to True in our receive ports on our production server, and have not found a solution. For now we will try to separate the polling and the update statement into different steps.

Background: http://social.msdn.microsoft.com/Forums/en-US/biztalkr2adapters/thread/8c043def-6c43-4892-93a4-6201a10a89bc

Comparing ways to get rowcount from a database table.

Kevin Shyr — Thu, 25 Feb 2010 16:07:06 GMT

Just read this and found it very interesting. The following are 2 queries to get rowcount on a database table:

Query 1 (the one everyone knows):

SELECT COUNT(RowID) AS [RowCount]

FROM elig_Inbox

Query 2 (the interesting one):

SELECT st.row_count AS [RowCount]

FROM sys.dm_db_partition_stats st

WHERE index_id < 2

and OBJECT_NAME(OBJECT_ID)='elig_Inbox'

What's really interesting to me is that once you look at the execution plan of the 2 queries, you'd find the improvement in performance.

In real life, we probably don't care about it as often as we should. For example, both queries, executed against a table of 260k + rows, returned result within a second.

However, one drawback I can think of immediately, is that Query 2 does not have the Table Hints option. Most of the time we do RowCount because we need some logic done and we need to lock the table while we do the count. I guess the way around that would be to manually initiate a lock before Query 2, then release it after. It will be interesting to see whether that puts the 2 query performance back on par.

Stop a SQL job in the first 5 days of the month, but report success

Kevin Shyr — Tue, 02 Feb 2010 19:12:58 GMT

It is actually quite simple when I think this through:

Make the first job step with this T-SQL query:
IF ((SELECT DAY(GETDATE())) < 6) BEGIN
SELECT 1/0
END
Go to the Advanced Tab of the Job Step property page. Select "Quit the job reporting sucess" on the "On Failure Action" drop down.

Note: You can use RAISERROR, but the Severity has to be set to greater than 10. To avoid future maintenance confusion, I decided to go with the basic fatal error generation.

Useful T-SQL queries on BizTalk Filter, Send Port, Receive Location, etc.

Kevin Shyr — Wed, 20 Jan 2010 17:33:10 GMT

[2011-02-17]

Disable all receive locations of Schedule adapter if the system has any suspended messages

UPDATE RL

SET RL.[Disabled] = -1

FROM BizTalkMgmtDb.dbo.adm_ReceiveLocation AS RL WITH(READPAST, ROWLOCK)

INNER JOIN BizTalkMgmtDb.dbo.bts_receiveport AS RP WITH(READPAST, ROWLOCK)

ON RL.ReceivePortId = RP.nID

INNER JOIN BizTalkMgmtDb.dbo.bts_application AS APP WITH(READPAST, ROWLOCK)

ON RP.nApplicationID = APP.nID

INNER JOIN BizTalkMgmtDb.dbo.adm_Adapter AS AD WITH(READPAST, ROWLOCK)

ON RL.AdapterId = AD.Id

WHERE

APP.nvcName = '?'

AND AD.[Name] = 'Schedule'

AND RL.Name NOT LIKE '%Disabled%'

AND RL.[Disabled] = 0

AND EXISTS(SELECT * FROM BizTalkMsgBoxDb.dbo.InstancesSuspended WITH(READPAST, ROWLOCK))

Looking for down Receive Locations and send ports

SELECT

SPTP.nvcAddress AS SendingURI

, SP.nvcName AS SendPortName

FROM

BizTalkMgmtDb.dbo.bts_sendport_transport AS SPTP WITH(READPAST, ROWLOCK)

INNER JOIN BizTalkMgmtDb.dbo.bts_sendport AS SP WITH(READPAST, ROWLOCK)

ON SPTP.nSendPortID = SP.nID

INNER JOIN BizTalkMgmtDb.dbo.bts_application AS APP WITH(READPAST, ROWLOCK)

ON SP.nApplicationID = APP.nID

WHERE

APP.nvcName = '?'

AND SP.nPortStatus <> 3

AND SP.nvcName NOT LIKE '%unenlisted%'

AND SPTP.nTransportTypeId IS NOT NULL

SELECT

RL.Name AS ReceiveLocationName

, RL.InboundTransportURL AS InboundTransportURL

, RP.nvcName AS ReceivePortName

FROM BizTalkMgmtDb.dbo.adm_ReceiveLocation AS RL WITH(READPAST, ROWLOCK)

INNER JOIN BizTalkMgmtDb.dbo.bts_receiveport AS RP WITH(READPAST, ROWLOCK)

ON RL.ReceivePortId = RP.nID

INNER JOIN BizTalkMgmtDb.dbo.bts_application AS APP WITH(READPAST, ROWLOCK)

ON RP.nApplicationID = APP.nID

WHERE

APP.nvcName = '?'

AND RL.[Disabled] = -1

AND RL.Name NOT LIKE '%Disabled%'

[2010-01-21]
Looking for all send ports and their applications that subscribe to a receive port's messages

WITH
TmpXMLNode
( SendPortName
    , ApplicationName
    , tmpcol
)
AS
(SELECT
    SP.nvcName AS SendPortName
    , APP.nvcName AS ApplicationName
    , CAST(REPLACE(REPLACE(REPLACE(CONVERT(NVARCHAR(MAX), SP.nvcFilter),'>','>'),'<','<'),
      'xmlns="http://www.w3.org/2001/XMLSchema-instance"','') AS XML) AS tmpcol
FROM
    bts_sendport AS SP
    INNER JOIN bts_application AS APP
      ON SP.nApplicationID = APP.nID
WHERE
    CONVERT(VARCHAR(MAX), nvcFilter) <> ''
)

SELECT
SendPortName
, ApplicationName
, CONVERT(VARCHAR(255), nref.query('data(@Value)')) AS FilterValue
, CONVERT(VARCHAR(255), nref.query('data(@Property)')) AS FilterProperty
FROM
TmpXMLNode
CROSS APPLY
    TmpXMLNode.tmpcol.nodes('/Filter/Group/Statement') AS R(nref)
WHERE
CONVERT(VARCHAR(255), nref.query('data(@Property)')) = 'BTS.ReceivePortName' -- filter type
AND CONVERT(VARCHAR(255), nref.query('data(@Value)')) = '[Receive port name]'

[2010-01-20]
Looking for all the send ports that send to a specific location:

SELECT

SPTP.nvcAddress AS SendingURI

, SP.nvcName AS SendPortName

, SP.nPortStatus AS SendPortStatus

, APP.nvcName AS ApplicationName

, APP.DateModified AS ApplicationDateModified

FROM

bts_sendport_transport AS SPTP

INNER JOIN bts_sendport AS SP

ON SPTP.nSendPortID = SP.nID

INNER JOIN bts_application AS APP

ON SP.nApplicationID = APP.nID

WHERE

SPTP.nvcAddress like 'ftp://ftp.[some company].com%'