BizTalk

Monitoring BizTalk Server Without SCCM, using SQL query

Kevin Shyr — Thu, 15 Sep 2011 03:37:04 GMT

For the environment that does not have SCCM to monitor BizTalk server health, the following SQL query can be incorporate into a SQL job and have BizTalk server at least on a "constant" check.

Note: be aware that the following was tested only against a single suspended instance with a send port. Since BizTalk SQL server does not allow select against the ServiceClass table, I couldn't use a single table join. With the data I currently have, I am not 100% sure the Receive Port, Pipeline, Orchestration parts work.

SELECT TOP 1000

_M.nvcName AS ApplicationName

, CASE

WHEN _BSP.nvcName IS NOT NULL

THEN 'Send Port: ' + _BSP.nvcName

WHEN _BSP.nvcName IS NOT NULL

THEN 'Receive Port: ' + _BSP.nvcName

WHEN _BPL.Name IS NOT NULL

THEN 'Pipeline: ' + _BPL.Name

WHEN _BO.nvcName IS NOT NULL

THEN 'Orchestration: ' + _BO.nvcName

ELSE 'Unknown'

END AS ServiceName

, _IS.dtCreated

, _IS.dtSuspendTimeStamp

, _IS.nvcAdapter

, _IS.nvcURI

, _IS.nvcErrorDescription

, _IS.nvcErrorProcessingServer

FROM [BizTalkMsgBoxDb].[dbo].[InstancesSuspended] AS _IS WITH(READPAST)

INNER JOIN [BizTalkMsgBoxDb].[dbo].[Services] AS _S WITH(READPAST)

ON _IS.[uidServiceID] = _S.uidServiceID

INNER JOIN [BizTalkMsgBoxDb].[dbo].[Modules] AS _M WITH(READPAST)

ON _S.nModuleID = _M.nModuleID

LEFT OUTER JOIN [BizTalkMgmtDb].[dbo].[bts_sendport] AS _BSP WITH(READPAST)

ON _IS.uidServiceID = _BSP.uidGUID

LEFT OUTER JOIN [BizTalkMgmtDb].[dbo].[bts_receiveport] AS _BRP WITH(READPAST)

ON _IS.uidServiceID = _BRP.uidGUID

LEFT OUTER JOIN [BizTalkMgmtDb].[dbo].[bts_pipeline] AS _BPL WITH(READPAST)

ON _IS.uidServiceID = _BPL.PipelineID

LEFT OUTER JOIN [BizTalkMgmtDb].[dbo].[bts_orchestration] AS _BO WITH(READPAST)

ON _IS.uidServiceID = _BO.uidGUID

WHERE

_M.nvcName = 'Your application name' -- change application name here to monitor different applications

BizTalk documentator

Kevin Shyr — Fri, 06 May 2011 16:53:59 GMT

This looks like a very useful tool: http://biztalkdocumenter.codeplex.com/

Notes:

For this tool to be effective, populate the "Description" property of BizTalk projects.
Documents ports, business rules policies, etc.

Automated BizTalk documentation

Kevin Shyr — Tue, 15 Feb 2011 20:09:08 GMT

Yay, this should help us going through old legacy app with no doc, at least some help.

http://biztalkdocumenter.codeplex.com/

Using BizTalk to bridge SQL Job and Human Intervention (Requesting Permission)

Kevin Shyr — Fri, 04 Feb 2011 21:29:09 GMT

I start off the process with either a BizTalk Scheduler (http://biztalkscheduledtask.codeplex.com/releases/view/50363) or a manual file drop of the XML message. The manual file drop is to allow the SQL

Job to call a "File Copy" SSIS step to copy the trigger file for the next process and allows SQL

Job to be linked back into BizTalk processing.

The Process Trigger XML looks like the following. It is basically the configuration hub of the business process

<ns0:MsgSchedulerTriggerSQLJobReceive xmlns:ns0="urn:com:something something">
<ns0:IsProcessAsync>YES</ns0:IsProcessAsync>
<ns0:IsPermissionRequired>YES</ns0:IsPermissionRequired>
<ns0:BusinessProcessName>Data Push</ns0:BusinessProcessName>
<ns0:EmailFrom>DoNotReply@company.com</ns0:EmailFrom>
<ns0:EmailRecipientToList>ProjectManagers@company.com</ns0:EmailRecipientToList>
<ns0:EmailRecipientCCList>Developers@company.com</ns0:EmailRecipientCCList>
<ns0:EmailMessageBodyForPermissionRequest>This message was sent to request permission to start the Data Push process. The SQL Job to be run is WeeklyProcessing_DataPush</ns0:EmailMessageBodyForPermissionRequest>
<ns0:SQLJobName>WeeklyProcessing_DataPush</ns0:SQLJobName>
<ns0:SQLJobStepName>Push_To_Production</ns0:SQLJobStepName>
<ns0:SQLJobMinToWait>1</ns0:SQLJobMinToWait>
<ns0:PermissionRequestTriggerPath>\\server\ETL-BizTalk\Automation\TriggerCreatedByBizTalk\</ns0:PermissionRequestTriggerPath>
<ns0:PermissionRequestApprovedPath>\\server\ETL-BizTalk\Automation\Approved\</ns0:PermissionRequestApprovedPath>
<ns0:PermissionRequestNotApprovedPath>\\server\ETL-BizTalk\Automation\NotApproved\</ns0:PermissionRequestNotApprovedPath>
</ns0:MsgSchedulerTriggerSQLJobReceive>

Every node of this schema was promoted to a distinguished field so that the values can be used for decision making in the orchestration. The first decision made is on the "IsPermissionRequired" field.

If permission is required (IsPermissionRequired=="YES"), BizTalk will use the configuration info in the XML trigger to format the email message. Here is the snippet of how the email message is constructed.

SQLJobEmailMessage.EmailBody
 = new Eai.OrchestrationHelpers.XlangCustomFormatters.RawString(
 MsgSchedulerTriggerSQLJobReceive.EmailMessageBodyForPermissionRequest +
 " " +
 "By moving the file, you are either giving permission to the process, or disapprove of the process." +
 " " +
 "This is the file to move: \"" + PermissionTriggerToBeGenereatedHere +
 "\" " +
 "(You may find it easier to open the destination folder first, then navigate to the sibling folder to get to this file)" +
 " " +
 "To approve, move(NOT copy) the file here: " + MsgSchedulerTriggerSQLJobReceive.PermissionRequestApprovedPath +
 " " +
 "To disapprove, move(NOT copy) the file here: " + MsgSchedulerTriggerSQLJobReceive.PermissionRequestNotApprovedPath +
 " " +
 "The file will be IMMEDIATELY picked up by the automated process. This is normal. You should receive a message soon that the file is processed." +
 " " +
 "Thank you!"
 );
SQLJobSendNotification(Microsoft.XLANGs.BaseTypes.Address) = "mailto:" + MsgSchedulerTriggerSQLJobReceive.EmailRecipientToList;
SQLJobEmailMessage.EmailBody(Microsoft.XLANGs.BaseTypes.ContentType) = "text/html";
SQLJobEmailMessage(SMTP.Subject) = "Requesting Permission to Start the " + MsgSchedulerTriggerSQLJobReceive.BusinessProcessName;
SQLJobEmailMessage(SMTP.From) = MsgSchedulerTriggerSQLJobReceive.EmailFrom;
SQLJobEmailMessage(SMTP.CC) = MsgSchedulerTriggerSQLJobReceive.EmailRecipientCCList;
SQLJobEmailMessage(SMTP.EmailBodyFileCharset) = "UTF-8";
SQLJobEmailMessage(SMTP.SMTPHost) = "localhost";
SQLJobEmailMessage(SMTP.MessagePartsAttachments) = 2;

After the Permission request email is sent, the next step is to generate the actual Permission Trigger file. A correlation set is used here on SQLJobName and a newly generated GUID field.

<?xml version="1.0" encoding="utf-8"?><ns0:SQLJobAuthorizationTrigger xmlns:ns0="somethingsomething"><SQLJobName>Data Push</SQLJobName><CorrelationGuid>9f7c6b46-0e62-46a7-b3a0-b5327ab03753</CorrelationGuid></ns0:SQLJobAuthorizationTrigger>

The end user (the human intervention piece) will either grant permission for this process, or deny it, by moving the Permission Trigger file to either the "Approved" folder or the "NotApproved" folder. A parallel Listen shape is waiting for either response.

The next set of steps decide how the SQL Job is to be called, or whether it is called at all. If permission denied, it simply sends out a notification. If permission is granted, then the flag (IsProcessAsync) in the original Process Trigger is used. The synchonous part is not really synchronous, but a loop timer to check the status within the calling stored procedure (for more information, check out my previous post: http://geekswithblogs.net/LifeLongTechie/archive/2010/11/01/execute-sql-job-synchronously-for-biztalk-via-a-stored-procedure.aspx) If it's async, then the sp starts the job and BizTalk sends out an email.

And of course, some error notification:

Footnote: The next version of this orchestration will have an additional parallel line near the Listen shape with a Delay built in and a Loop to send out a daily reminder if no response has been received from the end user. The synchronous part is used to gather results and execute a data clean up process so that the SQL Job can be re-tried. There are manu possibilities here.

Execute SQL Job synchronously for BizTalk via a Stored Procedure call

Kevin Shyr — Mon, 01 Nov 2010 20:18:30 GMT

The async one was very easy to do. However, in the process automation task I was given, there was a need to execute SQL Job and waiting for the execution result. I couldn't find anything on the web to do what I wanted, so I make the following stored procedure.

In addition, this stored procedure is quasi-synchronous. I start the job and then check the status. You can change the timing of the status check, but for the example below, it is "synchronous within 30 seconds." I wish there is a better way to do this, but at my current knowledge base I only came up with this:

CREATE PROCEDURE [SQLJobExecSyncByName]
 @SQLJobName AS [varchar](128),
 @SQLJobStepName AS [varchar](128),
 @MinutesToWait AS [int],
 @JobExecStatus AS [bit] output
AS
BEGIN
 -- SET NOCOUNT ON added to prevent extra result sets from
 -- interfering with SELECT statements.
 SET NOCOUNT ON;

 DECLARE @JobStartDateTime AS DATETIME
 DECLARE @SPQuitDateTime AS DATETIME
 DECLARE @JobStartDateAsString AS VARCHAR(8)
 DECLARE @JobStartTimeAsString AS VARCHAR(6)
 DECLARE @SPOutcome AS BIT
 SET @JobStartDateTime = DATEADD(mi, -1, GETDATE())
 SET @SPQuitDateTime = DATEADD(mi, @MinutesToWait, GETDATE())
 SET @SPOutcome = 0
 SELECT @JobStartDateAsString = CONVERT(VARCHAR(8), @JobStartDateTime, 112)
 SELECT @JobStartTimeAsString = REPLACE(CONVERT(VARCHAR(8), @JobStartDateTime, 108), ':', '')

 -- Create a temp table to store result
 CREATE TABLE #Result_sp_help_jobhistory
 (
 instance_id [int]
 , job_id [uniqueidentifier]
 , job_name [sysname]
 , step_id [int]
 , step_name [sysname]
 , sql_message_id [int]
 , sql_severity [int]
 , [message] [nvarchar](1024)
 , run_status [int]
 , run_date [int]
 , run_time [int]
 , run_duration [int]
 , operator_emailed [nvarchar](20)
 , operator_netsent [nvarchar](20)
 , operator_paged [nvarchar](20)
 , retries_attempted [int]
 , [server] [nvarchar](30)
 )

 -- Start the job @SQLJobName at step @SQLJobStepName
 EXEC msdb.dbo.sp_start_job
 @job_name = @SQLJobName,
 @server_name = @@SERVERNAME,
 @step_name = @SQLJobStepName;

 WHILE (GETDATE() < @SPQuitDateTime) -- not time to quit yet
 BEGIN
 -- Check to see if the job has finished successfully
 INSERT INTO #Result_sp_help_jobhistory
 (instance_id
 , job_id
 , job_name
 , step_id
 , step_name
 , sql_message_id
 , sql_severity
 , [message]
 , run_status
 , run_date
 , run_time
 , run_duration
 , operator_emailed
 , operator_netsent
 , operator_paged
 , retries_attempted
 , [server]
 )
 EXEC msdb.dbo.sp_help_jobhistory
 @job_name = @SQLJobName,
 @start_run_date = @JobStartDateAsString,
 @start_run_time = @JobStartTimeAsString,
 --@run_status = 1, -- Success
 @mode = 'FULL'

 IF(EXISTS(SELECT *
 FROM #Result_sp_help_jobhistory
 WHERE job_name = @SQLJobName
 AND step_name = '(Job outcome)'
 AND run_status = 1
 AND @JobStartDateTime < CONVERT(DATETIME, CONVERT(VARCHAR(8), run_date) + ' ' + SUBSTRING(CONVERT(VARCHAR(6), run_time), 1, 2) + ':' + SUBSTRING(CONVERT(VARCHAR(6), run_time), 3, 2) + ':' + SUBSTRING(CONVERT(VARCHAR(6), run_time), 5, 2), 112)
 )
 )
 BEGIN
 -- The job has succceeded at least once after it was called
 SET @SPOutcome = 1
 GOTO FinishingStoredProcedureCall;
 END

 IF(EXISTS(SELECT *
 FROM #Result_sp_help_jobhistory
 WHERE job_name = @SQLJobName
 AND step_name = '(Job outcome)'
 AND run_status = 0
 AND @JobStartDateTime < CONVERT(DATETIME, CONVERT(VARCHAR(8), run_date) + ' ' + SUBSTRING(CONVERT(VARCHAR(6), run_time), 1, 2) + ':' + SUBSTRING(CONVERT(VARCHAR(6), run_time), 3, 2) + ':' + SUBSTRING(CONVERT(VARCHAR(6), run_time), 5, 2), 112)
 )
 )
 BEGIN
 -- The job has failed after it was called
 GOTO FinishingStoredProcedureCall;
 END

 -- Clean up for the next run
 TRUNCATE TABLE #Result_sp_help_jobhistory

 -- check every 30 seconds
 WAITFOR DELAY '00:00:30';
 END

 FinishingStoredProcedureCall:
 SET @JobExecStatus = @SPOutcome
 DROP TABLE #Result_sp_help_jobhistory;

END

This SP was made to return an output parameter as result as to maximize reusability of the logic. In order to use this from BizTalk, it was hooked into the following SP:

CREATE PROCEDURE [SQLJobExecSyncByNameForBizTalk]
    @SQLJobName AS [varchar](128),
    @SQLJobStepName AS [varchar](128),
    @MinutesToWait AS [int]
AS
BEGIN
    -- SET NOCOUNT ON added to prevent extra result sets from
    -- interfering with SELECT statements.
    SET NOCOUNT ON;

    DECLARE @JobExecStatus AS BIT

    EXEC [ODS].[SQLJobExecSyncByName]
        @SQLJobName = @SQLJobName,
        @SQLJobStepName = @SQLJobStepName,
        @MinutesToWait = @MinutesToWait,
        @JobExecStatus = @JobExecStatus OUTPUT;

    SELECT CONVERT(VARCHAR(8), @JobExecStatus)
        FOR XML PATH(N''), ROOT(N'JobExecStatus');

END

BizTalk UseAmbientTransaction and System.ObjectDisposedException

Kevin Shyr — Wed, 25 Aug 2010 15:46:05 GMT

[UPDATE: 2010-10-15]

This update is long overdue. Once we implemented all the changes, we were still getting that error from time to time. The sympton was that we would not get that error for hours, even days, but once we get it, the error keeps coming. After fiddling with some settings, we found that setting the "PollWhileDataFound" to False did the trick. We have not seen the error since.

[UPDATE: 2010-08-26]

We were able to fix our problem by setting the WCF-SQL receive location transaction isolation level. Go to your BizTalk WCF-SQL receive location --> Click Configure --> Go to Behavior tab --> Set sqlAdapterInboundTransactionBehavior to "ReadCommitted". We have now set the "UseAmbientTransaction" property to True and have not had any problems for 2 days.

[Original Post]

We are continuing to experience the "System.ObjectDisposedException" error with UseAmbientTransaction set to True in our receive ports on our production server, and have not found a solution. For now we will try to separate the polling and the update statement into different steps.

Background: http://social.msdn.microsoft.com/Forums/en-US/biztalkr2adapters/thread/8c043def-6c43-4892-93a4-6201a10a89bc

BizTalk Business Rules Install steps for StaticSupport

Kevin Shyr — Wed, 25 Aug 2010 15:02:31 GMT

Damn, forgot to do this every time!

http://blogs.msdn.com/b/richardbpi/archive/2005/11/14/492489.aspx

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\BusinessRules\3.0\StaticSupport (DWORD)

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\BusinessRules\3.0\StaticSupport (DWORD)

0 - This is the default key, and pretty much mimics the behavior of BizTalk Server 2004 where an instance of an object is always required as an input fact, and the method is only called when the rule is evaluated or executed.
1 - An instance of the object is NOT required, and the static method is called whenever the rule is evaluated or executed
2 - An instance of the object is NOT required, but the static method will be called at rule translation time (only if the parameters are constants). This is primarily meant as a performance optimization. However, note that static members used as actions will NOT be executed at translation time, but static methods used as parameters may be.

Embarrassing Gaffe

LifeLongTechie — Wed, 20 Jan 2010 17:59:03 GMT

I was pretty sure the application didn't have an orchestration, and kind of freaked out when the send ports do not have any filter on ReceivePortName. It's definitely THE most embarrassing moment of my BizTalk life.

Check and double-check!

Useful T-SQL queries on BizTalk Filter, Send Port, Receive Location, etc.

Kevin Shyr — Wed, 20 Jan 2010 17:33:10 GMT

[2011-02-17]

Disable all receive locations of Schedule adapter if the system has any suspended messages

UPDATE RL

SET RL.[Disabled] = -1

FROM BizTalkMgmtDb.dbo.adm_ReceiveLocation AS RL WITH(READPAST, ROWLOCK)

INNER JOIN BizTalkMgmtDb.dbo.bts_receiveport AS RP WITH(READPAST, ROWLOCK)

ON RL.ReceivePortId = RP.nID

INNER JOIN BizTalkMgmtDb.dbo.bts_application AS APP WITH(READPAST, ROWLOCK)

ON RP.nApplicationID = APP.nID

INNER JOIN BizTalkMgmtDb.dbo.adm_Adapter AS AD WITH(READPAST, ROWLOCK)

ON RL.AdapterId = AD.Id

WHERE

APP.nvcName = '?'

AND AD.[Name] = 'Schedule'

AND RL.Name NOT LIKE '%Disabled%'

AND RL.[Disabled] = 0

AND EXISTS(SELECT * FROM BizTalkMsgBoxDb.dbo.InstancesSuspended WITH(READPAST, ROWLOCK))

Looking for down Receive Locations and send ports

SELECT

SPTP.nvcAddress AS SendingURI

, SP.nvcName AS SendPortName

FROM

BizTalkMgmtDb.dbo.bts_sendport_transport AS SPTP WITH(READPAST, ROWLOCK)

INNER JOIN BizTalkMgmtDb.dbo.bts_sendport AS SP WITH(READPAST, ROWLOCK)

ON SPTP.nSendPortID = SP.nID

INNER JOIN BizTalkMgmtDb.dbo.bts_application AS APP WITH(READPAST, ROWLOCK)

ON SP.nApplicationID = APP.nID

WHERE

APP.nvcName = '?'

AND SP.nPortStatus <> 3

AND SP.nvcName NOT LIKE '%unenlisted%'

AND SPTP.nTransportTypeId IS NOT NULL

SELECT

RL.Name AS ReceiveLocationName

, RL.InboundTransportURL AS InboundTransportURL

, RP.nvcName AS ReceivePortName

FROM BizTalkMgmtDb.dbo.adm_ReceiveLocation AS RL WITH(READPAST, ROWLOCK)

INNER JOIN BizTalkMgmtDb.dbo.bts_receiveport AS RP WITH(READPAST, ROWLOCK)

ON RL.ReceivePortId = RP.nID

INNER JOIN BizTalkMgmtDb.dbo.bts_application AS APP WITH(READPAST, ROWLOCK)

ON RP.nApplicationID = APP.nID

WHERE

APP.nvcName = '?'

AND RL.[Disabled] = -1

AND RL.Name NOT LIKE '%Disabled%'

[2010-01-21]
Looking for all send ports and their applications that subscribe to a receive port's messages

WITH
TmpXMLNode
( SendPortName
 , ApplicationName
 , tmpcol
)
AS
(SELECT
 SP.nvcName AS SendPortName
 , APP.nvcName AS ApplicationName
 , CAST(REPLACE(REPLACE(REPLACE(CONVERT(NVARCHAR(MAX), SP.nvcFilter),'>','>'),'<','<'),
 'xmlns="http://www.w3.org/2001/XMLSchema-instance"','') AS XML) AS tmpcol
FROM
 bts_sendport AS SP
 INNER JOIN bts_application AS APP
 ON SP.nApplicationID = APP.nID
WHERE
 CONVERT(VARCHAR(MAX), nvcFilter) <> ''
)

SELECT
SendPortName
, ApplicationName
, CONVERT(VARCHAR(255), nref.query('data(@Value)')) AS FilterValue
, CONVERT(VARCHAR(255), nref.query('data(@Property)')) AS FilterProperty
FROM
TmpXMLNode
CROSS APPLY
 TmpXMLNode.tmpcol.nodes('/Filter/Group/Statement') AS R(nref)
WHERE
CONVERT(VARCHAR(255), nref.query('data(@Property)')) = 'BTS.ReceivePortName' -- filter type
AND CONVERT(VARCHAR(255), nref.query('data(@Value)')) = '[Receive port name]'

[2010-01-20]
Looking for all the send ports that send to a specific location:

SELECT

SPTP.nvcAddress AS SendingURI

, SP.nvcName AS SendPortName

, SP.nPortStatus AS SendPortStatus

, APP.nvcName AS ApplicationName

, APP.DateModified AS ApplicationDateModified

FROM

bts_sendport_transport AS SPTP

INNER JOIN bts_sendport AS SP

ON SPTP.nSendPortID = SP.nID

INNER JOIN bts_application AS APP

ON SP.nApplicationID = APP.nID

WHERE

SPTP.nvcAddress like 'ftp://ftp.[some company].com%'

BizTalk FILE adapter mask is a prefix

LifeLongTechie — Wed, 13 Jan 2010 18:09:43 GMT

It seems like the file mask we can enter into the File Mask textbox in a BizTalk FILE port configuration is a prefix. I entered a mask that is "fix_*.txt" and the port is picking up everything with that prefix. I had quite a few files with .txt[datetimestamp] in the folder and everything was picked up, created quite a havoc.