SSIS

SSIS Tools

Kevin Shyr — Thu, 03 Nov 2011 20:03:37 GMT

Somebody told me this and I'm ready to try some of the components:

http://www.konesans.com/products.aspx

FileWatcher, Regex Clean, RowNumber Transform

SSIS design consideration

Kevin Shyr — Wed, 26 Oct 2011 03:16:37 GMT

The following is my experience with SSIS building:

Make sure you understand the 3 modes of Lookup. General, I would only do full cache if the lookup table only return less than 200 rows
Never use the table or view select unless you really need every field on that table or view.
- select * from a data source will break in the future if more columns are added to the table
- select * in a lookup brings back unnecessary fields and impact performance
Never use "Sort" in SSIS unless it's absolutely necessary. Sort in database first.
- Merge join requires sorted input; you can do order by in your select, then configure the output to specify the "sorted" column.
Transaction matters. If you only need to read data, use the snapshot option to ensure fastest reading and without blocking the table.
Don't use config file unless you are sharing values across multiple packages. Not using config files makes the packages more dynamic as values can be set though SQL Job or xp_cmdshell
Now that I've mentioned it, don't use xp_cmdshell to start package unless you have to. By properly storing variables in well constructed tables should eliminate the need to dynamically pass in values.
use variable value expression instead of using script task to set values
Just like in T-SQL, don't do a select with 15 tables joined together. Do an insert of the key first, then update the destination table.
Learn C# (well, for 2008 or later, otherwise, learn VB as well)

Enjoy your SSIS project!

Monitoring BizTalk Server Without SCCM, using SQL query

Kevin Shyr — Thu, 15 Sep 2011 03:37:04 GMT

For the environment that does not have SCCM to monitor BizTalk server health, the following SQL query can be incorporate into a SQL job and have BizTalk server at least on a "constant" check.

Note: be aware that the following was tested only against a single suspended instance with a send port. Since BizTalk SQL server does not allow select against the ServiceClass table, I couldn't use a single table join. With the data I currently have, I am not 100% sure the Receive Port, Pipeline, Orchestration parts work.

SELECT TOP 1000

_M.nvcName AS ApplicationName

, CASE

WHEN _BSP.nvcName IS NOT NULL

THEN 'Send Port: ' + _BSP.nvcName

WHEN _BSP.nvcName IS NOT NULL

THEN 'Receive Port: ' + _BSP.nvcName

WHEN _BPL.Name IS NOT NULL

THEN 'Pipeline: ' + _BPL.Name

WHEN _BO.nvcName IS NOT NULL

THEN 'Orchestration: ' + _BO.nvcName

ELSE 'Unknown'

END AS ServiceName

, _IS.dtCreated

, _IS.dtSuspendTimeStamp

, _IS.nvcAdapter

, _IS.nvcURI

, _IS.nvcErrorDescription

, _IS.nvcErrorProcessingServer

FROM [BizTalkMsgBoxDb].[dbo].[InstancesSuspended] AS _IS WITH(READPAST)

INNER JOIN [BizTalkMsgBoxDb].[dbo].[Services] AS _S WITH(READPAST)

ON _IS.[uidServiceID] = _S.uidServiceID

INNER JOIN [BizTalkMsgBoxDb].[dbo].[Modules] AS _M WITH(READPAST)

ON _S.nModuleID = _M.nModuleID

LEFT OUTER JOIN [BizTalkMgmtDb].[dbo].[bts_sendport] AS _BSP WITH(READPAST)

ON _IS.uidServiceID = _BSP.uidGUID

LEFT OUTER JOIN [BizTalkMgmtDb].[dbo].[bts_receiveport] AS _BRP WITH(READPAST)

ON _IS.uidServiceID = _BRP.uidGUID

LEFT OUTER JOIN [BizTalkMgmtDb].[dbo].[bts_pipeline] AS _BPL WITH(READPAST)

ON _IS.uidServiceID = _BPL.PipelineID

LEFT OUTER JOIN [BizTalkMgmtDb].[dbo].[bts_orchestration] AS _BO WITH(READPAST)

ON _IS.uidServiceID = _BO.uidGUID

WHERE

_M.nvcName = 'Your application name' -- change application name here to monitor different applications

Frustration and rewards on using SSIS Pivot and Unpivot

Kevin Shyr — Fri, 20 May 2011 20:36:30 GMT

Granted, if you are just transforming data in the same SQL server, why would you do it in SSIS instead of just using Pivot and Unpivot in T-SQL query? But what about in the case where data comes from various sources?

What I had fun doing was the ability to skip using Linked server, pull in data from different sources, like XML over web service. Another point that made me happy was the ability to simulate ROW_NUMBER() function with the merged data by using a Script Component.

Anyway, I highly recommend everyone to try this out. It's very frustrating throughout the process, but the result is very rewarding.

Pivot: http://msdn.microsoft.com/en-us/library/ms140308.aspx

Unpivot: http://msdn.microsoft.com/en-us/library/ms141723.aspx

Parallel programming in Visual Studio

Kevin Shyr — Thu, 31 Mar 2011 17:26:03 GMT

Not much experience with these products yet.

Free: http://developer.nvidia.com/object/nsight.html

Not free: http://software.intel.com/en-us/articles/intel-parallel-studio-purchase/

For SSIS, this looks very interesting:

http://extendedssispkgexec.codeplex.com/

Using BizTalk to bridge SQL Job and Human Intervention (Requesting Permission)

Kevin Shyr — Fri, 04 Feb 2011 21:29:09 GMT

I start off the process with either a BizTalk Scheduler (http://biztalkscheduledtask.codeplex.com/releases/view/50363) or a manual file drop of the XML message. The manual file drop is to allow the SQL

Job to call a "File Copy" SSIS step to copy the trigger file for the next process and allows SQL

Job to be linked back into BizTalk processing.

The Process Trigger XML looks like the following. It is basically the configuration hub of the business process

<ns0:MsgSchedulerTriggerSQLJobReceive xmlns:ns0="urn:com:something something">
<ns0:IsProcessAsync>YES</ns0:IsProcessAsync>
<ns0:IsPermissionRequired>YES</ns0:IsPermissionRequired>
<ns0:BusinessProcessName>Data Push</ns0:BusinessProcessName>
<ns0:EmailFrom>DoNotReply@company.com</ns0:EmailFrom>
<ns0:EmailRecipientToList>ProjectManagers@company.com</ns0:EmailRecipientToList>
<ns0:EmailRecipientCCList>Developers@company.com</ns0:EmailRecipientCCList>
<ns0:EmailMessageBodyForPermissionRequest>This message was sent to request permission to start the Data Push process. The SQL Job to be run is WeeklyProcessing_DataPush</ns0:EmailMessageBodyForPermissionRequest>
<ns0:SQLJobName>WeeklyProcessing_DataPush</ns0:SQLJobName>
<ns0:SQLJobStepName>Push_To_Production</ns0:SQLJobStepName>
<ns0:SQLJobMinToWait>1</ns0:SQLJobMinToWait>
<ns0:PermissionRequestTriggerPath>\\server\ETL-BizTalk\Automation\TriggerCreatedByBizTalk\</ns0:PermissionRequestTriggerPath>
<ns0:PermissionRequestApprovedPath>\\server\ETL-BizTalk\Automation\Approved\</ns0:PermissionRequestApprovedPath>
<ns0:PermissionRequestNotApprovedPath>\\server\ETL-BizTalk\Automation\NotApproved\</ns0:PermissionRequestNotApprovedPath>
</ns0:MsgSchedulerTriggerSQLJobReceive>

Every node of this schema was promoted to a distinguished field so that the values can be used for decision making in the orchestration. The first decision made is on the "IsPermissionRequired" field.

If permission is required (IsPermissionRequired=="YES"), BizTalk will use the configuration info in the XML trigger to format the email message. Here is the snippet of how the email message is constructed.

SQLJobEmailMessage.EmailBody
 = new Eai.OrchestrationHelpers.XlangCustomFormatters.RawString(
 MsgSchedulerTriggerSQLJobReceive.EmailMessageBodyForPermissionRequest +
 " " +
 "By moving the file, you are either giving permission to the process, or disapprove of the process." +
 " " +
 "This is the file to move: \"" + PermissionTriggerToBeGenereatedHere +
 "\" " +
 "(You may find it easier to open the destination folder first, then navigate to the sibling folder to get to this file)" +
 " " +
 "To approve, move(NOT copy) the file here: " + MsgSchedulerTriggerSQLJobReceive.PermissionRequestApprovedPath +
 " " +
 "To disapprove, move(NOT copy) the file here: " + MsgSchedulerTriggerSQLJobReceive.PermissionRequestNotApprovedPath +
 " " +
 "The file will be IMMEDIATELY picked up by the automated process. This is normal. You should receive a message soon that the file is processed." +
 " " +
 "Thank you!"
 );
SQLJobSendNotification(Microsoft.XLANGs.BaseTypes.Address) = "mailto:" + MsgSchedulerTriggerSQLJobReceive.EmailRecipientToList;
SQLJobEmailMessage.EmailBody(Microsoft.XLANGs.BaseTypes.ContentType) = "text/html";
SQLJobEmailMessage(SMTP.Subject) = "Requesting Permission to Start the " + MsgSchedulerTriggerSQLJobReceive.BusinessProcessName;
SQLJobEmailMessage(SMTP.From) = MsgSchedulerTriggerSQLJobReceive.EmailFrom;
SQLJobEmailMessage(SMTP.CC) = MsgSchedulerTriggerSQLJobReceive.EmailRecipientCCList;
SQLJobEmailMessage(SMTP.EmailBodyFileCharset) = "UTF-8";
SQLJobEmailMessage(SMTP.SMTPHost) = "localhost";
SQLJobEmailMessage(SMTP.MessagePartsAttachments) = 2;

After the Permission request email is sent, the next step is to generate the actual Permission Trigger file. A correlation set is used here on SQLJobName and a newly generated GUID field.

<?xml version="1.0" encoding="utf-8"?><ns0:SQLJobAuthorizationTrigger xmlns:ns0="somethingsomething"><SQLJobName>Data Push</SQLJobName><CorrelationGuid>9f7c6b46-0e62-46a7-b3a0-b5327ab03753</CorrelationGuid></ns0:SQLJobAuthorizationTrigger>

The end user (the human intervention piece) will either grant permission for this process, or deny it, by moving the Permission Trigger file to either the "Approved" folder or the "NotApproved" folder. A parallel Listen shape is waiting for either response.

The next set of steps decide how the SQL Job is to be called, or whether it is called at all. If permission denied, it simply sends out a notification. If permission is granted, then the flag (IsProcessAsync) in the original Process Trigger is used. The synchonous part is not really synchronous, but a loop timer to check the status within the calling stored procedure (for more information, check out my previous post: http://geekswithblogs.net/LifeLongTechie/archive/2010/11/01/execute-sql-job-synchronously-for-biztalk-via-a-stored-procedure.aspx) If it's async, then the sp starts the job and BizTalk sends out an email.

And of course, some error notification:

Footnote: The next version of this orchestration will have an additional parallel line near the Listen shape with a Delay built in and a Loop to send out a daily reminder if no response has been received from the end user. The synchronous part is used to gather results and execute a data clean up process so that the SQL Job can be re-tried. There are manu possibilities here.

Using SSIS to send a HTML E-Mail Message with built-in table of Counts.

Kevin Shyr — Wed, 02 Feb 2011 21:47:05 GMT

For the record, this can be just as easily done with a .NET class with a DLL call. The two major reasons for this ending up as a SSIS package are:

There are a lot of SQL resources in the organization for maintenance, but not as many .NET developers.
There is an existing automated process that links up SQL Jobs (more on that in the next post), and this is part of that process.

To start, this is what the SSIS looks like:

The first part of the control flow is just for the override scenario.

In the Execute SQL Task, it calls a stored procedure, which already formats the result into XML by using "FOR XML PATH('Row'), ROOT(N'FieldingCounts')". The result XML string looks like this:

<FieldingCounts>
<Row>
 <CellId>M COD</CellId>
 <Mailed>64</Mailed>
 <ReMailed>210</ReMailed>
 <TotalMail>274</TotalMail>
 <EMailed>233</EMailed>
 <TotalSent>297</TotalSent>
</Row>
<Row>
 <CellId>M National</CellId>
 <Mailed>11</Mailed>
 <ReMailed>59</ReMailed>
 <TotalMail>70</TotalMail>
 <EMailed>90</EMailed>
 <TotalSent>101</TotalSent>
</Row>
<Row>
 <CellId>U COD</CellId>
 <Mailed>91</Mailed>
 <ReMailed>238</ReMailed>
 <TotalMail>329</TotalMail>
 <EMailed>291</EMailed>
 <TotalSent>382</TotalSent>
</Row>
<Row>
 <CellId>U National</CellId>
 <Mailed>63</Mailed>
 <ReMailed>286</ReMailed>
 <TotalMail>349</TotalMail>
 <EMailed>374</EMailed>
 <TotalSent>437</TotalSent>
</Row>
</FieldingCounts>

This result is saved into an internal SSIS variable with the following settings on the General tab and the Result Set tab:

Now comes the trickier part. We need to use the XML Task to format the XML string result into an HTML table, and I used Direct input XSLT

And here is the code of XSLT:

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="html" indent="yes"/>
<xsl:template match="/ROOT">
 <table border="1" cellpadding="6">
 <tr>
 <td></td>
 <td>Mailed</td>
 <td>Re-mailed</td>
 <td>Total Mail (Mailed, Re-mailed)</td>
 <td>E-mailed</td>
 <td>Total Sent (Mailed, E-mailed)</td>
 </tr>
 <xsl:for-each select="FieldingCounts/Row">
 <tr>
 <xsl:for-each select="./*">
 <td>
 <xsl:value-of select="." />
 </td>
 </xsl:for-each>
 </tr>
 </xsl:for-each>
 </table>
</xsl:template>
</xsl:stylesheet>

Then a script task is used to send out an HTML email (as we are all painfully aware that SSIS Send Mail Task only sends plain text)

using System;

using System.Data;

using Microsoft.SqlServer.Dts.Runtime;

using System.Windows.Forms;

using System.Net.Mail;

using System.Net;

namespace ST_b829a2615e714bcfb55db0ce97be3901.csproj

{

[System.AddIn.AddIn("ScriptMain", Version = "1.0", Publisher = "", Description = "")]

public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase

{

#region VSTA generated code

enum ScriptResults

{

Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,

Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure

};

#endregion

public void Main()

{

String EmailMsgBody = String.Format("<HTML><BODY>{0}{1}</BODY></HTML>"

, Dts.Variables["Config_SMTP_MessageSourceText"].Value.ToString()

, Dts.Variables["InternalStr_CountResultAfterXSLT"].Value.ToString());

MailMessage EmailCountMsg = new MailMessage(Dts.Variables["Config_SMTP_From"].Value.ToString().Replace(";", ",")

, Dts.Variables["Config_SMTP_Success_To"].Value.ToString().Replace(";", ",")

, Dts.Variables["Config_SMTP_SubjectLinePrefix"].Value.ToString() + " " + Dts.Variables["InternalStr_FieldingDate"].Value.ToString()

, EmailMsgBody);

//EmailCountMsg.From.

EmailCountMsg.CC.Add(Dts.Variables["Config_SMTP_Success_CC"].Value.ToString().Replace(";", ","));

EmailCountMsg.IsBodyHtml = true;

SmtpClient SMTPForCount = new SmtpClient(Dts.Variables["Config_SMTP_ServerAddress"].Value.ToString());

SMTPForCount.Credentials = CredentialCache.DefaultNetworkCredentials;

SMTPForCount.Send(EmailCountMsg);

Dts.TaskResult = (int)ScriptResults.Success;

}

Note on this code: notice the email list has Replace(";", ","). This is only here because the list is configurable in the SQL Job Step at Set Values, which does not react well with colons as email separator, but system.Net.Mail only handles comma as email separator, hence the extra replace in the string.

The result is a nicely formatted email message with count information:

Informative SSIS error message to be used in notification expression

Kevin Shyr — Wed, 12 Jan 2011 22:53:15 GMT

"The " + @[System::TaskName] + " task captured error in the " + @[System::PackageName] + " package at " + (DT_WSTR, 50) @[System::EventHandlerStartTime] + ". The message is:

" + @[System::ErrorDescription]

SSIS Send Mail Task and ForceExecutionValue Error

Kevin Shyr — Fri, 03 Dec 2010 17:34:41 GMT

I tried to use the "ForcedExecutionValue" on several Send Mail Tasks and log the execution into a ExecValueVariable so that at the end of the package I can log into a table to say whether the data check is successful or not (by determine whether an email was sent out)

I set up a Boolean variable that is accessible at the package level, then set up my Send Mail Task as the screenshot below with Boolean as my ForcedExecutionValueType. When I run the package, I got the error described below.

Just to make sure this is not another issue of SSIS having with Boolean type ( you also can't set variable value from xp_cmdshell of type Boolean), I used variables of types String, Int32, DateTime with the corresponding ForcedExecutionValueType. The only way to get around this error, was to set my variable to type Object, but then when you try to get the value out later, the Object is null.

I didn't spend enough time on this to see whether it's really a bug in SSIS or not, or is this just how Send Mail Task works. Just want to log the error and will circle back on this later to narrow down the issue some more. In the meantime, please share if you have run into the same problem. The current workaround is to attach a script task at the end.

Also, need to note 2 existing limitation:

Data check needs to be done serially because every check needs to be inner join to a master table. The master table has all the data in a single XML column and hence need to be retrieved with XQuery (a fundamental design flaw that needs to be changed)
The next iteration will be to change this design into a FOR loop and pull out the checking query from a table somewhere with all the info needed for email task, but is being put to the back of the priority.

Error Message:

Error: 0xC001F009 at CountCheckBetweenODSAndCleanSchema: The type of the value being assigned to variable "User::WasErrorEmailEverSent" differs from the current variable type. Variables may not change type during execution. Variable types are strict, except for variables of type Object.
Error: 0xC0019001 at Send Mail Task on count mismatch: The wrapper was unable to set the value of the variable specified in the ExecutionValueVariable property.

Screenshot of my Send Mail Task setup:

Execute SQL Job synchronously for BizTalk via a Stored Procedure call

Kevin Shyr — Mon, 01 Nov 2010 20:18:30 GMT

The async one was very easy to do. However, in the process automation task I was given, there was a need to execute SQL Job and waiting for the execution result. I couldn't find anything on the web to do what I wanted, so I make the following stored procedure.

In addition, this stored procedure is quasi-synchronous. I start the job and then check the status. You can change the timing of the status check, but for the example below, it is "synchronous within 30 seconds." I wish there is a better way to do this, but at my current knowledge base I only came up with this:

CREATE PROCEDURE [SQLJobExecSyncByName]
 @SQLJobName AS [varchar](128),
 @SQLJobStepName AS [varchar](128),
 @MinutesToWait AS [int],
 @JobExecStatus AS [bit] output
AS
BEGIN
 -- SET NOCOUNT ON added to prevent extra result sets from
 -- interfering with SELECT statements.
 SET NOCOUNT ON;

 DECLARE @JobStartDateTime AS DATETIME
 DECLARE @SPQuitDateTime AS DATETIME
 DECLARE @JobStartDateAsString AS VARCHAR(8)
 DECLARE @JobStartTimeAsString AS VARCHAR(6)
 DECLARE @SPOutcome AS BIT
 SET @JobStartDateTime = DATEADD(mi, -1, GETDATE())
 SET @SPQuitDateTime = DATEADD(mi, @MinutesToWait, GETDATE())
 SET @SPOutcome = 0
 SELECT @JobStartDateAsString = CONVERT(VARCHAR(8), @JobStartDateTime, 112)
 SELECT @JobStartTimeAsString = REPLACE(CONVERT(VARCHAR(8), @JobStartDateTime, 108), ':', '')

 -- Create a temp table to store result
 CREATE TABLE #Result_sp_help_jobhistory
 (
 instance_id [int]
 , job_id [uniqueidentifier]
 , job_name [sysname]
 , step_id [int]
 , step_name [sysname]
 , sql_message_id [int]
 , sql_severity [int]
 , [message] [nvarchar](1024)
 , run_status [int]
 , run_date [int]
 , run_time [int]
 , run_duration [int]
 , operator_emailed [nvarchar](20)
 , operator_netsent [nvarchar](20)
 , operator_paged [nvarchar](20)
 , retries_attempted [int]
 , [server] [nvarchar](30)
 )

 -- Start the job @SQLJobName at step @SQLJobStepName
 EXEC msdb.dbo.sp_start_job
 @job_name = @SQLJobName,
 @server_name = @@SERVERNAME,
 @step_name = @SQLJobStepName;

 WHILE (GETDATE() < @SPQuitDateTime) -- not time to quit yet
 BEGIN
 -- Check to see if the job has finished successfully
 INSERT INTO #Result_sp_help_jobhistory
 (instance_id
 , job_id
 , job_name
 , step_id
 , step_name
 , sql_message_id
 , sql_severity
 , [message]
 , run_status
 , run_date
 , run_time
 , run_duration
 , operator_emailed
 , operator_netsent
 , operator_paged
 , retries_attempted
 , [server]
 )
 EXEC msdb.dbo.sp_help_jobhistory
 @job_name = @SQLJobName,
 @start_run_date = @JobStartDateAsString,
 @start_run_time = @JobStartTimeAsString,
 --@run_status = 1, -- Success
 @mode = 'FULL'

 IF(EXISTS(SELECT *
 FROM #Result_sp_help_jobhistory
 WHERE job_name = @SQLJobName
 AND step_name = '(Job outcome)'
 AND run_status = 1
 AND @JobStartDateTime < CONVERT(DATETIME, CONVERT(VARCHAR(8), run_date) + ' ' + SUBSTRING(CONVERT(VARCHAR(6), run_time), 1, 2) + ':' + SUBSTRING(CONVERT(VARCHAR(6), run_time), 3, 2) + ':' + SUBSTRING(CONVERT(VARCHAR(6), run_time), 5, 2), 112)
 )
 )
 BEGIN
 -- The job has succceeded at least once after it was called
 SET @SPOutcome = 1
 GOTO FinishingStoredProcedureCall;
 END

 IF(EXISTS(SELECT *
 FROM #Result_sp_help_jobhistory
 WHERE job_name = @SQLJobName
 AND step_name = '(Job outcome)'
 AND run_status = 0
 AND @JobStartDateTime < CONVERT(DATETIME, CONVERT(VARCHAR(8), run_date) + ' ' + SUBSTRING(CONVERT(VARCHAR(6), run_time), 1, 2) + ':' + SUBSTRING(CONVERT(VARCHAR(6), run_time), 3, 2) + ':' + SUBSTRING(CONVERT(VARCHAR(6), run_time), 5, 2), 112)
 )
 )
 BEGIN
 -- The job has failed after it was called
 GOTO FinishingStoredProcedureCall;
 END

 -- Clean up for the next run
 TRUNCATE TABLE #Result_sp_help_jobhistory

 -- check every 30 seconds
 WAITFOR DELAY '00:00:30';
 END

 FinishingStoredProcedureCall:
 SET @JobExecStatus = @SPOutcome
 DROP TABLE #Result_sp_help_jobhistory;

END

This SP was made to return an output parameter as result as to maximize reusability of the logic. In order to use this from BizTalk, it was hooked into the following SP:

CREATE PROCEDURE [SQLJobExecSyncByNameForBizTalk]
    @SQLJobName AS [varchar](128),
    @SQLJobStepName AS [varchar](128),
    @MinutesToWait AS [int]
AS
BEGIN
    -- SET NOCOUNT ON added to prevent extra result sets from
    -- interfering with SELECT statements.
    SET NOCOUNT ON;

    DECLARE @JobExecStatus AS BIT

    EXEC [ODS].[SQLJobExecSyncByName]
        @SQLJobName = @SQLJobName,
        @SQLJobStepName = @SQLJobStepName,
        @MinutesToWait = @MinutesToWait,
        @JobExecStatus = @JobExecStatus OUTPUT;

    SELECT CONVERT(VARCHAR(8), @JobExecStatus)
        FOR XML PATH(N''), ROOT(N'JobExecStatus');

END