I come across this problem a lot. I often get work to fix problems that largely stem from bad interface design.
Architects know how important interfaces are whether they be WSDL, C# interfaces, stored procedure parameters etc. Developers in my experience should get a grip on this stuff.
Once you have some requirements spending some quality time working out what your interface is important. Developers need to hammer this into their skulls, it's pritty simple. This isn't an exhaustive post but try and follow these simple steps for good interface design (as I'm doing WSDLs at the moment so the examples will be WSDL\Xsd Schema interfaces). Please feel free to feed back your agreements\disagreements other posts which are better etc...
1) You need a list of requirements to start i.e. data fields, 1 to many relationships, min and max occurrences, business rules and how these fields should be used.
2) Ask what are the important pieces of data here i.e. if you are writing a document delivery system, the document is probably the parent and will be closer to the root of your xsd schema how, where, why, what is delivered will come under that.
3) Group common data elements together. If your document delivery system needs to email and fax documents create a common schema type called DeliveryChannel or something and put ALL the email and fax data elements in there. Maybe look up on line for some example generic\flexible schemas that deal with address information and copy them or get ideas. Good xsd design is a big topic and I won't cover it here.
4) You should now have made a start i.e. a schema that is a best attempt with what you know so far.
5) TEST YOUR INTERFACE: this isn't code testing, your interface might just be on a bit of paper at this stage it doesn't matter. I can think of two tests the interface must pass:
- The interface must be able to implement your business requirements, trick them off one at a time.
- The interface must work with existing and future interfaces which need to call it and it will need to call find out what the data fields, min and max occurrences, business rules are for the api's, components or services that will use your interface or your interface will use. Create an excel spreadsheet and map this stuff.
6) Modify your interface when it fails these tests until it passes them.
7) Finally check you don’t have redundant fields, the schema is flexible enough for the future BUT NOT RIDICULOUSLY GENERIC and related data is in the same place not littered throughout.
Try to do this or something similar at the very least before you rush in and start coding.
I uninstalled AppFabric Beta 2 and installed AppFabric RC.
The AppFabricEventCollection Service is started (runs under Local Service which is a dbo_owner on the Monitoring Database to prove this wasn’t the issue).
The SQLServerAgent Service is started.
Nothing is being written to the Monitoring DB Staging Table and thus nothing is being written to the Event tables or seen in the AppFabric Dashboard.
Nothing has been written to the following event logs
- Microsoft-Windows-Application Server-System Services\Admin
- Microsoft-Windows-Application Server-System Services\Operational
The Microsoft-Windows-Application Server-System Services\Debug event log is not shown in the event viewer.
The WCF configuration appears fine the connection string to the Monitoring DB is correct. Monitoring is set to “Trouble Shooting” and no errors are shown on the “Configure WCF and WF for Application” dialog.
So the problem seems to lie with either AppFabric which writes to the event log or the AppFabricEventCollection Service.
I thought I was flummoxed...
However one of my colleagues said have you checked the etwProviderId? I was using a config which was created under AppFabric Beta 2 which had a different etwProviderId. So I deleted the following section and all other references to AppFabric monitoring from the web.config and then recreated them using IIS the “Configure WCF and WF for Application” dialog and set the level to TroubleShooting.
<diagnostics etwProviderId="6b44a7ff-9db4-4723-b8cf-1b584bf1591b">
<endToEndTracing propagateActivity="true" messageFlowTracing="true" />
</diagnostics>
I then called a service to create some log entries. Still nothing was written to the Monitoring DB Staging Table...
I checked the Microsoft-Windows-Application Server-System Services\Admin event log. It had the following entry...
Configuration error. Please see the details to correct the problem. \rDetailed information:\r Filename: \\?\C:\Users\xxx\Documents\dotnetdev\Frameworks\SOA\xxx.SOA.Framework\xxx.SOA.Framework.MockServices\SimpleServiceParent\web.config
Error: Cannot read configuration file due to insufficient permissions
System.UnauthorizedAccessException: Filename: \\?\C:\Users\xxx\Documents\dotnetdev\Frameworks\SOA\xxx.SOA.Framework\xxx.SOA.Framework.MockServices\SimpleServiceParent\web.config
Error: Cannot read configuration file due to insufficient permissions
And guess who the user was... Local Service yes yes I should have used a better User in the AppFabric RC setup to run the AppFabricEventCollection Service under!
So I changed the user to a more appropriate one and removed Local Service as a DBO and hay presto!
In the past I have noticed that system testers find it difficult to test middleware applications such as those implemented using BizTalk. System testers generally don't want to read BizTalk xml messages dumped out to disk, they like to be able to query for their test results via some form of GUI. Many are adept at writing SQL to return result sets containing the test data they want to analyze.
I came up with the idea of using BAM relationships between different activities within a single view to log before and after snapshots of BizTalk messages. So when messages were transformed by maps, orchestrations or the BRE system testers could test those transforms. You can then give system testers’ access to the BAM Portal Website where they could view and query these before and after snapshots and match the actuals against the expected results to pass or fail their tests.
This solution for system testing only works for certain scenarios’ with certain types of schemas as each message logged cannot contain repeating records if it is to be logged to BAM.
So what am I on about? Take a single record schema which validates an xml message like the one below as I said before this technique won't work with messages which contain multiple records.
<ns1:SampleTransaction TransactionID="123456" ProductName="ABC" Amount="1234.56" Quantity="24" xmlns:ns1="http://SynbogalooSamples/SampleTransaction/1.0.0.0" />
Each time a map, orchestration or business rules were applied to a message I wanted to send messages to BAM so I was basically just logging messages to BAM each time the contents of the message changed. I used BAM relationships to associate the message before it was transformed to the message after it was transformed so the tester could assess whether the transformation worked.
OK this was all easy but very time consuming using the Excel add-in to create the xlsb files for each schema so I wrote a simple xlsb file generator. Note: For this code to work with schemas that have xsd types other than DateTime, Integer, Double and String you will need to add to the switch statement in the CreateActivity method.
using System;
using System.Collections.Generic;
using System.Text;
using System.Xml;
using System.IO;
namespace BAMXslbGenerator
{
class Program
{
private const string UCName = "Name";
private const string LCName = "name";
private const string View = "View";
private const string BAMDefinition = "BAMDefinition";
private const string BAMDefinitionNamespace = "http://schemas.microsoft.com/BizTalkServer/2004/10/BAM";
private const string Extension = "Extension";
private const string OWC = "OWC";
private const string OWCNamespace = "urn:schemas-microsoft-com:office:excel";
private const string ActivityView = "ActivityView";
private const string ActivityRef = "ActivityRef";
private const string Activity = "Activity";
private const string Alias = "Alias";
private const string CheckpointRef = "CheckpointRef";
private const string Id = "ID";
private const string Checkpoint = "Checkpoint";
private const string XpathToAttribute =
"(//*[local-name()='element'])[last()]/*[local-name()='complexType']/*[local-name()='attribute']";
private const string DataType = "DataType";
private const string DataLength = "DataLength";
private const string Length = "length";
private const string MaxLength = "maxLength";
private const string XpathToLength = "*[local-name()='simpleType']/*[local-name()='restriction']/*[local-name()='length']/@value";
private const string XpathToMaxLength = "*[local-name()='simpleType']/*[local-name()='restriction']/*[local-name()='maxLength']/@value";
private const string SqlDateTime = "DATETIME";
private const string SqlNvarchar = "NVARCHAR";
private const string SqlInt = "INT";
private const string SqlFloat = "FLOAT";
private const string XsdDateTime = "xs:dateTime";
private const string XsdString = "xs:string";
private const string XsdInteger = "xs:int";
private const string XsdDouble = "xs:double";
private static StringBuilder xlsbText = null;
static void Main(string[] args)
{
try
{
//Check for help
if (args.Length == 0)
{
ShowUsageMessage();
}
else if ((args[0] == "/?") || (args[0] == "/help"))
{
ShowUsageMessage();
}
else
{
//Parse out command line arguements
string xlsbFileNameArg = GetCommand("/x", args);
string viewNameArg = GetCommand("/v", args);
string activityNameArg = GetCommand("/a", args).Replace("[","").Replace("]","");
string schemaArg = GetCommand("/s", args).Replace("[", "").Replace("]", ""); ;
if (xlsbFileNameArg != string.Empty && viewNameArg != string.Empty
&& activityNameArg != string.Empty && schemaArg != string.Empty)
{
string[] activityNames = activityNameArg.Split(new string[] { "," }, StringSplitOptions.None);
string[] schemas = schemaArg.Split(new string[] { "," }, StringSplitOptions.None);
GenerateXlsbXml(viewNameArg, activityNames, schemas);
//Write out the file
File.Delete(xlsbFileNameArg);
File.AppendAllText(xlsbFileNameArg, xlsbText.ToString(), Encoding.Unicode);
}
else
{
Console.WriteLine("Incorrect arguements!");
}
}
}
catch (System.Exception ex)
{
Console.Write(ex.ToString());
}
}
private static string GetCommand(string commandToken, string[] args)
{
string commandValue = string.Empty;
for (int i = 0; i < args.Length; i++)
{
if (args[i] == commandToken)
{
commandValue = args[i + 1];
break;
}
}
return commandValue;
}
private static void ShowUsageMessage()
{
Console.WriteLine("Generates an xlsb BAM definition file conaining one view and multiple schema definitions to log to BAM.");
Console.WriteLine("Syntax: BAMXlsbGenerator");
Console.WriteLine("Commands:");
Console.WriteLine("\t/x \txlsb file name");
Console.WriteLine("\t/v \tBAM View definition name");
Console.WriteLine("\t/a[] \tActivity names enclosed in [] comma separated, there must be one activity name per schema.");
Console.WriteLine("\t/s[] \tSchema names enclosed in [] comma separated, there must be one schema name per activity.");
Console.WriteLine("\t/? or /help \tDisplay this usage message");
}
private static void GenerateXlsbXml(string viewName, string[] activityNames, string[] schemas)
{
//Create an writer for building the xlsb
xlsbText = new StringBuilder();
XmlWriterSettings xmlWriterSettings = new XmlWriterSettings();
xmlWriterSettings.Indent = true;
XmlWriter xlsbWriter = XmlWriter.Create(xlsbText, xmlWriterSettings);
//Start building xml
xlsbWriter.WriteStartDocument();
xlsbWriter.WriteStartElement(BAMDefinition, BAMDefinitionNamespace);
if (schemas.Length != activityNames.Length)
{
throw new ApplicationException("The number of activity names and schemas don't match.");
}
Dictionary<string, string>[] checkpointIDsForSchemas = new Dictionary<string, string>[schemas.Length];
string[] activityIDs = new string[schemas.Length];
//Create the activities
for (int i = 0; i < schemas.Length; i++)
{
Dictionary<string, string> checkpointIDs;
string activityID;
CreateActivity(xlsbWriter, schemas[i], activityNames[i],
out checkpointIDs, out activityID);
checkpointIDsForSchemas[i] = checkpointIDs;
activityIDs[i] = activityID;
}
//Now create the view
xlsbWriter.WriteStartElement(View);
xlsbWriter.WriteAttributeString(UCName, viewName);
string viewID = Id + Guid.NewGuid().ToString("N");
xlsbWriter.WriteAttributeString(Id, viewID);
//Create Activity Views
for (int i = 0; i < schemas.Length; i++)
{
CreateActivityView(xlsbWriter, activityNames[i],
checkpointIDsForSchemas[i], activityIDs[i]);
}
xlsbWriter.WriteEndElement(); //END View
xlsbWriter.WriteStartElement(Extension);
xlsbWriter.WriteElementString(OWC, OWCNamespace, "");
xlsbWriter.WriteEndElement(); //END Extension
xlsbWriter.WriteEndElement(); //END BAMDefinition
xlsbWriter.WriteEndDocument();
xlsbWriter.Flush();
}
private static void CreateActivityView(XmlWriter xlsbWriter, string activityName,
Dictionary<string, string> checkpointIDs, string activityID)
{
xlsbWriter.WriteStartElement(ActivityView);
xlsbWriter.WriteAttributeString(UCName, View + activityName);
xlsbWriter.WriteAttributeString(Id, Id + Guid.NewGuid().ToString("N"));
xlsbWriter.WriteAttributeString(ActivityRef, activityID);
//Loop through the dictionary and create the Aliases
foreach (string key in checkpointIDs.Keys)
{
xlsbWriter.WriteStartElement(Alias);
xlsbWriter.WriteAttributeString(UCName, key);
xlsbWriter.WriteAttributeString(Id, Id + Guid.NewGuid().ToString("N"));
xlsbWriter.WriteElementString(CheckpointRef, checkpointIDs[key]);
xlsbWriter.WriteEndElement(); //END Alias
}
xlsbWriter.WriteEndElement(); //END ActivityView
}
private static void CreateActivity(XmlWriter xlsbWriter, string schema,
string activityName, out Dictionary<string, string> checkpointIDs,
out string activityID)
{
xlsbWriter.WriteStartElement(Activity);
xlsbWriter.WriteAttributeString(UCName, activityName);
activityID = Id + Guid.NewGuid().ToString("N");
xlsbWriter.WriteAttributeString(Id, activityID);
//Create a dictionary object to store the IDs in
checkpointIDs = new Dictionary<string, string>();
//Open the schema
XmlDocument xmlDocument = new XmlDocument();
xmlDocument.Load(schema);
//The rules are
XmlNodeList xmlNodeList = xmlDocument.SelectNodes(XpathToAttribute);
//Loop through all the attributes and create the new xslb schema
foreach (XmlNode xmlNode in xmlNodeList)
{
string checkpointID = Id + Guid.NewGuid().ToString("N");
checkpointIDs.Add(xmlNode.Attributes[LCName].Value, checkpointID);
//Add the data type attributes
if (xmlNode.OuterXml.Contains(XsdDateTime))
{
xlsbWriter.WriteStartElement(Checkpoint);
xlsbWriter.WriteAttributeString(UCName, xmlNode.Attributes[LCName].Value);
xlsbWriter.WriteAttributeString(Id, checkpointID);
xlsbWriter.WriteAttributeString(DataType, SqlDateTime);
xlsbWriter.WriteEndElement(); //End Checkpoint
}
else if (xmlNode.OuterXml.Contains(XsdInteger))
{
xlsbWriter.WriteStartElement(Checkpoint);
xlsbWriter.WriteAttributeString(UCName, xmlNode.Attributes[LCName].Value);
xlsbWriter.WriteAttributeString(Id, checkpointID);
xlsbWriter.WriteAttributeString(DataType, SqlInt);
xlsbWriter.WriteEndElement(); //End Checkpoint
}
else if (xmlNode.OuterXml.Contains(XsdDouble))
{
xlsbWriter.WriteStartElement(Checkpoint);
xlsbWriter.WriteAttributeString(UCName, xmlNode.Attributes[LCName].Value);
xlsbWriter.WriteAttributeString(Id, checkpointID);
xlsbWriter.WriteAttributeString(DataType, SqlFloat);
xlsbWriter.WriteEndElement(); //End Checkpoint
}
else if (xmlNode.InnerXml.Contains(XsdString))
{
xlsbWriter.WriteStartElement(Checkpoint);
xlsbWriter.WriteAttributeString(UCName, xmlNode.Attributes[LCName].Value);
xlsbWriter.WriteAttributeString(Id, checkpointID);
xlsbWriter.WriteAttributeString(DataType, SqlNvarchar);
if (xmlNode.InnerXml.Contains(Length))
{
xlsbWriter.WriteAttributeString(DataLength,
xmlNode.SelectSingleNode(XpathToLength).InnerText);
}
else if (xmlNode.InnerXml.Contains(MaxLength))
{
xlsbWriter.WriteAttributeString(DataLength,
xmlNode.SelectSingleNode(XpathToMaxLength).InnerText);
}
else
{
xlsbWriter.WriteAttributeString(DataLength, "255");
}
xlsbWriter.WriteEndElement(); //End Checkpoint
}
else if (xmlNode.OuterXml.Contains(XsdString))
{
xlsbWriter.WriteStartElement(Checkpoint);
xlsbWriter.WriteAttributeString(UCName, xmlNode.Attributes[LCName].Value);
xlsbWriter.WriteAttributeString(Id, checkpointID);
xlsbWriter.WriteAttributeString(DataType, SqlNvarchar);
xlsbWriter.WriteAttributeString(DataLength, "255");
xlsbWriter.WriteEndElement(); //End Checkpoint
}
else
{
Console.Write("Unknown xsd datatype: " + xmlNode.OuterXml + " continuing to create xlsb.");
checkpointIDs.Remove(xmlNode.Attributes[LCName].Value);
}
}
xlsbWriter.WriteEndElement(); //End Activity
}
}
}
How to call example: BAMXlsbGenerator /x "C:\Projects\SynboogalooSamples\Test.xlsb" /v TestView /a [TestActivityA,TestActivityB] /s [C:\Projects\SynboogalooSamples\SampleTransactionA.xsd,C:\Projects\SynboogalooSamples\SampleTransactionB.xsd]
This saved me a lot of time all I needed to do now to finish setting up BAM was deploy the xlsb BAM definition using the BM.exe tool and use the Tracking Profile Editor to associate the message payload with the BAM activity definition about 5 minutes of drag n' drop work. In the BAM Portal system testers can query transformed data and can even query the BAM tables if they want. They are able through BAM relationships to view the message contents each time it has been modified and logged to BAM.
I found this technique was a quick and simple way of empowering system testers to test BizTalk applications.
I should note another option for System Testing EAI solutions is to go with an implementation of FIT (Framework For Integration Testing) here is a blog with more info.
Rob
This is a simple explanation of how to join together two schemas with a 1:1 relationship between them in the Biztalk Mapper. I came across this problem when I had 2 legacy csv files which needed to be merged (using a unique identifier field in each file) before further processing could occur in a Biztalk Orchestration.
I will show by example. The example is a bit convoluted, but stay with me ;) different departments are ordering the same parts but are paying different prices we want to see the difference in price for the parts. There are also fields missing from each field which we want to add to the finished schema.
1) So first create 2 schemas one for each department, note the common PartID which is the unique identifier and price, but also not that one file comes with a Part name and another with a manufacturer:
2) Create the Output Schema
3) Now create an orchestration which will call the Map to join the two schemas by taking them both in and using a transform shape to map the two then it will output them
4) Set the transform properties as below create a new map set the destination to the output map
5) Which will create a map
6) Finally the biz … ok on the map join the common fields from the first schema (but not the second) with fields in the output schema now add a script functoid and set it up like this. Using an XSLT template it will search the second message for the values (simple xpath) where the PartID in the first message matches the PartID in the second message. It will then return the value of the price attribute in the second message. From this you can see how powerful using inline xslt is.
7) Add another script block to get the manufacturer field set the inline script buffer to:
9) Join up the schemas in the final map like so … THE END!
R. Addis