BizTalk Server 2004/2006

MS BRE: Using Compensation handlers

Charles Young — Wed, 23 Jan 2008 00:48:52 GMT

Being known for my interest in rules processing, I quite often get asked to help with problems with MS BRE. A couple of days ago, I was asked to help investigate an issue occurring in production for a BizTalk Server application. Occasionally, in a fairly high throughput system, BizTalk logs an error stating that a problem has been encountered while executing a rule set. That is the only information provided, with no hint of what the problem might be, and because the issue only occurs intermittently under real-world conditions in the production environment, it was not obvious how to obtain further insight without disrupting live operations.

This article investigates one way of handling this deilemma through the use of the compensation handling feature of Microsoft's Business Rules Engine. It goes on to discuss the broader use of compensation handling in rule processing.

http://geekswithblogs.net/cyoung/archive/2008/01/23/118804.aspx

MS BRE: The Rules Engine Update (REU) Service and Policy Execution

Charles Young — Sun, 13 Jan 2008 22:13:24 GMT

A question came up tonight on BizTalkGurus on my favourite subject of rule engines. I don’t blog enough these days, so this gives me an excuse. Essentially, the question concerned an incorrect, but understandable, suspicion that MS BRE may be using remoting to execute rule sets out-of-process. This is not the case. You can find an article describing what actually happens at:

http://geekswithblogs.net/cyoung/archive/2008/01/13/118506.aspx

Using .NET 3.5 and Visual Studio 2005

Charles Young — Sat, 01 Dec 2007 11:23:21 GMT

The recent release of Visual Studio 2008 and .NET Fx 3.5 is causing some confusion. Microsoft released these two technologies together for good reason. The wonderful new LinQ technologies introduced in .Net 3.5 rely on explicit compiler-level support, and therefore require LinQ aware compilers in Visual Studio. The new version of visual Studio provides these compilers, allowing developers to take advantage of the new monadic syntax. In addition, Visual Studio has several new features designed to make it easier to exploit NET 3.5 features such as Ajax and the foundation libraries (WCF, WF and WPF).

The problem is that by tying the release of .NET 3.5 to Visual Studio 2008, the impression is given that, unless you are ready to upgrade to the new version of the IDE, there is no point thinking about upgrading to the new version of the framework. This is simply not the case. The .NET framework does not have any built-in dependency on Visual Studio, let alone a specific version of Visual Studio. More to the point, Microsoft has long since split the versioning of the framework from the versioning of the run-time environment. .NET 3.5 continues to exploit version 2.0 of the CLR. Visual Studio 2005 is perfectly happy to compile your code against .NET 3.5 assemblies. They are just assemblies. Even more compelling is the realisation that most of the assemblies in .NET 3.5 are identical to those in .NET 3.0 (same version number). There are some new assemblies with new features. .NET 3.5 is just .NET 3.0 with extra stuff.

Why is this important? Well, not everyone is ready to upgrade to Visual Studio 2008. Apart from the expense this involves, consider the dilemma of BizTalk Server developers. Currently, there are no Visual Studio 2008 bindings for BizTalk Server (i.e., you can't create BizTalk Server project types in the new IDE). This, we are assured, will be addressed at some point, but that could be months away. For the time being, BizTalk developers are stuck with Visual Studio 2005 :-( Hence, some people are currently discounting the possibility of using .Net 3.5 because they believe, quite incorrectly, that it requires an upgrade to Visual Studio 2008.

There are issues, of course. As well as the absence of compiler support for LinQ, Visual Studio 2005 does not have access to various new project and file templates and tools that support the new version of the framework. Developers may need to do more coding in Visual Studio 2005 than would be necessary in Visual Studio 2008. This is often a small price to pay, however, in order to access the improvements in 3.5. As an example, consider the new integration between WF and WCF, provided in the new System.WorkflowServices assembly. The integration is provided via the new WorkflowServiceHost class and a couple of new activities. Visual Studio 2008 has new template support for building workflow services, and comes with a very useful new WCF test harness. However, exploiting this new functionality in Visual Studio 2005 is trivial. Create a WF workflow library, add a reference to System.WorkflowServices and add the new activities to your tool box. Finally, use the WCF Service template to add a service class to your project and you are just about in the same position as you would be in Visual Studio 2008 if you used the new Workflow Service project template. You'll need to write a couple of lines of code to use WorkflowServiceHost to host your service, of course. Off you go, and enjoy .NET 3.5.

DebugView and the elusive 'Capture Global Win32' option

Charles Young — Sun, 16 Sep 2007 13:24:45 GMT

Like most BizTalk Server developers I am addicted to the use of DebugView. This, in case you are one of the three BizTalk developers out there who are still not aware of it, is a free utility written by the SysInternals people. Microsoft bought SysInternals a while back, and the utility can now be downloaded from their web site at:

http://www.microsoft.com/technet/sysinternals/utilities/debugview.mspx

DebugView provides a viewer for traces created using the Win32 OutputDebugString() API. You can liberally instrument your .NET code using calls to System.Diagnostics.Debug.Write<xxx> methods, then open up DebugView and run you code. The output is nicely displayed in the DebugView console. For BizTalk developers, this is an indispensable tool because there is no way to single-step through XLANG/s expressions. The Orchestration debugger is built on top of the orchestration engine, and is not a 'true' Win32 debugger. It can only single-step through interactions between the orchestration and the orchestration engine which occur at a much coarser level of granularity than individual lines of XLANG/s code.

I've spent the last couple of weeks pretending to be an ASP.NET 2.0 developer. The code I was writing does a fair amount of re-direction. It supports a number of different log-in models based on the Forms Authentication model in ASP.NET 2.0, but integrated with a remote identity provider. The code is 'proof of concept' stuff, and supports different ‘candidate’ approaches, including the ability to switch to local authentication in order to demonstrate the code in scenarios where we have no remote connectivity. To begin with I (rather stupidly, in hindsight) was trying to develop using the new Web Site project type in Visual Studio. Oh dear. I needed to use SSL sessions to debug the code because we are using CardSpace. You can't do this under the local web server. I tried, without success, to debug my code under IIS, but found that, for some reason, I was unable to attach to IIS worker processes! Then, I really came unstuck when ASP.NET completely lost the ability to debug. It was creating pdb files OK, but one of them appeared to be corrupted somehow each time it was built (for example, no version number), and Visual Studio insisted that I had no symbols loaded, even though the Module window clearly showed that I did!

I gave up, and ported the code over to a Web Application project. I still have problems. Visual Studio currently reports a set of spurious build errors 'dynamically' while using the designer (these change dynamically over time even if I don't touch the keyboard - yes I mean it! I just sit there and stare at the screen for a minute or two and see the error messages change before my eyes), but then builds and publishes the application anyway. It has lost the ability to see Master Pages if I use generated paths with the '~' specifier. It has also become hopelessly confused in regard to a custom control that I am using - a problem, incidentally, for which there are hundreds of hits in Google, but only one conclusion - 'tough - it's a long-standing bug - you can't fix it and there's no known workaround - rebuild your project from scratch'. Apart from that, the Web Application works OK, and I can now debug my code smoothly under SSL sessions. A good lesson learned. I certainly won't use Web Site projects in future, and will try not to use ASP.NET unless I have to, until such time as Microsoft sees fit to stabilise the design-time ASP.NET tools.

[Update: I had another go at clearing the persistent "Unknown server tag" problem, which is the issue I reported above with the custom control. I tried (yet again) a whole variety of approaches, including registering the control via the web.config file, rather than using the @Register directive. I eventually hit on an approach which seemed to work. I recompiled the control using a strong name. I had tried this before, but when I GACd the assembly, I got a new error in which ASP.NET reported confusion at runtime over having two copies of the assembly (one in the ASP.NET TEMP folders and one in the GAC). So this time, having encountered the same problem, I simply removed the control from the GAC (duh). And suddenly I can use the control in the designer again! As far as I am aware, this is the only change I have made (of course, you have to update the assembly name in the @Register directive to the strong name). I note that Visual Studio automatically updates an assembly reference for the control in the web.config file. Of course, everything was originally working fine for some time with a simple-named version of the same control (I haven't changed the control code in any way), so I fully expect everything to break down again for no apparant reason].

[Update 2: Oh yes. As predicted. Within an hour or two the designer magically lost its ability to handle my strongly-named version of the control. How do ASP.NET developers cope with this day in, day out?]

The point of this, apart from letting off steam about the general flakiness of ASP.NET 2.0 under Visual Studio 2005 SP1 (it's horrible), is that while developing using the Web Site, I needed to use DebugView quite extensively. Even now, it is of some use due to the re-direction. I can trace my code in one page, even though this page redirects to another so that I can't see my trace output using the built-in tracing features of ASP.NET. DebugView shows me what I need to see, as long as I remember to place '#define DEBUG' statements in my 'CodeBehind' files.

I ran into a problem with DebugView which I have seen before. I've been trying to research it to come up with a definitive answer, but I don't know the underlying cause - just the symptoms. My problem is that my ASP.NET code runs in a different session to the desktop. This means that the output does not appear in DebugView unless I enable the 'Capture Global Win32' option. However, this option was not available in DebugView.

At this point, I should say that I am developing on Windows Server 2003 R2. I run Vista on my notebook. When I open the same version of DebugView on my Vista notebook, I can see the 'Capture Global Win32' option. However, I can't see it when I run DebugView on the Windows Server 2003 box.

I was tearing my hair out over this one, and it isn't the first time I have run into this problem. I spent some time trying to find an answer on Google. I didn't turn anything up, but somewhere I read something about remote connections. I had a thought. I powered up a Remote Desktop session to my own development box and then opened DebugView. Success! The 'Capture Global Win32' option appeared, and once selected, DebugView started happily displaying the output from my ASP.NET application.

This approach cannot be used on XP (or, I think, Vista) because, even with SP2, XP does not support concurrent remote desktop sessions. Apparently, Microsoft was going to lift this restriction at some point, but then decided not to. As I say, when running DebugView under Vista, I don't get the problem anyway.

I experimented further with DebugView by setting up a 'client' instance (surely this should be termed a 'server' instance') and then connecting to this on the same machine. In my first attempt, this appeared to work, but when I tried to reproduce the approach the following day, it failed. The Remote Desktop approach works every time though.

As I say, I don't currently have an explanation. I know, from Googling, that there are some interesting issues when using OutputDebugString regarding the permission set granted to an underlying Mutex. Maybe the behaviour of DebugView has something to do with this. Maybe the explanation is simpler. It may have something to do with having another registered debugger in the same Windows session. I'm not sure. What I do know is that remoting into your own desktop on Windows Server 2003 works very nicely indeed.

WF Rules and MS BRE - Comparing Performance

Charles Young — Tue, 14 Aug 2007 17:27:29 GMT

I’ve been asked a few times how the performance of WF (Windows Workflow Foundation) Rules compares with that of the Microsoft Business Rules Engine (MS BRE). Having done no testing, I could only guess at the answer.

I’ve now undertaken some initial performance testing to compare WF and MS BRE, and decided to publish the results. You can read my write-up of the results here.

http://geekswithblogs.net/cyoung/archive/2007/08/12/114597.aspx

BizTalk Server or WF for rules and tracking?

Charles Young — Thu, 19 Jul 2007 19:13:27 GMT

I got an email today requesting help in deciding the appropriate selection of rule processing technology for a workflow application. I’ve got requests like this before, so I’ve decided to post a reply publically.

http://geekswithblogs.net/cyoung/archive/2007/07/19/114061.aspx.

BizTalk Server 2006: Auto-Construction of Orchestration Variables Considered Harmful

Charles Young — Sun, 04 Feb 2007 20:27:00 GMT

Don't get me wrong. I think the world of BizTalk as a product. But, like any complex piece of software, there are one or two things that annoy me. I blogged a few weeks ago about the xpath() function, and the way BizTalk decides how to initialise an internal XmlSerializer instance based on the type of the variable to which you assign the xpath() result. If you type your variable inappropriately, your code breaks at run time with a problem that is difficult to diagnose.

Another orchestration issue which caused me some suffering a few days ago is the auto-construction of orchestration variables. If you create an orchestration variable of a type that has a default constructor, BizTalk will, by default, auto-construct your variable using that constructor. I've never really understood the logic of this design. First, it seems a little inconsistent. Some types don't have a default constructor, and hence cannot be auto-constructed by BizTalk. Second, the fact that auto-construction, where applicable, is set as the default behaviour is just downright dangerous.

Here is a real-world scenario. As always, you are up against a tight deadline to complete your work. Everything is going well, but you have been asked to add a small and simple piece of functionality to a particular orchestration which, hitherto, has been working properly. You drop a loop shape into the orchestration and set it up to enumerate some values provided by a custom class. Disaster. When you run the orchestration, strange behaviour is observed. An instance of the orchestration is activated, but then...nothing. The very first orchestration shape is an expression whose first line is a trace. This line never gets executed. Worse than that, the BizTalk process has all but locked up. The CPU is maxed out at 100% and it takes a couple of minutes of patient waiting to coax the host instance into stopping. You then find it virtually impossible to get rid of the running orchestration instance. In fact, there are two service instances involved. The orchestration’s parent (it is a 'called' orchestration) receives a message via a published web service, and you have to stop the isolated host's IIS app pool in order to terminate the EPM (messaging) service instance associated with the isolated host. If you try also to suspend or terminate the orchestration service instance, the action is marked as pending. If you restart the host instance in order to complete the pending action, the process just locks up again. Because you can't get rid of the active orchestration service instance, you can't redeploy a new version of your BizTalk code.

Progress slows to a crawl. In desperation you eventually resort to manually truncating various BizTalk message box tables in order to recover from the lock-up (not a recommended strategy, though very effective). You can't trace or debug your orchestration code because none of it appears to ever run. You remove the loop that caused the problem and redeploy, and still you code doesn't work. Hours start to pass while your blood pressure slowly rises, and your deadlines come and go. You try everything you can think of, including recreating your orchestration from scratch, testing it carefully at each stage. Everything is fine until you recreate that darned loop. Again, BizTalk locks up.

The best thing to do in these situations is to walk away from the problem for a while. With a little encouragement from the project manager, this is what I did. The functionality represented by the loop was important, but I could carry on making progress without it for a while. Progress is duly made, and then, a few hours later, walking down the street after the day's work is done, my mind turns back naturally to the problem...and bingo (as we say here in the UK)...a moment of clarity and enlightenment. I know in an instant what is wrong. Next morning, it takes me about a minute to locate and fix the bug.

My loop uses a custom class. I created a variable at the outer scope of the orchestration to hold an instance of this class. The default constructor invokes some code that contains a loop. I had managed to write that loop so that it never hit the 'break' statement. So obvious. So basic. As soon as BizTalk instantiated my orchestration, it ran the code to auto-construct my custom class. The construction code hit the never-ending loop and everything locked up.

In future, I intend to adopt a new BizTalk coding standard. I shall rigorously switch off auto-construction for all my orchestration variables, and include explicit code in expression shapes to construct objects. This kind of bug takes a minute or so to diagnose when using a language like C#. In BizTalk orchestrations, it becomes opaque and very difficult to find. Worse than that, BizTalk does not provide sufficient facilities for easy recovery from the worst effects of auto-construction. This is a feature too far.

BizTalk Server 2006: An xpath() Function problem

Charles Young — Tue, 12 Dec 2006 23:19:00 GMT

I spent a couple of hours today looking at the problem we encountered yesterday with the xpath() function in BizTalk. We were attempting to use an XPath to extract the value of a nested element and assign it to a string variable. The nested element (<Direction>) was in the global (anonymous) namespace, but was a child of an element in a named namespace. The code failed on XmlSerializer de-serialisation with an error saying "<Direction xmlns=''> not expected".

At first, I strongly suspected that the problem was due to the use of the anonymous namespace, especially as the element was defined in a schema that was then imported into the message schema. Both schemas had the 'Element FormDefault' attribute set to default (unqualified), and we had ended up with a horrible interleaved mess of elements, some of which were in defined namespaces, whilst others were in the global namespace. Always, always set Element FormDefault to 'Qualified' in your schemas, unless you have a really good reason not to. We are now in the process of changing all our schemas to eliminate the use of global namespaces.

After a while, I convinced myself that there was a problem with the .NET XmlSerializer, especially after reading a number of news group threads on the Internet. From what I read, it seemed that the XmlSerializer might have problems with "xmlns=''" declarations. To prove the point, I wrote a little code to reproduce the issue. Unfortunately, however, the XmlSerializer worked perfectly, whatever I did. It seems that it has no issues at all with "xmlns=''".

I then spent some time with Reflector, digging into the BizTalk code to see what it was doing, and eventually I discovered the problem...and it was all our fault!!

We had a line of code in the orchestration that attempted to use the BizTalk xpath() function to return the value held in the <Direction> element and assign it to a string variable. The only problem was that the XPath was addressing the <Direction> node, and not its contents. The XPath processor uses XML DOM internally, and expressions return XML nodes or node sets. In our case, the XPath we used was returning an XmlElement node from our XML message. We were trying to assign this to a string variable (doh)!

This very basic XPath error was made opaque by the fact that we got no compile-time error, but instead suffered an run-time exception thrown by the XmlSerializer class. You might reasonably expect that the exception would indicate some kind of type mismatch or cast failure. Instead, we got a general-purpose XmlSerializer exception statibg that the XML content was unexpected. The reason for this becomes clear when we looked a little deeper into what is happening when we call the xpath function. BizTalk generates C# code for the line that calls the xpath function. The generated code calls a method to which it passes the Type of the variable to which the result of the method will be assigned. This Type is used to initialise an instance of XmlSerializer used internally by BizTalk. In our case, we ended up with an instance of XmlSerializer that attempted to de-serialise an XmlElement (addressed by the XPath) as a string object. This, of course, didn't work, and the XmlSerializer returned an exception saying that the XmlElement 'was not expected'.

The XmlSerializer object is used to deserialise XML content obtained using XPathNavigator and XmlNodeReader. In a reversal of the principle of strong typing, BizTalk's XLang/s langauge effectively 'trusts' the developer to select the right type of variable to hold the results of the xpath() function. If the developer gets this wrong (as we did), the code blindly attempts to perform de-serialisation to the incorrect type, and fails at run time.

The best fix in our case was simply to extend the end of the XPath with '/text()'. This returns the text node contained in the <Direction> node (XML DOM sees the text node as a nested node). This nicely de-serialises to a string. Another option would be to retain the XPath as is, but assign the result to an XmlElement variable, and then extract the inner text. This is less direct, and may require an additional atomic scope because XmlElement is not serialisable.

So, the problem was nothing to do with the use of anonymous namespaces or the .NET XmlSerializer. It was a basic logical error in our own code that took ages to spot due to the exception that was raised. If you use the xpath function and get an "<xxxxx xmlns=''> not expected" exception, check your XPath to see what it is actually returning, and the variable you are assigning the results to.

BizTalk Server 2006: Using .NET Classes for Orchestration Message Types

Charles Young — Wed, 29 Nov 2006 17:34:00 GMT

I had some discussion today with Christof Claessens about the merits of using .NET classes as orchestration message types in BizTalk Server 2006. Here are some of the reasons I came up with, and also some of the issues.

http://geekswithblogs.net/cyoung/articles/99464.aspx

BizTalk 2006: Accessing flat file content in orchestration messages

Charles Young — Tue, 11 Jul 2006 14:27:00 GMT

I was recently asked if I would provide some insight into solving a BizTalk problem. It's some time since I have blogged on BizTalk in depth, and the question gave me an excuse to write something on my favourite integration tool. Advanced BizTalk developers will know all about this subject, but if your are fairly new to BizTalk, this might be of help.

In the given scenario, data stored in flat files in being delivered to the BizTalk message box via a pass-through pipeline. An orchestration subscribes to these messages and creates XML output. Each time a file is processed, the developer wants to 'dump' the contents of the flat file into an element in the output. Other information, including the file name, is also included in the XML output.

The question I was asked was 'how do you access the content of the flat file messge in an orchestration'. The answer is fairly straight-forward, but there are things to learn on the way...

Read on at http://geekswithblogs.net/cyoung/articles/84735.aspx