Part 2 of my "Reloaded-Series"!

In the orginal post here TCPBinding seemed to be the solution. But it is not the best and in some cases just not usable.

This only applies to large requests!

Sabrina and Philippe, both colleagues in the current project, discovered something far better and much more interesting...

What happnes here is DOS-attack-prevention by IIS! Fortunately this did not apply to us that much, sitting behind the DMZ and a far way from the outside world...

Solution: Change the allowed HTTP-request-lenghts... How is this done? One way (and not the worst)

In  the web.config of your WCF-service there are 2 sections to insert the important stuff. At least IIS 7.0 and above.

 <system.webServer>...
    <security>
      <requestFiltering>
        <requestLimits maxAllowedContentLength="209715200"/>
      </requestFiltering>
    </security>
  </system.webServer>

  <system.web>...
    <httpRuntime maxRequestLength="2097151" />
  </system.web>

One is legacy (IIS 6....) and the other more today. But both are needed for success.

The numbers are important. "maxRequestLength" is in kb, maxAllowedContentLength in byte. Both are set to very high values in this example. Be careful about that!

You can set some values in IIS but it is not bad practice, I think, to set it on svc-level only...

 

 

This error plagued me some time ago but then some change in the called service(i think they sped up processing...) was the solution.

But in my new project it happend again. We call a WebService hostet on WebLogic. No problem if the traffic is low and only few calls are made. But then calls increased and *boom*. Not again! 

It took us several days to tackle it, trying things like the "keepAlive"-customWCF-binding (see earlier post) and lots of config-changes on WebLogic and so on. Nothing seemed to help.

To make a long story short: BizTalk only sends 2 concurrent requests at a time, if not configured otherwise. This leads to some strange behaviour: In the Admin-Console it looks like all the calls actually left the BizTalk-machine. On the receiving side you can see that only 2 connections are actually used (e.g. via netstat). The other calls are kind of queued. If processing is not fast enough on the service-side either a timeout occurs or the dreaded "The request was aborted: The request was canceled.".

Solution (in this case at least...):

Just configure BizTalk to allow more than 2 concurrent calls. How is this done?

http://msdn.microsoft.com/en-us/library/ee290739(BTS.10).aspx

Include the snippet into your BizTalk-Config (BTSNTSvc64.exe.config and/or BTSNTSvc.exe.config depending on the system/hosts you are using. Be sure to choose the correct config!). Here the MS-example:

 <system.net>
    <connectionManagement>
      <add address="www.contoso.com" maxconnection="20" />
      <add address="*" maxconnection="10" />
    </connectionManagement>
  </system.net>
 

And take the advice very seriously that this might lead to flooding target-services... Be careful and set it to the least possible value, that keeps your system running and use the restrictions for the addresses !

 

A WCF Service we call from BizTalk using WCF BasicHTTP usually works fine but all of a sudden it started returning 401 errors for some calls while others continued working as expected so it could not have been a "real" 401. The difference was the size of the message. One parameter of the service is a rather complex object. In the cases we got a 401 it got quite big (containing a lot of customer-data), say 5 MB. So we turned on tracking. The messages we traced out where about 20MB. Not too big for WCF one should suppose...

A bit of research led us to increasing maxItemsInObjectGraph in the behaviours but that did not help.

The service we call is in the same network as we are and is a WCF service. So we tried changing from BasicHTTP to net.tcp and Bingo! Ok, we had to use CustomBinding in BizTalk to set all the Quotas, etc. but it worked in the end.

A strange error occured recently on our BizTalk. It suddenly showed up in the EventLog and the Orchestration got suspended. "Index was outside the bounds of the array" happened at the Receive shape connected to a Request-Response-Port where we called a WebService. It had worked properly a day before. The response message from the service we called was not very large and the data seemed ok. The error came from here: Void Write[our type]ToXml(System.Runtime.Serialization.XmlWriterDelegator, System.Object, System.Runtime.Serialization.XmlObjectSerializerWriteContext, System.Runtime.Serialization.CollectionDataContract). From within BTS/WCF Messaging. The error was reproducible. As it turned out that BizTalk was throttling (State 4 : Memory Pressure)! Obviously it was not able to create the array in memory. I never had this error before. So: lesson learned. :-) After restarting the HostInstances, everything worked fine again.

It's been a long time... I am very busy in my new project in Switzerland. Zürich is great! :-)

On one of our testing-environments the EventViewer of BTS Admin Console suddenly didn't show events anymore. All categories (Application, System, etc.) were just blank.

The EventViewer of Windows Server 2008 still worked but I prefer the one in BTS as it is much faster.

It took me a bit to find out what happend. Someone had connected the EventViewer to another computer. The caption still said "Classic Event Viewer (Local)" but it obviously was not connected to any computer. I connected it to the local computer again and voilà! Back to normal. :-).

I am writing this post using IE9. The editor is not rendered correctly. All icons appear at the very left of the screen. Beware of clicking any button in the editor itself or copy&paste! If a pop-up comes up you are trapped! You cannot close the popup and all your work is gone. I wrote this three times... In the endI was not able to post it saying "your post must have a body". Frustrating! So I had to download another browser to post this article...

I included a screenshot below.

But please don't get me wrong. I love IE9! It's just great. :-D

Two posts on one day? Is he on his holidays? Yes!

We had a lot of trouble with this error:

...Error Description: System.ServiceModel.CommunicationException: An error (The request was aborted: The request was canceled.) occurred while transmitting data over the HTTP channel....

You find a few posts about it in the www but the answers are somewhat conflicting and not very nice. No, I will not tell you the answer in this post, because we still do not really have it, but I want to show you how to perform one step that might lead you closer to it.

The error occurs under high load only in one of our environments. Of course the production environment.. Thank you, Mr. Murphy...

We are using a rather complicated Orchestration structure and make a lot of WCF-calls from BizTalk to WCF-Services. As a binding we use WCFBasicHttpBinding with SSL and Basic Authentication. Most of the posts in the www point to setting the KeepAlive-flag to false. MS-Suport asked us to it, too. But this is not a property present in the usual WCFBasicHttpBinding. So we had to switch to customBinding in WCF-Custom Transport. I was not able to find posts about how to do that and had to try out a few things until I was successful. To save you some time, here is what worked:

1. Import the custom-binding that was created when generating the service-reference (switching completely from DynamicPort to a static Port, but that is another story...)
2. Configuring the Transport I switched the BindingType in Binding to "customBinding"
3. Remove the "httpTransport" extension and add a "httpsTransport" extension, as you want to use SSL. You have to remove "httpTransport" first because otherwise "httpsTransport" will not show up in the extensions.
4. Move "httpsTransport" to the last place of the extensions, otherwise BizTalk/WCF does not like it...
5. In "httpsTransport" set "authenticationScheme" to "Basic"
6. Here you can set "keepAliveEnabled" to "false" and other important properties like "maxReceivedMessageSize", too!
7. In the "textMessageEncoding" node you have to set "messageVersion" from "Default" to "Soap11WSAddressingAugust2004" as you want to use basicHttp under the hood. Under "textMessageEncoding" you will find the "ReaderQuotas", too.
8. In the "Credentials"-tab set username and password for basic authentication, or use SSO

When trying for the first time with the newly created SendPort I got an error that communication with the service could not be established. The binding was changed to custom-binding so that was not too surprising...

To make things a little easier I exported a web.config in the "Import/Export"-tab in the WCF-Custom Transport Properties and adjusted the web.config of the service I called based on the exported web.config. Here we were fortunate enough that the Service is under our control and it is a WCF-service...

Now it works all fine. If it really helps us with error we will find out soon. To be continued...

*****

Update: All the work did not help with the error at all. It turned out it was the service on the other side producing the error. But I don't know what has been changed to get it working.

You created classes from your XML-Schemas (using XSD.exe for example) and want to use them in BizTalk. There may be several reasons for and against doing it. But I do not want to discuss that here. Just assume you want to do it. ;-)

After you deployed your BizTalk App and start sending the first messages to your ReceiveLocation with an XML Receive Pipeline of a MessageType that a schema and a class exist for. BizTalk is not able to decide whether  class or schema should be used and  gives you a nice error in the event log: 

There was a failure executing the receive pipeline: "Microsoft.BizTalk.DefaultPipelines.XMLReceive, .....

 Reason: Cannot locate document specification because multiple schemas matched the message type http://myschema...#Order.

 

When you have a look at the schemas in the BTS-Admin-Console you discover that you have two Schemas listed there with identical namespaces and root-names, resulting in identical MessageTypes.

To show BizTalk which Assembly it should use you can use the property "DocumentSpecNames" in the configuration of the XMLReceive Pipeline of your ReceiveLocation. Just type or paste the full strongname of the Assembly and the type that should be used, e.g. MySchemas.Schemas.Order, Schema.Order, Version=1.0.0.0, Culture=neutral, PublicKeyToken=88c5c240c6b0ed6f.

Please ignore the namespaces in my post. They are just for demonstration... ;-)

 

Pekka from Frends gave me the opportunity to have a look at the beta-version of their Helium 2.0.

For all of you, who don't know the tool:

Helium is a web-application that collects management-data from BizTalk which you usually have to tediously collect yourself, like performance-data (throttling, throughput (like completed Orchestrations/hour), other perfomance-counters) and data about the state of BTS-Applications and presents the data in clearly structured diagrams and overviews which (often) even allow drill-down. 

Installing Helium 2 was quite easy. It comes as an msi-file which creates the web-application on IIS. Aditionally a windows-service is deployt which acts as an agent for sending alert-e-mails and collecting data. What I missed during installation was a link to the created web-app at the end, but the link can be found under Program Files/Frends...

On the start-page Helium shows two sections:

• An overview about the BTS-Apps (Running?, suspended messages?)
• Basic perfomance-data

You can drill-down into the BTS-Apps further, to see ReceiveLocations, Orchestrations and SendPorts. And then a very nice feature can be activated: You can set a monitor to each of the ports and/or orchestrations and have an e-mail sent when a threshold of executions/day or hour is not met. I think this is a great idea. The following screeshot shows the configuration of this option.

Conclusion:

Helium is a useful monitoring  tool for BTS-operations that might save a lot of time for collecting data, writing a tool yourself or documentation for the operations-staff where to find the data.

Pros:

• Simple installation
• Most important data for BTS-operations in one place
• Monitor for alerts, if throughput is not met
• Nice Web-UI
• Reasonable price

Cons:

• Additional Perormance-counters cannot be added

Im am not sure when the final version is to be shipped, but you can see that on Frend's homepage soon, I guess...

A trial version is available here

A WCF Service I provided took a very simple data contract as parameter (containing one string and one int...) and had a very simple task to do. A .NET 3.5 client was created using the VS2008 feature "Add Service Reference". Everything worked as expected.

Then a slight change came in: The client was expected to run on machines with .NET 2.0 only. So we set the Target  Framework to .NET 2.0, removed the references to System.ServiceModel, System.Runtime.Serialization and the ServiceReference and created a new Reference to the Service using the old "Add Web Reference" . A matter of 2 minutes.

 When testing, the int value in the data contract arriving at the WCF Service suddenly was 0, instead of 38 as we expected.

What happened? When generating an old  Web Reference on a WCF data contract an additional boolean field for each value-type field is created called [Fieldname]Specified (e.g. AgeSpecified) which defaults to "false". WCF inspects these boolean fields to determine if a value was provided for the value-type field. If the "Specified"-field is "false", WCF translates that to using the default-value of the value-type field. For int this is 0.

So we had to insert  setting the "Specified"-field  for the int-value to "true" and everything was fine again. That was what we forgot after setting the Framework-version to 2.0...

 

Imagine a huge BizTalk-project. Loose coupling has been taken very seriously and a lot of this is done via the MessageBox and subscriptions. Very complex workflows are distributed over lots of Orchestrations. Commands in the messages tell the system which Orchestration to call next, which backend-system to call, etc. Nice... :-)

Then one day during developer-testing something strange happend. We started a process with Command_A. We inspected the progress of our Workflow-Armada in DebugView. It started out normally like "Start processing. Command_A". Then the next Orchestration started and it was the wrong one: "Start Processing. Command_B", but we expected it to be Command_Y! But it started getting better: After processing the wrong Command_B the Orchestration to cal the backend-system started and told us: "Invalid Command: Command_C". Our first reaction was: "What!?!?!". We spent a long time searching where the wrong commands were set, examining all Orchestration-paths and helpers. Nothing. It should have worked. We took another test-system. It worked as expected, but we were able to reproduce the strange behaviour on the main-system.

After a while the penny dropped: The commands were defined in an Enum. We thought we were fine, because we added numbers explicitly to the commands (like Command_A = 456). But someone had included new commands just in the middle of our existing commands without definfing numbers. So the index of the enum was not correct. We had to recompile the whole lot (enums are included in the IL of the classes using them) and it worked again. What a day... ;-)