Review: ODP.NET Developer’s Guide

Beginner complexity

In most industries beginners work on simple, short projects using only the most basic concepts that they have learned. The experts in those industries, meanwhile, work on things that are very large, complicated, difficult to manage, and require lots of time, energy and resources. The end result is that the experts produce results that are very complicated while beginners produce results that are very simple.

For example, take an architect. A beginning architect might design a simple house. You might look at the plan for the house and say, "it's basic, simple, easy to understand ... clearly the work of a beginner." An expert, meanwhile, designs a sky scraper, or an airport, or a sports stadium that boggles the mind in it's complexity. You would look at the plan for the sky scraper and not even begin to comprehend it -- you truly would recognize that it must be the work of a professional and you'd admire it and possibly be intimidated by its complexity.

Yet in software development it's the beginners who end up creating large, complicated, unmanageable projects that require lots of resources. Instead of using only the basic concepts, they usually completely disregard them! Looking at a beginner's code, it is often amazing at how much work and complexity they forced upon their project, and how much harder they worked than they needed to. Someone might look at a 1,000 line SQL statement with 5 levels of nested cursors that requires 2 1/2 hours to run and think "clearly, this is advanced, complicated stuff written by an expert! It requires a super-computer to run!"

An expert programmer, meanwhile, uses those basic, simple fundamentals to write very short, compact, clear code. That same 1,000 line SQL statement will end up being a simple 1 line SELECT. Looking at that code, one might say "that's it? Seems to simple ... runs in only a few seconds ... clearly, it was written by a beginner!"

Software is kind of backwards compared to lots of professions, isn't it?

The end of the Bill Gates era

People have been hypothesizing about this moment for years. The rumours became even more heightened with the hiring/purchase of Ray Ozzie. Bill Gates' non-appearance at the keynote for TechEd, was another hint. Well, the day has come. Bill Gates has announced that he is transitioning away from his daily duties at Microsoft to focus on his charitable work with the Bill & Melinda Gates Foundation, though he will remain chairman. Everyone saw it coming, but I personally didn't think the announcement would come this soon. The transition will take place over a two year period. Ray Ozzie, currently CTO, will take on Gates' title of Chief Software Architect. Craig Mundie, another CTO, will take on the title of Chief Research and Strategy Officer.

In honor of Stephen Covey's 7 Habits of Highly Effective People...

The Seven Habits of Highly Effective BizTalkers

Adapted from BizTalk 2004 Patterns and Practices from Alan Smith in Stockholm

1 One-Click Deployment

This topic has been covered in quite a few articles and blogs. Personally, I use the NAnt deployment script, developed by Scott Colestock. I sometimes create a “Quick NAnt Deploy” script, and add a task to the menu and use it when I am modifying a small part of the project. Even with a deployment and build script, it can sometimes take five minutes to run through everything, with a simpler script it can take a minute to deploy part of a project.

 

When working with modifications to an orchestration, it’s possible to re-GAC the orchestration dll, and then restart the BizTalk host that hosts the orchestration. The following batch script is an example:

 

REM GAC BizTalk dll   C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\gacutil /if bin\Development\TransformHub.dll     REM Restart BizTalk Server   net stop "BizTalk Service BizTalk Group : BizTalkServerApplication" net start "BizTalk Service BizTalk Group : BizTalkServerApplication"   PAUSE

 

2 One-Click Testing

I’ve seen a few posts and questions about automated testing for BizTalk applications, but have not seen any real implementations. I’ve started to look at this, and have been building automated tests for my current project.

 

A simple one-click test can be created by writing a batch file that will drop a number of test documents in the target folder for a file adapter. It’s easy to run this test after a build, or modification, and check that no orchestrations get suspended, and that the output messages are as they should be.

 

When working with other adapters, such as SOAP, SQL Server, or MQ Series, creating a test may require developing a test framework. I followed the testing guidelines on the Enterprise Integration Patterns site, and created the test framework as a C# project. This was able to create documents from XML templates, validate the output documents, and also interact with a database.

 

Stress testing is also important, especially when using the SQL adapter for database updates, as there can be problems with transaction deadlocks. For batch file driven testing, you can create a folder, and place 1,000 or so documents there, the batch copy them into a target folder. With a C# test framework you can use a for-next loop to create a large number of messages.

 

Before stress testing a BizTalk project, make sure you have set up all the BizTalk database cleanup and backup tasks in SQL Sever. (But you did that straight after the install, right ;-).) If this is not done, your message box, or tracking database will start taking over all your hard drive space.

 

3 Prototype Outside the Project

I’ve always recommended prototyping outside the project as a good way to solve problems. With BizTalk projects, I find it even more beneficial.

 

When working on a medium or large project, trying to implement a new idea or unfamiliar technique can often take far longer then expected, and results can be messy. Worse, it can disrupt other developers on the team, break the build, or leave the project in an unexecutable state.

 

Creating a small prototype project that focuses on a specific idea or technique has several advantages:

• The code/build/deploy/test/undeploy/fix cycle is much shorter
• The prototype can be focused on one specific problem
• The prototype is disposable, if it gets messy, throw it away and start again
• You can experiment with the techniques, trying things you would not try in a production project.
• When you have solved the problem, you can implement it quickly and cleanly in a larger project.
• You build a resource of prototype projects to share with your team.

 

I have probably created over a hundred small projects when I needed to get my head around new ideas. It’s good to be able to dig them up, and play around with them when I need to.

 

4 Follow the Patterns

The Enterprise Integration Patterns book and website provide a great resource for integration architects and developers. The patterns and ideas are not specific to BizTalk, but then integration projects are rarely specific to BizTalk. Quite a few of the patterns are provided out-of-the box with BizTalk, but a lot of the message processing patterns require implementation in an orchestration.

 

It’s a good learning experience to implement a simple aggregator, or resequencer as an orchestration. It’s also valuable to use “Enterprise Integration Pattern Language” within a project, often your dealing with a lot of other systems, outside the Microsoft world, and it helps if everyone knows what a splitter, or a wire-tap is.

 

5 Learn to Debug

I’ve spent quite a lot of time debugging orchestrations, and discovered a few techniques I use, as well as the standard tools.

Event Viewer

The event viewer is usually the first place to look when something goes bad in an orchestration. It’s quicker and easier than starting HAT, but does not provide too much information.

BizTalk posts errors, warnings, and information to the Application section of the Windows event log. The text of these events give you a good indication of the error that occurred, such as an XLANG/s error, or undefined reference error. Often this can be enough to figure out what you have done wrong. What it won’t tell you is where the error occurred in the orchestration.

 

HAT

HAT provides a view of the messages, and orchestration instances (Service instances) that are present in the message box, and tracking databases. As it’s specific to the BizTalk databases, and not the server where the orchestration was run on, it’s great for use in multi server environments.

 

HAT also provides the option to save tracked messages, and to view orchestrations in an orchestration debugger.

System.Diagnostics.Debug.WriteLine

Use the System.Diagnostics.Debug.WriteLine method in your orchestration code, then attach Visual Studio to the BizTalk service (Tools -> Debug Process -> BTSNTSve.exe, and check “Common language runtime”). You can then view the debug trace in the output window.

 

This is great for checking on the flow of an orchestration, and getting a dump of all the orchestration variables, and messages. You can dump a message by assigning it to an XmlDocument object, then dumping the OuterXml of the document.

 

testXmlDocument = InputMsg; System.Diagnostics.Debug.WriteLine (testXmlDocument.OuterXml)

 

Performance Monitor

PerfMon is good when you are running stress tests, or optimizing orchestrations. It can also give you a good insight into what is going on in the orchestration engine. Looking at the number persistence points for a complex orchestration can be a good way to target performance bottlenecks. I usually create a PerfMon tamplate with several of these counters, so I can quickly monitor the application.

 

The full list is here: ms-help://BTS_2004/SDK/htm/ebiz_prog_orch_nfup.htm

 

 

XLANG/s Orchestrations

Orchestrations completed, Orchestrations created, Orchestrations resident in memory, Orchestrations suspended

These are great when you are sending a batch or messages to an orchestration, and want to see the progress of the orchestration instances. HAT can give you this information, but with PerfMon, it’s displayed real time, and as a graph, (HAT can sometimes get slow when your box is under heavy load). You can also take a coffee break or lunch, and see how things went when you get back.

 

Persistence points, Persistence points/sec

The number of persistence points an orchestration makes as it executes is important, especially if you have large messages or orchestration variables. I usually check this, make changes to the orchestration, then test again. The number of points per second can give some indication as to the amount of data that is being persisted, as the message size gets larger, this will decrease.

 

BizTalk:Messaging

Documents received, Docuemnts processed, Docuemnts suspended

These counters can be used in much the same way as the orchestration counters, but provide statistics on messages.

 

6 View the Source

I posted about viewing the XLANG/s of an orchestration a while back, and I have used this technique quite a lot. I usually rename the file to C# (I like the color coding for comments), and chop out the attributes and XML stuff.

 

With a complex orchestration, the code is hidden in the shapes, and it’s hard to spot exactly what goes on inside. The code view is great when you want to optimize things, check for unused messages, or spot bugs that are caused by code in different shapes.

Another good use is when building a new orchestration based on a pattern from an existing one.

 

7 Join the Community

Last but not least, get connected with other developers.

Sharepoint: GUI Version of stsadm

Get it here http://www.microsoft.com/sharepoint/downloads/components/detail.asp?a=443.

Owen Allen talks about the role of SOA architects

As I have been working with BizTalk for the last half of year I have been trying to understand what the role of SOA should be in the large scope of enterprise IT operations. I have also been trying to understand the definition of the SOA architect. I have come accross a great article by Jason Bloomberg that I feel is is worth reading because it does a good job introducing SOA principles as they relate to the function of a SOA architect.

Adapted from Owen Allen (MSFT PacWest)

What is a SOA Architect?

The full article is here.

 

Jason describes SOA in a straightforward and familiar way,

"The power and flexibility that SOAs can potentially offer the enterprise is substantial. If an organization abstracts its IT infrastructure so that it presents its functionality in the form of coarse-grained services that offer clear business value, then consumers of those services (whether they are at the same company or one of that company's business partners) can access those services independent of the underlying technology issues that support them."

 

Jason's Principles of SOA

• The business drives the services, and the services drive the technology
• Business agility is a fundamental business requirement
• A successful SOA is always in flux

Jason continues by comparing a SOA architecture approach to Model-Driven Architecture, and describes how a SOA Architect must take into account the Use Cases for a service when designing their SOA

David Hurtado talks about code inside orchestration vs. components

Adapted from David Hurtado's Integration Traces

Programming inside orchestrations: Where should I put the code?

Abstract: A common question from my customers: ‘how much code should be in the orchestration, and how much code should be leveraged into .NET components?’

In the BPEL standard, a business workflow is used only to coordinate the execution of components (web services). With XLANG, you can add .NET code, so the workflow [sip] will have not only the orchestration logic, but also the implementation of that logic.
[sip] But the fact that you can code inside of the orchestration does not mean that you should. Why? For one thing, it’s difficult to maintain and difficult to debug. From my experience, the guideline is to put as little code inside the orchestration as possible, with the following rules:

The orchestration should contain:

• Control flow. That is, shapes and messages.
• Code needed to make the control flow run. Loop variables, initializations, helper calls, etc. --> if it’s simple.

The code leveraged to components should be:

• Code dealing with technical aspects. Resources, Xml/XPath handling, conversion algorithms, anything that requires error handling, etc,
• Control flow code, if complex, and/or requires technical stuff.
  

My justification is that leveraging code to components has the following advantages:

• Easy to debug.
• Easy to maintain.
• Can have error handling.
• Can use any CLR feature that you want.
• Use the .NET language of your choice :-)

Hopefully there will be some Cabana sessions at TechEd where we can talk about integration patterns using BizTalk 2004.

What do you guys think?  

ODP.NET and Oracle Developer Tools for Visual Studio .NET

I am working on a data integration project right now where I have a need to load data from a Siebel instance running on Oracle 8i into a typed ADO.NET dataset. I then load another typed dataset from SQL server and compare the two. Foreach record in the Oracle dataset, if there exists a matching record in the SQL dataset I see if I need to update that record or if the record does not exist in the SQL dataset, I add it. (More details on design and implementation to follow as I implement this solution). 

Given the need to connect to an Oracle instance and load a typed dataset, I began researching providers and tools. I am going to read the data out of Oracle using a stored procedure that returns a refCursor. I began using Oracle's ODP.NET provider which is glorified all over the Internet as the best Oracle provider for .NET. I have only written one other app that connects up to Oracle and I used Microsofts OLEDB provider for Oracle and a DataReader which worked just fine for that project but this result set has the potential to return many 10's of thousands of records and so I wanted to work disconnected. The typed dataset will be nice because I can set a key (on multiple columns) and leverage the FindByFoo() method that is generated for me.

So I downloaded and installed ODP.NET and referenced Oracle.DataAccess.dll in my project and I was off. Very nice object model... ODP.NET gave me the following objects which will do the trick [OracleConnection, OracleCommand, OracleDataAdapter]. I found the help that came with the download to be excellent. It is rich with C# code examples.

Now came the hard part... how can I generate a strongly typed dataset from a stored procedure inside of an Oracle package? I looked into the Oracle Developer Tools for Visual Studio .NET beta 2 which is pretty cool. It gives me a neat little Oracl Explorer which is pretty similar to the Server Explorer which I can use to write, debug and run PLSQL right in Visual Studio! That is nice, however, you cannot just drag and drop an object from the database schema in the Oracle explorer onto the DataSet design surface and have the schema autogenerate :( which is annoying. I had to write the schema by hand.

I will keep posting details on this project as they evolve but for now I reccomend ODP.NET and  the Oracle Developer Tools for Visual Studio .NET. I cant wait for TechEd when I can ask the ADO.NET guys about Oracle support in Whidbey and since Oracle is a new member of the Visual Studio Industry Partner Program, maybe they will have some reps there that I can ask about the schema generation piece.

 

Reporting Services / SPS WebParts

Mixing Reporting Services and SharePoint WebParts makes possible take advantages of Reporting Services functionality in our SharePoint applications. In this fantastic article  Bryant Likes explain how to develop WebParts that are used to display Reporting Services reports. The first version of RsWebParts can be downloaded here.

Adding security groups to Reporting services security doesn't take effect until IIS is restarted

If you grant Reporting services permissions (via http://servername/Reports  | folder properties | security) to Windows security groups, you may find out that people belonging to those groups don't immediately get the expected access rights.

For performance reasons IIS caches authorization tokens. For more details on how to configure it, see the following KB article:

http://support.microsoft.com/kb/q152526/

IIS reporting using Log Parser and SQL Reporting Services

ASP.NET 2.0 and SP WebParts

Will ASP.NET 2.0's webparts fit in the current version of SharePoint? Mike Fitzmaurice has written a good post about what you should take into account when you will be developing web parts.

What a good SharePoint devloper should know

http://timheuer.com/blog/archive/2005/02/24/1691.aspx

10 XML Specifications Unfortuneately Rejected by the W3C

10. WS-IrishSpring: for scented, more pleasing SOAP packets

9. WS-UPS: for sending SOAP packets in real envelopes

8. WS-USPS: for sending SOAP packets that dont need to get there

7. WS-PrisonShower: for picking up the dropped SOAP packets

6. X-Wife: protocol for monetary transfer

5. WS-Insecurity: dating protocol for web services programmers

4. WS-Monopoly: protocol used to keep antitrust penalties to manageable levels

3. NICKLE: for encoding smaller binary attachments

2. SFFCI: Syndication Format for Complete Idiots

1. WS-XXX: bringing a business model to XML, e.g.

<xxx:image xmlns:xxx="uri://hustler.com/2003/oohlala">
  <xxx:setting>the storeroomxxx:setting>
  <xxx:model gender="female" tattoo="skull" />
  <xxx:model gender="male" moustache="true" />
  <xxx:pose>saucyxxx:pose>
xxx:image>

:: Adapted from the Applied XML Developer's Conference West 2003 ::

:: http://www.sellsbrothers.com/fun/ ::