SOA

Microsoft ESB Guidance for BizTalk Server 2006 R2 is live!

Kent Brown — Fri, 09 Nov 2007 16:06:04 GMT

One of the best attended sessions at the SOA Conference in Redmond last week was actually the last session on the last day when usually most people would be headed towards the airport. It was on the ESB Guidance that has been in the works for several months. Marty Wasznicky demo'd the final release bits and discussed the most recent refinements and changes. He was basically done at that point, just waiting for the final i's to be dotted and t's to be crossed on the deployment package and for it all to be put up live on MSDN. I've been hitting refresh on my browser all week trying to get the latest for a demo we are working on. Finally it's here: http://msdn2.microsoft.com/en-us/library/bb931189.aspx!

I won't go into detail here on what the ESB Guidance is since its all in the documentation, but in a nutshell it is a set of free components that work with BizTalk and enable Enterprise Service Bus (ESB) scenarios. This is huge because ESB is currently all the rage in SOA-land. Time and again we see BizTalk in head-to-head competition with the likes of IBM/Tibco/BEA to be the ESB of a large SOA initiative at a company. BizTalk was SOA-before-SOA-was-cool, but it does lack out of the box a handful of the features currently expected in an ESB. This ESB Guidance provides the needed components (mostly implemented as pipeline components) to supplement BizTalk so it can legitamately claim to be an ESB. The architecture and components here were harvested from large customer engagements where BizTalk is successfully being used as the ESB.

Congrats to Marty Wasznicky for his vision and relentless hard work to make this a reality.

Is BizTalk Dead?

Kent Brown — Tue, 06 Nov 2007 19:20:36 GMT

Ok, so I have a theme going and thought I might as well run with it. I promise this is the last "Is x Dead?" post.

Of course BizTalk isn't dead. But it is going to change in the next couple of years. What I am talking about here is "Oslo", the recently announced, next-generation distributed computing vision from Microsoft.

Oslo takes SOA to the Internet

I was out at the SOA conference last week where Microsoft first publicly shared the vision that they are code-naming Oslo. There is a great story here and I can tell you from meeting with people on the product team that this is real - there are large numbers of people in the Connected Systems division at Microsoft already working on various products to support this vision. However, it will take some time, roughly two years, before significant portions of this will be ready for release. In the meantime, it is important as always to understand where things are going in order to make good decisions with the technology available today. The following is my current impression of the vision, the components, and the impact of Oslo on BizTalk Server.

The Vision of Oslo
Oslo is first of all a vision about the future of distributed application development and hosting. If Windows DNA was about realizing the vision of distributed computing through Microsoft technology, and .NET Connected Systems was about extending this vision to the entire enterprise by embracing interoperability standards, Oslo is about extending the vision one more time out to the breadth of the internet.

This vision had been previously announced at the MIX conference and termed "Software + Services". Software + Services, or S+S, is a subtle but meaningful variation on the currently popular Service-Oriented Architecture (SOA) and Software as a Service (SaaS) themes. The best way to explain it is to compare and contrast it with the current buzzwords.

Whereas SOA is largely used to describe the architecture within the enterprise, S+S creates a world in which services inside and outside the enterprise are tied together seamlessly to build a new breed of applications - "Composite Applications". While the Enterprise Service Bus (ESB) is currently all the rage, Microsoft wants to take it up a step and build the Internet Service Bus (ISB).

And while SOA is primarily targeted toward Corporate IT as an enabler for controlling quality and consistency within an organization, S+S actually encourages and facilitates the building of "opportunistic application development" by power users. These users will be more tech-savvy than ever before (having grown up with video games and blogs and iPods) and will not want to wait on the Corporate IT group to create the applications they need to be more productive. Think of the wild and crazy days of the PC revolution. But the advantage is that it will be easier for Corporate IT to adopt these renegade applications and officially support them when they gain critical mass because they will be built in a service-oriented fashion.

Software as a Service (SaaS) is best exemplified by SalesForce.com, which is an immensely successful CRM system that is completely hosted externally and consumed by subscribing companies. However, SaaS so far has only been successful when an entire application is hosted externally. The vision of S+S is that services will be hosted externally and applications within the enterprise will be built consuming services from within and without the enterprise security boundaries. The crux of this an availability of services built on interoperable web service standards, tools for building composite applications, and a federated security model that allows enterprise applications to trust externally hosted services.

The Components of Oslo

1. Continued emphasis on and evolution of core Service-Oriented programming technologies (WCF, WF). This will become ".NET Framework 4.0".

2. A first class Repository for storing, discovering, and governing services.

3. Simplified developer experience for building distributed applications through better modeling tools. This will of course be realized in a new version of Visual Studio.

4. Federated security model (InfoCard).

5. Leveraging of Groove techniques for getting around firewall barriers to collaboration between services.

6. Evolution of BizTalk -

o Adapters migrating out from BizTalk-specific realm to general .NET availability. This has already begun through the WCF Adapter framework and the WCF Adapter Pack.

o Other BizTalk tools, such as the schema editor and mapper become more mainstream as part of the .NET Framework.

o Orchestration will be done using WF as the XLANG/S engine is deprecated (this has been public knowledge for over a year)

o A new in-memory pub/sub engine will make persistence/durability of messages optional, making BizTalk more suitable for low-latency applications.

o BizTalk as a server product may be primarily about the hosting model and look something like the old AppCenter.

7. Potential for Microsoft (and others) to host useful services "in the cloud" that will be subscribed to by enterprise clients to reuse in their composite applications. There is a CTP version of some utility services already hosted at http://biztalk.net/Default.aspx.

What it means for the future of BizTalk

Eventually BizTalk as we know it will be dead in the sense that it will be deprecated and replaced by a shiny new technology. But I see it just like the change from COM to .NET. COM as a technology was dead, while the core concepts of component-based programming and interoperability between components lived on bigger and better than ever in .NET. And in hindsight, COM was a quirky technology and most of us are glad to be freed from GUIDs and the registry and vTables, etc.

In the same way, I expect that the architectural principals that BizTalk made real (interoperability, pub/sub, legacy integration, web service orchestration, long running transactions, massive scalability, model-driven development, etc.) will live on, in fact be magnified, in Oslo. But some of the specific implementation details that we may have grown sentimentally attached to (the MessageBox, XLANG/S, etc), will go away. And while we'll moan and fret as this transition happens, we'll eventually embrace the new and be happy we did.

In fact, Oslo represents an expansion of the role BizTalk plays in the entire Microsoft platform. While today BizTalk development is a niche are that a relatively small number of developers have braved, in Oslo, the tools will become more mainstream.

In the meantime, while we wait for the next version of BizTalk and the other tools that will make up Oslo, we have to make decisions about the applications we need to build today. This will be the topic of many blogs and white papers to come (in fact I have a white paper in the works regarding choosing between BizTalk and WF). For now I will summarize my view like this:

Oslo confirms Microsoft's commitment to Service-Oriented Architecture and development. So keep moving in that direction. Embrace WCF. Use WF where appropriate. And use BizTalk where appropriate.
You have to build your applications today on the available technology. Oslo is really too far away to significantly impact your development plans today. If you need what BizTalk Server provides, you should use it. Maybe when a Beta 2 or RC version is out with firm RTM dates, then you might want to delay a project to use the new technology.
Microsoft cannot make promises about upgrade paths at this point in the product cycle, but I feel there is no way they will fail to provide a way to upgrade orchestrations built on BizTalk 2006 R2 to run in the next WF-based orchestration engine. First of all, WF's roots are in XLANG/S so it isn't that hard to translate most features/constructs. Secondly, BizTalk is just too important and has too many mission-critical enterprise applications running on it for Microsoft to leave it hanging.
Finally, although the current BizTalk engine will be deprecated, it will still be available and supported for a long time to come.

Is BizTalk Dead?

Kent Brown — Tue, 06 Nov 2007 19:12:56 GMT

Ok, so I have a theme going and thought I might as well run with it. I promise this is the last "Is x Dead?" post.

Oslo takes SOA to the Internet

The Components of Oslo

1. Continued emphasis on and evolution of core Service-Oriented programming technologies (WCF, WF). This will become ".NET Framework 4.0".

2. A first class Repository for storing, discovering, and governing services.

3. Simplified developer experience for building distributed applications through better modeling tools. This will of course be realized in a new version of Visual Studio.

4. Federated security model (InfoCard).

5. Leveraging of Groove techniques for getting around firewall barriers to collaboration between services.

6. Evolution of BizTalk -

o Adapters migrating out from BizTalk-specific realm to general .NET availability. This has already begun through the WCF Adapter framework and the WCF Adapter Pack.

o Other BizTalk tools, such as the schema editor and mapper become more mainstream as part of the .NET Framework.

o Orchestration will be done using WF as the XLANG/S engine is deprecated (this has been public knowledge for over a year)

o A new in-memory pub/sub engine will make persistence/durability of messages optional, making BizTalk more suitable for low-latency applications.

o BizTalk as a server product may be primarily about the hosting model and look something like the old AppCenter.

What it means for the future of BizTalk

Oslo confirms Microsoft's commitment to Service-Oriented Architecture and development. So keep moving in that direction. Embrace WCF. Use WF was appropriate. And use BizTalk where appropriate.
You have to build your applications today on the available technology. Oslo is really too far away to significantly impact your development plans today. If you need what BizTalk Server provides, you should use it. Maybe when a Beta 2 or RC version is out with firm RTM dates, then you might want to delay a project to use the new technology.
Microsoft cannot make promises about upgrade paths at this point in the product cycle, but I feel there is no way they will fail to provide a way to upgrade orchestrations built on BizTalk 2006 R2 to run in the next WF-based orchestration engine. First of all, WF's roots are in XLANG/S so it isn't that hard to translate most features/constructs. Secondly, BizTalk is just too important and has too many mission-critical enterprise applications running on it for Microsoft to leave it hanging.
Finally, although the current BizTalk engine will be deprecated, it will still be available and supported for a long time to come.

Contract First and BizTalk

Kent Brown — Wed, 15 Mar 2006 11:43:00 GMT

Not only is BizTalk sort of “Contract First” out of the box, there is now a tool for BizTalk to actually give you the same experience the thinktecture WSCF tool gives you(http://www.thinktecture.com/Resources/Software/WSContractFirst/default.html).

The new tool is written by Molnar Tibor and is called BiztalkWscfClient: http://geekswithblogs.net/mtex/. It currently supports BizTalk 2004 and VS.NET 2004, but he is working to upgrade it for BizTalk 2006 and VS.NET 2005 very soon.

This is very cool! I can't wait to get the latest version.

Winning developers over to the BizTalk way

Kent Brown — Wed, 15 Mar 2006 09:58:00 GMT

A few of us who evangelize BizTalk have noticed that it is sometimes tough to get developers excited about BizTalk. In general, the BizTalk class at a developer conference or event, e.g. VSLive, will not be the biggest draw. I think there are several reasons why this is:

General lack of knowledge and understanding of exactly what BizTalk does.
The name probably doesn't convey what it does very well. Microsoft Enterprise Integration Server might have been a better name (but its too close to Host Integration Server I guess).
The earlier versions of BizTalk were tough to work with.
A lot of developers simply aren't doing integration type work - BizTalk won't help you build web pages or Windows apps.
If people are trying to do SOA, ASP.NET makes web services so easy that a lot of people don't see the need to look further.
Developers like code. They may be threatened by the “drag and drop“ impression that you get from BizTalk at first. Where's the code?
Related to #6, a lot of people take pride in the gory multi-threaded, low level plumbing code they write and aren't eager to hand that over to BizTalk.
If you do try to get into BizTalk it can be overwhelming at first how much you have to learn.
The impression is that there aren't that many BizTalk projects out there so the career opportunities aren't so great.
Somehow the impression is that BizTalk is not as “cool“ as say Indigo or something like that.

So besides helping customers understand the value of BizTalk, one of my missions it to help developers get it. I've recently given talks at the NYC Code Camp, and at a local NJ user group, that I think went over very well. My angle is to help developers see that BizTalk a) is very much .NET, b) adds value in SOA implementations, c) is currently in very high demand.

Here are some of my reasons why you should look at BizTalk as a technology to get into if you are a developer:

BizTalk forces you to think in “Contract First“ terms, something most Web Services developers are only recently coming around to.
BizTalk is probably the most “SOA“ technology that Microsoft ships. Even after WCF ships, it still might be a dead heat.
While BizTalk is highly “SOA“, meaning it supports the latest XML and Web Service standards, it is also very pragmatic in that it hooks in nicely with legacy technologies that aren't yet up to the latest standards.
BizTalk 2004, and now BizTalk 2006 are vastly improved over the 2000/2002 versions.
BizTalk 2004/2006 development is done right in Visual Studio. There is actually quite a bit of .NET code in the typical BizTalk project.
Letting BizTalk's well-tested code handle the gory plumbing frees you up to focus on design and architecture and ship faster.
Interest from customers to do BizTalk projects is actually rising dramatically in the last few months. Right now there is more work than qualified people. The career opportunities are excellent.
It takes a fairly senior .NET developer to make a good BizTalk developer. I personally see it as a bar that the junior/intermediate developer should strive for to demonstrate they are ready to be called “senior“, “architect“, etc.
Because BizTalk is tough to learn at first, there is a high barrier to entry - it's one of the ways to differentiate yourself at a time when development skills are being commoditized by off-shoring.
The suite of individual features BizTalk provides (scalability, fault tolerance, Orchestration, Rules Engine, Business Activity Monitoring, Enterprise Single Sign-on, etc.) combine to make an overwhelming whole that is very compelling for using BizTalk to host true “Services“.

NYC Code Camp

Kent Brown — Thu, 23 Feb 2006 08:36:00 GMT

I am presenting at my first Code Camp this Saturday in NYC. The matrix of sessions is here: http://nyc.codecamp.us/Matrix.htm

The registration is full, but you can register for the wait list here: http://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032289984&Culture=en-US

My talk is on Orchestrating Web Services with BizTalk Server 2006. I've found from my .NET user group in NJ and other venues that its often tough to get the average developer enthused about BizTalk. I'm not sure why it is. Some may be overwhelmed by the fact that it is so different. Some may feel threatened that it de-emphasizes custom code. I personally see it as the culmination of a lot of architectural principles I've come to hold over the years. My angle at Code Camp is to leverage developers' interest in Web Services and SOA by showing the value BizTalk adds in “choreographing” web services and how that helps realize the SOA dream.

Great introductory article on WCF

Kent Brown — Mon, 06 Feb 2006 12:25:00 GMT

I'm a little late to the WCF bandwagon. In a past job as instructor and architectural consultant, I used to be the Beta Hound, sniffing out the latest cool technology coming out of Microsoft, downloading it, “playing” with it, then writing or teaching about it. After a few years of that, especially with the release of .NET, I missed being in the trenches and felt the need to get back to writing some production code. So I got involved in designing and building into some large, multi-year .NET projects to dig out that true knowledge that only comes from using a technology on a large, complicated real-life application.

I find that when I am immersed in writing production code to meet deadlines, I slip out of Beta Hound mode into Old Dog mode (can't teach him new tricks). It's hard to focus on vaporware, no matter how cool and imminent it is, if it isn't going to help me deliver on time.

I am now back in a position that affords me a little more time for forward thinking and tinkering with new technology. With my main interests being distributed computing, integration, and interoperability, Indigo is the coolest thing to happen in a while and its one of the main technologies I am crunching to catch up on. Especially since it isn't far from release. Although a huge step forward, I am finding WCF fairly easy to learn as it builds on or has similarities with all my favorite technologies - .NET Remoting, MSMQ, BizTalk Server, ASP.NET web services, etc. In fact, the goal is to unify and simplify all these things into a unified whole, and so far I like what WCF does.

All that to say that the MSDN latest article by Aaron Skonnard is an excellent place to start learning WCF: http://msdn.microsoft.com/msdnmag/issues/06/02/WindowsCommunicationFoundation/default.aspx.

NYC Connected Systems User Group - Inaugural Meeting

Kent Brown — Wed, 01 Feb 2006 13:34:00 GMT

Things are coming together for the new NYC Connected Systems user group (NYCCSUG).

We now have:

A web site: www.nyccsug.org.
A meeting location: The Microsoft NYC office.
An Inaugural Meeting planned:

6-8:30 PM on Thursday Feb 23rd
Speakers:

I will open up the meeting to introduce the group, discuss the purpose and logistics.
Ian Murphy, a Business Process and Integration (BPI) Technical Specialist with Microsoft to present a BizTalk 2006 Overview.

We are going to be an INETA group which hooks us into their great Speakers Bureau.
We have a Microsoft sponsor in Redmond who is going to help arrange for speakers from the product teams.
I have started contacting other big-name presenters and it seems being in NYC makes it a little easier to attract them.

Please go register and tell your friends about the group.

NYC Connected Systems User Group

Kent Brown — Thu, 05 Jan 2006 10:50:00 GMT

For a while now several us have thought “there oughta be an Enterprise Integration-oriented user group in the NY/NJ area“. Luckily we've bumped into one another and “I oughta” has turned into “we're gonna“. The “we“ includes myself (leader of the NJ DotNet Architecture and Development User Group) and Andrew Brust, Stephen Forte, and Bill Zack, the moderators of the NY DotNet Developers Group, so we're not just whistling Dixie.

We've got strong support from Microsoft here in NYNJ and in Redmond. This will mean a meeting space at the NYC Microsoft office, occasionally speakers from the product teams in Redmond, and hopefully lots of SWAG. We are modelling the group after the Northwest Connected Systems User Group, which was started a few months ago in Seattle.

The topics of discussion will include platform-agnostic topics such as architecture and design principles, patterns and best practices, etc. Along the lines of the Enterprise Integration Patterns book. The focus will be broad enough to include SOA, EAI, B2B, and BPM.

When we get down to code and tools the focus will be on the Microsoft platform covering BizTalk Server, Windows Workflow Foundation, Windows Communication Framework, WSE, Host Integration Server, Commerce Server, in about that order of emphasis, depending on the interest of the group membership of course.

We are pushing to hold the first meeting by the end of February. Drop me a line if you are interested in participating in the group.

A season for every trick under heaven

Kent Brown — Wed, 28 Dec 2005 13:54:00 GMT

When pontificating on subjects like SOA, one has usually has to decide which side of the fence to stand on:

Unqualified, wild-eyed, bandwagon-riding enthusiasm. This usually involves pronouncements that the new paradigm will make computing infinitely easier and more fun, that it should universally and immediately be adopted for all new development and most if not all legacy systems should immediately be re-written because the new paradigm renders them hopelessly obsolete. In addition any developer who cares about his career had better start learning and using the new paradigm, and any tools that do not facilitate the new paradigm should immediately be discarded, while any tools that do support it are worth it, no matter how much they cost because of the ensuing productivity, efficiency, 'robustness“, flexibility, and general bliss.
Cool sophisticated skepticism. I love David Ing's post on this: “Unlike the word 'Objects', which found it's mindshare reality place in language constructs and the long winding road of design concepts through modeling languages ... 'Services' is a bit of an unbounded chimera at the moment. I can't think of a single architectural concept that is left behind. Reuse, Integration, Business Process Management, Macro-componentisation, Distributed Systems, Asynchronous Messaging all join the clowns and elephants of the SOA Circus Parade as it rumbles into town. ...This is a fantastic industry to work in as we get to watch a lot of these parades; they come every couple of years. The downside is that we often end up with elephant sh** on the street that needs to be cleared up by someone (certainly never the clowns in the parade, they're long gone...). “ LOL.

The problem is the tendency for people to take a specific paradigm for solving a specific set of problems and to generalize and extrapolate it out to the full universe of computing problems. I try to take a different approach:

There is a time for every paradigm,
       and a season for every trick under heaven:
a time to in-line and a time to abstract ,
       a time to inherit and a time to aggregate,
a time to build and a time to buy,
       a time to refactor and a time to patch,
a time to design and a time to ship,
       a place for Remoting and a place for SOAP,
a time for indirection and a time to hard code,
       a time to use a pattern and a time to refrain,
a time to accommodate feature requests and a time to push back,
       a time to push hard to met a deadline and a time to give up,
a time to train and a time to fire,
       a time to scale and a time to simplify,
a time to architect and a time to hack,
       a time to document and a time to run to the next project.

(sung to the tune of Turn, Turn, Turn, by the Byrds).

To me, architecting a system is finding the right solution for the given set of constraints. Those constraints involve the technical and functional requirements of the system being built, but also include a whole host of other business and cultural realities of the organization for which the system is being designed. This is so obvious that I don't want to expound on it, but surprisingly I hear and read so much along the line of “best practices” that does not make this clear. For example, in a web app, should you use session variables or should you keep things completely “stateless“? It depends. Are you are building Amazon.com, or an intranet application? Will you ever scale beyond a single web server? How much simpler or faster will your app be if you use some session state? But I often hear people state categorically that they always make web apps completely stateless because “you never know when it might need to scale“. There is of course some truth to that, but sometimes you do know it will never need to scale.

While a big proponent of SOA (I actually invented SOA, but that is for a future post ;-) ), I recently architected and built a large system that used .NET Remoting extensively - very little XML and not a web service in sight at a time when the conventional wisdom is “don't use Remoting, its not the future of the platform“. Why? Because while it was an extremely large and complex system, there were no requirements to integrate with other platforms, and the performance and simplicity of .NET Remoting outweighed any advantage of being ready for interoperability in case the need arises in the future. Shortsighted? Maybe, but in my opinion the right choice given the requirements, time line, budget, team expertise, etc. Besides, SOA is Just a Facade (another post to come soon), so we can always create an SOA interface to this system when needed.

On another project, the salesman at my previous consulting company had sold a project as using BizTalk Server 2004, a technology we were very eager to use. However, after some initial proof of concept work in BizTalk, I was asking some questions about the development staff at the client company. It turned out they had mostly SQL Server experience, which a couple of developers who had dabbled in VB6. No .NET experience. No XML experience. No BizTalk experience. What the heck were they going to do with the system after we left? It turned out the need was easily handled by some slightly complex DTS packages. Not quite as “elegant” as using BizTalk, but a much better fit for the situation.

It's tougher to take this approach to architecture and design because you have to be more aware of the various ways of solving a given problem, and the strengths and weaknesses of each solution. You have to really dig into all the requirements, including business and cultural, and you have to make value judgments. You can't just repeat the current industry mantra. You have to be willing to prescribe what fits the situation, not just what is cool and exciting for you as a technologist. It's more work, but its what we owe to those we serve. I don't want a mechanic who always changes the break pads at a certain mileage whether they need it or not. I don't want an architect who has designed skyscrapers to blindly use the same techniques on the new addition to my house.

Hopefully I don't sound like I am on the skeptical side of the fence when it comes to SOA. My job is to manage and grow a consulting practice around Enterprise Integration. I see SOA as an important paradigm with very real benefits that should be embraced, and I think it is actually underutilized in the industry. I just think we have to be careful not to oversell it because then we lose credibility.