Geeks With Blogs

News View Michael Stephenson's profile on BizTalk Blog Doc View Michael Stephenson's profile on LinkedIn
Michael Stephenson keeping your feet on premise while your heads in the cloud

A few days ago I read an article by Richard Seroter comparing the different cloud storage options and comparing Windows Azure and Amazon S3. I commented on his blog about how Id like to see companies use these more and more for B2B data exchange when you have a batch file rather than the traditional solutions using FTP and the painful infrastructure piece that often goes with this kind of project. The normal challenges include:

 

  • Who hosts the FTP service or do we both
  • What kind of security do we use
  • Are their any special infrastructure requirements eg leased line

 

FTP based solutions are often a pain unless the client has a well setup and established service and then there is the assumption that your B2B partner also has a similar capability in place.

From the comment on Richards blog post and some thoughts about requirements we may have at one of my clients I began thinking about this as an option to seriously consider. I also wondered what there was available in the Amazon space as it's a couple of months since I have looked at the Amazon cloud stack.

Following a quick search I came across the NSoftware adapters which had a ready made BizTalk adapter for Amazon S3

http://www.nsoftware.com/products/biztalk/adapters/s3.aspx

From here I thought it would be nice to have a walk through setting up a simple B2B scenario involving Amazon S3 and BizTalk.

 

Walk-Through

In this scenario we will have a business partner (Contoso) who will send us a file containing a batch of orders. The business partner will upload the file to a bucket in AWS S3 with the file name ending with a .input extension.

BizTalk will poll the bucket with a receive location configured to look for files with the .input extension and then pull the file down into BizTalk on premise in our company (Acme). In our BizTalk implementation we will simply push out the file to a folder to simulate the orders being sent to the order processing application. When the order processing application has finished processing orders it will output a response file which BizTalk will collect and push back out to the bucket in AWS S3 with a .output extension. Contoso will then be able to download this file as required and process it in their applications.

Setting up Amazon S3

I'm not going to go too much into the configuration and setup of AWS S3 but it didn't take long at all to set this up. Once you have an AWS account you can subscribe for the S3 service and setup a bucket. When setting up a bucket you need to choose its location. A bucket is simply like a root folder where you will put files and you can create files with a folder type key structure which will make them appear to be in folders within the bucket.

At time of writing AWS has a pretty good deal on the S3 package, particularly if you are a low user where you will be paying very little if anything if you are only playing around with the service or are only transferring/storing up to a certain amount of data.

(Make sure you review the pricing page for this though so you know where you stand).

 

Installing the NSoftware Adapter

The S3 adapter is available from NSoftware under a commercial and community license in their adapter bundle. For this post I was experimenting with their community edition. The install of the adapter was very straightforward and also includes the setup of a few other adapters which come together with the bundle.

 

I was using the adapter pack on BizTalk 2010 running on Windows 2008 R2.

 

Setting up the BizTalk Process to Receive from S3

The receive process was very simple to setup I firstly created a receive port which with a receive location which used the S3 adapter. The configuration settings I used are as follows:

 

Setting

Description

Access Key

This setting is the access key for your S3 account which you wish to access the bucket with

Secret Key

This is the secret key from your S3 account which you wish to access the bucket with

Bucket Name

This is the AWS S3 bucket which you want to access. The bucket is like a folder in the cloud which you want to access.

Object Mask

The object mask indicates the path type information for files you want to look for. This setting caused me some confusion initially because I wanted to have folders within my bucket initially and I had been trying to specify the folder as part of the bucket name property. I have provided some more samples combinations below to help explain this setting.

Delete Mode

This lets you specify what happens to the file in the cloud when BizTalk has collected it. If you don't set this to something other than the default then BizTalk would continue to download the same file over and over so I am not sure why you would want this setting to be anything other than Delete on Success.

  

 

To provide some more information on the object mask property here are some examples:

 

 

Scenario

Example

Collect all files in the bucket

*

Collect all files of the .input extension in the bucket

*.input

Collect all files of in a folder called "in"

in/*

 

On the send port side I had a port configured with a subscription for all files from the Receive port we have setup above and the send port would simply send the inbound file to a folder on the BizTalk developer machine. We will call this folder c:\LOBApplicationIn

Simulating the LOB Application processing the file

Once the BizTalk send port has written the file to the local disk simulating delivering if to an on premise line of business application we will simulate an application processing the file and making a response available for BizTalk by simply copy and pasting the file to a different folder. We will call this folder c:\LOBApplicationOut

Setting up the BizTalk process to send the file back to S3

Once the output file from the application is available we will setup a BizTalk receive port and location using the File adapter to poll for files coming out of the LOB application. These files will be picked up and sent to a send port which will use the S3 adapter to send the file back to S3.

 

In setting up the send port I simply used a subscription based on the send port name, however of more interest are the configuration properties of the S3 adapter which are shown in the below table.

 

Setting

Description

Access Key

This setting is the access key for your S3 account which you wish to access the bucket with

Secret Key

This is the secret key from your S3 account which you wish to access the bucket with

Bucket Name

This is the AWS S3 bucket which you want to access. The bucket is like a folder in the cloud which you want to access.

Object Key

This is the name/key of the file you wish to create. You can configure this with folder type semantics if you want to create folders in S3 (remembering that this is just a representation of the key heirachy).

 

One point of interest here is that the object key property supports most of the macros that the file adapter supports so you could set the object key to be something like %MessageID%.xml to create a unique file name each time for an xml file.

Business Partner uploads a file

The final part of the walk through is kicking off the process. I could use the AWS SDK to create a client and upload files but just as easily I can use the AWS console to manually upload the file. The good thing about this is it shows that in a B2B scenario this would support low-tech business partners who may simply have a user who uploads and downloads files as well as someone advanced with system integration to publish and read from S3.

 

When the business partner uploads their input file they could monitor the bucket and once BizTalk had pushed a response file back to S3 the partner would be able to download it as required.

Considerations around AWS S3

Following the walk through I thought I would make a few observations and comments about things I would consider when using adapter in a real world scenario.

 

High Availability for the BizTalk download

One of the points to consider when using this in a production like setup is configuring the receive ports for high availability. The way the S3 adapter works is very similar to the BizTalk FTP adapter in this aspect. You would want to setup a clustered BizTalk host to run the receive location on so that you can achieve high availability but not have to worry about the same file being picked up twice if it was concurrently read from two different servers.

 

What about multiple partners

If you were working on a scenario involving multiple partners I think it would be good practice to keep different partners data in separate buckets in S3. Looking at the way the security permissions work for S3 this would seem to be a sensible way to ensure that partners can not access each others data. (Happy for someone who may know better to challenge this assumption).

 

Bucket Permissions and ACL

I think before doing a real project involving this adapter you should have a clear understanding of the ACL usage for buckets. I would refer you to the following link for more information:

http://docs.amazonwebservices.com/AmazonS3/latest/dev/UsingAuthAccess.html

 

Summary Notes

In a proof of concept scenario I found this adapter really easy to work with and quite nice. To a BizTalk developer or administrator it looks very similar to the File and FTP adapters which is a good thing.

I was able to get my scenario up and running very quickly.

 

Useful Resources

The following links are some of the resources which may be useful to you as follow ups to this article:

AWS S3 Bucket documentation:

http://docs.amazonwebservices.com/AmazonS3/latest/dev/BucketAccess.html

AWS S3 API:

http://awsdocs.s3.amazonaws.com/S3/latest/s3-api.pdf

S3 Adapter:

http://www.nsoftware.com/products/biztalk/adapters/s3.aspx

S3 Adapter Community Edition:

http://www.nsoftware.com/products/biztalk/community.aspx

Posted on Wednesday, February 22, 2012 2:07 PM BizTalk | Back to top


Comments on this post: BizTalk Amazon S3 Adapter

# re: BizTalk Amazon S3 Adapter
Requesting Gravatar...
Hi, This is a good article on how to use this adapter. However my question is, does this work with BizTalk 2013 and 2013 R2. Please reply ASAP.
Left by Venkata Praveen Puvadi on Jun 28, 2016 6:37 AM

Your comment:
 (will show your gravatar)


Copyright © Michael Stephenson | Powered by: GeeksWithBlogs.net