NIEM Guy - NIEM, XML, and .NET

New Blog, Real Name

niemguy — Sun, 14 Dec 2008 05:19:06 GMT

After much thought I decided to begin blogging under my own domain. I've had it for a while but am going to slowly move things over there. Thanks to all the readers who have left comments and e-mails here. I'll be picking up the NIEM and .NET posts over at my permanent home on the web. Stop by and subscribe to the feed if you're interested!

As I transition, I'll link things here until everyone gets caught up.

Passing a Transaction from a WCF Operation to a LINQ Data Context

niemguy — Thu, 29 May 2008 02:23:16 GMT

I have a situation where I want to use a transaction to keep track of a particular message process in order to roll back various parts if there is a failure. The service operation is listening on a queue using the NetMSMQBinding and the operation is marked with the transaction properties of the OperationBehavior like so:
<OperationBehavior(TransactionAutoComplete:=True,TransactionScopeRequired:=True)> Public Sub SendHello( ...

Using the above attribute enables the operation for transaction support. The next trick is to grab the ambient transaction. Doing this requires pulling in the System.Transactions namespace.

Dim ambientTransaction as System.Transactions.Transaction = System.Transactions.Transaction.Current

Once you have the ambient transaction you can pass that to your LINQ data context using the data context's Connection.EnlistTransaction method.

myDataContext.Connection.EnlistTransaction(ambientTransaction)

Note: Enlisting the transaction will require you to open and close your connection. The code block worked for me. I tested by throwing an exception after this block and my database insert (usp_InsertHello is a stored procedure) did not commit.

Using myDataContext as new HelloWorldDataContext

myDataContext.Connection.EnlistTransaction(ambientTransaction)
myDataContext.Connection.Open
myDataContext.usp_InsertHello(helloText)
myDataContext.Connection.Close

End Using

Many objects have an enlist transaction method that you can pass your transaction to. The obvious benefit is that now all of the calls required for processing of the message will be handled in the scope of this transaction and if the method runs to completion all changes are committed - data operations are saved and the message is removed from the queue. In the event of a failure the whole operation is rolled back and the message is returned to the queue.

For a poison message strategy just putting back at the top of the queue is not ideal; this is just an example of how transactions can be used across multiple calls in the context of an operation.

Technorati Tags: MSMQ,WCF,Transactions,LINQ

Contract First Design: A Study of OneDOJ

niemguy — Wed, 21 May 2008 03:37:38 GMT

Disclaimer: This post speaks from an outsiders perspective. My interactions with this initiative have been from a regional implementers standpoint and I am not privy to any of the decisions that led to the OneDOJ initiative. This perspective comes from actual project work and reviewing open source information regarding the broader initiative.

One of my first web service projects involved implementing a service from a WSDL provided to us by the federal partner in charge of a regional data sharing project. This was my first brush with contract first and it did not occur to me until much later what this really meant.

As part of the Department of Justice OneDOJ initiative a new focus on information sharing created solutions which leveraged contract first / SOA approaches and principles. Now the articles tout this as "one database" but in reality what is happening behind the scenes is a lot of federated searching among different data sources throughout the country. The idea was that many regions or large cities have central databases already so instead of a lot of data duplication the DOJ could leverage this by creating a standard means for communicating justice data between involved partners. The FBI has been at this for a while with things like NCIC but that data is limited as to what it communicates. Extended data such as investigative records is not shared openly or easily.

The answer to this is a standard called LEXS (Law Enforcement Exchange System) which is a subset of NIEM that defines structures and messages for exchanging different types of justice information between justice entities (Law Enforcement, Courts, etc). LEXS also includes WSDL's for two major feature areas: Publish Discover (PD) and Search Retrieve (SR).

PD contains operations and messages to share the capabilities of your implementation. Things like the version of LEXS you support, operations you support, etc. When I implemented the WSDL for our exchange I did not implement PD so I cannot speak to much of it other than what I reviewed of the WSDL itself and the documentation. I see the idea very much inline with the meta-data discovery that is mentioned in a lot of SOA discussions. It is implemented (from what I can see) using a standard web service. PD does not appear to implement any of the WS-Discovery protocols. Not that it is a bad thing, I'm sure interop was a prime concern for this.

SR contains operations and messages to share data. This is the real meat of LEXS. LEXS defines basic operations to request data using free text or structured searches and messages to return data formatted using the LEXS subset of NIEM. There is also some extensibility built in so that entities can utilize existing subsets of NIEM within those return structures. When I implemented SR for the project we are involved in I implemented version 2.0. Currently LEXS is up to version 3.1.1.

3.1.1 includes some major changes from 2.0 including the use of NIEM 2.0. The WSDL's have changed slightly due to this so that they can accommodate for things you can do with NIEM 2.0 that were not possible in NIEM 1.0 or Justice XML.

In the public sector very few companies would be concerned with actually implementing this specific schema. I bring it up because I think the approach is very cutting edge; particularly because it is being driven by some of the largest and oldest institutions of our government that historically have problems sharing. The DOJ has adopted an SOA approach to their information sharing and are following through on all fronts including governance. I have spoke on a few conference calls with some of the decision makers and influencers behind this program. I was very impressed that they spoke in terms of the WSDL contracts (using operation names) and were linking the business process to the actual capabilities expressed in LEXS. They are addressing concerns from different partners on what and how much information to share, etc. All in all a fascinating initiative to get a glimpse of.

Information sharing will not be the same going forward. The dinosaur models of closed, custom formats are going bye-bye and being replaced with open formats and open standards.

You can study more on LEXS by reviewing the schemas and documentation from the Office of Justice Programs web site.

Technorati Tags: NIEM,SOA,OneDOJ

A Contract First Approach with WCF

niemguy — Wed, 21 May 2008 03:13:37 GMT

One of the goals of a large state-wide multi-agency data sharing project my organization is participating in was to start with a partner designed WSDL and then implement it in the underlying technology of the partner's choice. For my organization this technology is Windows Communication Foundation. From it's inception the WSDL had to be simple and interoperable. This is the second time I had started with a WSDL and then created a conformant service.

My organization was one of the few that actually had a good base of experience with web services and NIEM. We were best-suited to provide input into the WSDL and work with the leading partner (the grant receiver and "pockets"). This was a great exercise in a few things:

Virtual project management: Several hundred miles separate all the partners. My counter part at the lead partner is a young guy like me and preferred email over telephone (most times). We hashed out the WSDL and several other items with a few brief to medium sized e-mails.

Interoperability: The lead partner speaks Java, we speak .NET. Other partners speak neither. We learned to meet in the middle and actually came to appreciate the underlying technologies more (for the record I have never been a huge fan of Java - it's mostly the runtimes fault I think, not the language itself).

Architectural leadership: This collaborative, open standards approach required a new outlook on connected systems and leadership in pushing change and promoting it to the other partners who have not caught up with recent developments like SOA, web services, and open standards.

After two cuts we had a WSDL that encompassed the 7 operations we would need and defined the input and output messages. Besides a few naming items (I was new to creating a WSDL from scratch) we ended up with a contract that is clear on the intent of each operation and what inputs/outputs it will have. The outputs were generally simple, requiring only an acknowledgement of message receipt. One of the design goals is to minimize the amount of time web service calls have to wait because much of the processing required will happen after the messages are received.

To implement the WSDL I researched WCF and found that Svcutil would do exactly what I needed. I tinkered with the options and successfully got the WSDL to import in a very short amount of time. The implementation language is VB.NET and everything came right in. I did not like that the importer created a single file with all the requisite classes (data contract, message contract, service contract, and service stub). I had started modeling a few of these early on and so I ended up cutting and pasting a few things but other than that it was very smooth. I saw very few warning messages and overall was happy with the speed and results of using Svcutil.

I sent the sample soap output (captured using WCF tracing) and received positive feedback from the lead partner. The message conformed to our contract and we will be able to proceed with a test soon.

Now, I wish I could say that another partner involved had the same luck. This partner is particularly behind in the web service area so in lieu of seeking training and developing their own knowledge base they purchased a magic box product. I will not name this vendor as they have at least admitted to their fault in this. This particular product is not able to import a WSDL the way WCF and Java can. This led to a very sticky situation and some finger pointing that was eventually resolved when the vendor did state the product could not support what we were doing. An unfortunate turn of events as we had already put considerable time into drafting the WSDL and message schemes and had already opted out of the bus approach in favor of the distributed contract based approach.

Due to the agility of the lead partner they were able to create a client to work with the disfigured WSDL generated by the third-party utility (which incidentally is Java based).

I oft wondered when the magic boxes would replace coders entirely but it seems as though we are not quite there. I cannot say I was disappointed at this "set back". The cost of this magic box and its training was about 4 times the cost that will be billed to the grant to have me do the work manually using an open design and open standards that will allow us to change the underlying implementation with minimal impact (other than time involved).

Everything has it's place and I'm sure one day I will have a need for such a device. Until then I will continue leveraging automated tools where handy and doing some manual labor to ensure services and components conform to good designs and practices.

Finding a NIEM Element using XDocument.Descendants

niemguy — Sun, 09 Mar 2008 19:56:15 GMT

This is part one of a multi-part post where I will show some of the techniques we've been using to parse NIEM XML documents using LINQ.

Microsoft has definitely put a lot of time into thinking out the System.Xml.Linq features. When working with NIEM XML the essential thing to remember is that the only thing that ever really changes is the name of the tag and sometimes the namespace. This makes NIEM very easy to parse with LINQ because LINQ does not look at an XML document like the DOM - a hierarchical structure, but it can look at it as a flat document as well and this is the power we will leverage.

The key to this are a couple function calls (need to be refactored and probably could be abstracted a little more) named GetElement and GetElements. GetElement will return the first instance of the element found based on the namespace and element name. GetElements is the same but it returns an IEnumerable of XElement.

Public Shared Function FindElement(ByVal doc As XDocument, ByVal elementName As String, ByVal xNamespace As XNamespace) As XElement
Return doc.Descendants(xNamespace + elementName)(0)
End Function

Public Shared Function FindElements(ByVal doc As XDocument, ByVal elementName As String, ByVal xNamespace As XNamespace) As IEnumerable(Of XElement)
Return doc.Descendants(xNamespace + elementName)
End Function

I tucked those methods in a class called XmlParser. Notice the use of the + and not the &. In VB when you join strings the + will throw compiler errors. However for joining a XNamespace with a string element name you must use the + sign. I didn't see this exact case documented anywhere but I just took a guess after the & was giving me compiler errors.

Here is a sample doc - declared inline using the Xml literals feature in VB.
Dim doc As XDocument = <?xml version="1.0"?>
                       <myDoc xmlns="http://niemguy.com/mydoc/1.0" xmlns:mydoc="http://niemguy.com/mydoc/1.0" xmlns:nc="http://niem.gov/core/2.0">
                           <people>
                               <nc:Person>
                                   <nc:PersonName>
                                       <nc:PersonGivenName>Chris</nc:PersonGivenName>
                                   </nc:PersonName>
                                   <nc:PersonBirthDate>
                                       <nc:Date>1978-08-12</nc:Date>
                                   </nc:PersonBirthDate>
                               </nc:Person>
                               <nc:Person>
                                   <nc:PersonName>
                                       <nc:PersonGivenName>Sheila</nc:PersonGivenName>
                                   </nc:PersonName>
                                   <nc:PersonBirthDate>
                                       <nc:Date>1977-02-24</nc:Date>
                                   </nc:PersonBirthDate>
                               </nc:Person>
                               <nc:Person>
                                   <nc:PersonName>
                                       <nc:PersonGivenName>David</nc:PersonGivenName>
                                   </nc:PersonName>
                                   <nc:PersonBirthDate>
                                       <nc:Date>1982-06-05</nc:Date>
                                   </nc:PersonBirthDate>
                               </nc:Person>
                           </people>
                       </myDoc>

And finally, here is how we can use the GetElement functions to find some elements.

Dim nameEl As XElement = XmlParser.FindElement(doc, "PersonGivenName", GetXmlNamespace(nc))
Console.WriteLine(nameEl.Value)

Dim birthDays As IEnumerable(Of XElement) = XmlParser.FindElements(doc, "PersonBirthDate", GetXmlNamespace(nc))

For Each birthday In birthDays
Console.WriteLine(birthday...<nc:Date>.Value)
Next

A key thing to note here is the use of GetXmlNamespace. The GetXmlNamespace function is a framework method that returns the XNamespace based on the prefix of a namespace you have imported at the project or file level. For reference are the file level imports I used:

Imports <xmlns:mydoc="http://niemguy.com/mydoc/1.0">
Imports <xmlns:nc="http://niem.gov/core/2.0">

The nice thing about GetXmlNamespace is that you get intellisense on the namespaces you have in your project or file. This makes it really handy. The drawback is that you can only get them from there, or at least I have not found a way to get them listed from an external source or the document itself. For most exchanges that would be ok if you're custom tailoring it a little - but if you're after a generic parser then this may not be the way to go.

Next post will describe a abstract (mustInherit) bass class that we created as a base document parser and then we extend it for special documents.

VS 2008: Importing XML Namespaces at the project level

niemguy — Thu, 31 Jan 2008 13:15:51 GMT

While working on a NIEM parser library I found that you can import XML namespaces at the project level. Previously I posted about doing this at the top of your .VB class file:

Imports <xmlns:nc="http://niem.gov/niem/niem-core/2.0">

That allows you to use the prefix nc: in LINQ queries to work with NIEM documents. Well today I discovered you can add those at the project level. Right click your project and go to properties. In the properties window go to references. Near the bottom you can type / paste your name space in (leave out the imports statement of course) and then click add user reference. Voila! You can now use that in any class in your project.

Article: How to use the NIEM Schema Subset Generator

niemguy — Tue, 18 Dec 2007 17:13:08 GMT

I wrote a very simple article on using the NIEM Schema Subset Generator. There is a lot more to IEPDs and the NIEM model than what I conveyed but this will be a good starting point.

How-to: Use the NIEM Schema Subset Generator to Build a NIEM Subset Schema

Web Services - Exposing the Schema Contract when Using NIEM Schemas?

niemguy — Mon, 10 Dec 2007 15:26:43 GMT

In the new paradigm of contract first designs for services a lot of it revolves around exposing a contract that defines the operations and messages for a service. In terms of Web Services (or WCF exposed via a web service) this also means exposing the schemas via the WSDL so that consumers know the inputs and outputs for that service and its methods.

This is my big question: When working with NIEM how many are exposing their subset schemas to their web services and how many are using a generic "Package as string" approach where the package details are agreed upon outside the context of your service?

I have seen this done both ways and recently have chosen to follow one direction. In my most recent service adventure I modeled my messages to contain some required input parameters (need to know who is making the request and for what agency) and in my return message I have the generic "Package as string" where the package is going to be this very large collection of documents formatted in NIEM.

Returning "Package as string" is very ambiguous and will make a few people cringe or sneer when (if) they read this. But there is a reason for this and it's two fold:

1) In terms of Interop there are several technologies out there that do not support mulitple imported schemas in a WSDL and they also do not support substitutions via the WSDL. (Noting that currently interop is not a major concern for my service but that does not mean I shouldn't be prepared for it).

2) Imposing that any consumers must deserialize or validate that structure upon receipt. This one really gets into the realm of performance. Deserialization is an expensive process and so is schema validation and imposing that on a client could cause performance issues.

The disconnect in the contract will be that consumers will have to validate (if they choose to) the message against the schema post receipt. This obviously has the consequence that if somehow the service returns an invalid message package than the consumer has to make a subsequent request to get a valid package. So, in effect, that also would be a performance hit. Far less likely to happen (yes, I trust my service) but it could.

In other exchanges not exposing the full NIEM / Justice XML schema subset via the service has been the norm. The schemas are handled outside the context of the actual receiving of a message from ther service(s) involved.

During one project the weakness was thought to be in .NET; ironically it turns out .NET far better support for schemas in WSDL than others. Which does illustrate Microsofts great adoption of XML and all it's related technologies.

Technorati Tags: NIEM WCF Contract First

Piecing it all together

niemguy — Thu, 06 Dec 2007 12:05:11 GMT

The subtitle of this blog started out as gluing it all together. I quickly realized that when thinking in terms of a loosely coupled environment glue isn't necessarily the best tool in your tool box.

I will try to post a little more in depth follow up to Querying a NIEM XML Document With LINQ as soon as I can. After some review I realized just how reusable some of the parsing methods I wrote are. Wouldn't it be nice to have a single NIEM XML parser that you could extend if you received additional elements not in the base NIEM schemas?

Querying a NIEM XML Document Using LINQ

niemguy — Thu, 06 Dec 2007 03:44:08 GMT

If you are not familiar with the new LINQ syntax in .NET 3.5 I recommend you read up on that first. Language Integrated Query (LINQ) is the new foundation for querying anything (I mean anything) in .NET and a lot of effort has been put into LINQ and XML integration. I did some research a few months back on it but it was only recently that I realized how much Microsoft put into LINQ to XML.

I am a huge fan of intellisense. That is the best thing that MS ever perfected in Studio. I spend a lot of time making sure things show up properly via intellisense and it really irks me if something isn't there. Studio 2008 (I've used Professional and now Team Development edition) has the best intellisense support of any version yet and MS pulled out all the stops and you can get intellisense darn near everywhere in a VB, XML, or other types of files.

Before you go on I highly recommend spending the 15 minutes watching this video from Beth Massi of the VB Team. She covers how to add intellisense support for XML Schemas and documents inside of LINQ queries.

Prerequisites:
1) A NIEM conformant XML schema loaded into your Visual Studio Solution. Be sure to have it properly seated in its folders. Here is a screenshot of mine:

If you generated your schemas from the Schema Subset Generator on the NIEM tools page just copy the folder structure from the zip file into your project. Studio should immediately start parsing it.

2) Import your XML namespaces into the class file you are working in. This is done via the new Import <xmlns> statement like so:

[The blacked out line is because I covered over one that I didn't want you to see :)]

3) Load a valid XML Document. If you have one on disk you can load it by creating an XDocument and calling the load method and passing the file path like so:
Dim Doc as XDocument = XDocument.Load("C:\myfile.xml")

If you have a string of XML in a variable you can load it calling the parse method of the XDocument:

Dim MyXml as string = "<root><tag1>test</tag2></root>"

Dim Doc as XDocument = XDocument.Parse(MyXml)

Once you have those things set up you are ready to begin. As with any type of query language you must know the structure. Studio will give you intellisense during the queries but you have to closely look for a check mark indicator to know that your selection is valid for where you are at in your structure. Example:

Note the small green circle with the check. That indicates that my choice is valid for where I was in the path -nc:Person.nc:PersonName.nc:PersonGivenName. I could have typed something else in there and it would have accepted it, however it would fail at runtime.

In the query above I am selecting a Person element from the document based on the Reference Id. From there I am drilling down to their name. When calling that method I am passing the string Id I want to find.

Example Document:
<j:CitationSubject>

<nc:RoleOfPersonReference s:ref="P1">

</j:CitationSubject>

<nc:Person s:id="P1">

<nc:PersonName><nc:PersonGivenName>Chris</..></..>

</..>

The Query would return "Chris" as it is the value of the PersonGivenName element. Because FirstNameQuery is an IEnumerable of XElement I use the index to return the first result because that is the one I want (and when matching by Id like I am there should only ever be one). [NB: Because I was demoing the intellisense I had to cut off the .Value at the end of the select. The full select should look like: Select person...<nc:PersonName>...<nc:PersonGiveName>.Value]

The only caveat is that LINQ's Intellisense does not support substitution groups. In the normal XML document intellisense in Studio 2008 you will get a list of valid substitutions (nc:AddressRepresentation anyone?). It is not a big deal, you just have to know what tag to put in. For example, under nc:Location.nc:LocationAddress there is a substitution called "AddressRepresentation". I substitute nc:LocationStreet at that point. With LINQ, I have to manually type that in, but after I get to the nc:LocationStreet the intellisense picks back up and I get choices like nc:StreetName, etc. I know, they can't spoil us but if they added that I would fly to Seattle to buy someone coffee!

So how does this fare versus serialization? I ran some very rudimentary benchmarks. For loading a document with 50 of my citations in it (for reference I inherited from j:CitationType and extended it...a bunch) it took LINQ 00:00:00.0624724 ms to load the document.

LINQ:
Start: 21:17:59.2930220
Document Loaded: 21:17:59.3554944
Load Time 00:00:00.0624724

The same document after invoking a serializer to deserialize the XML to the matching class structure:

Deserializer
Start: 21:20:38.8631497
Serialize Complete: 21:20:39.5815823
Load Time 00:00:00.7184326

That is quite a substantial difference to me considering the parser I'm developing may need to chew through thousands of citations at a time.

I see a lot of potential for using LINQ to query NIEM conformant XML. On the XML generation side I will still use the serializer for the time being. But when it comes to parsing, I'm going to push using LINQ because of its speed and flexibility.

Technorati Tags: NIEM,XML,LINQ,.NET XML,VB.NET,LINQ to XML