Architecture

Dependency Management - Update

Remi Sabourin — Fri, 04 Sep 2009 06:03:14 GMT

I posted last year regarding a dependency management tool I was contemplating. Well, I can happily announce that I did create and launch the tool as I had envisioned it. (I just actually completed a lunch and learn today where I unveiled it to a few more people within the company)

In essence, it works by spidering the BUILD environment nightly and records all dependencies from code to code, from code to database and from database to database. We have over 100 .Net solutions with multiple projects within and all the references down to the method and database object level are recorded. It was very satisfying recently to clean up 1/3 of the code from a project as it was found to be no longer referenced (around 100 out of 300 stored procedures were removed). This alone shaved about 1 month of developer effort out of a major refactoring initiative.

I can't post or easily demo the tool as I developed it to work within our company intranet, but I thought I should point out some of the libraries I settled on to make this tool possible (these differ from the ones I was considering when I first posted about this last year). If you haven't played with these but are the type that like to create innovative developer tools or efficiencies, these are a must for your arsenal:

Spidering DLLs: Mono.Cecil
Spidering Code: NRefactory
Spidering Databases: SMO

The Definition of a “Bad Design”

Remi Sabourin — Fri, 14 Aug 2009 03:24:06 GMT

I have been recently dealing with a somewhat tangled set of components where over time, developers have added references as needed to get things done quickly, creating circular and upstream references, resulting in low level layers depending on high level layers. While looking for a nice little refresher online on dependency injection (incidentally, this is a great refactoring technique to fix this situation fast), I found this nice concise definition on bad design provided in an article by Robert C. Martin, Thoughtworks

This seems like a great step to facilitate design reviews in a repeatable manner while avoiding digression into personal preference or opinion. I look forward to reviewing designs on the basis of Rigidity, Fragility and Immobility. For the full article, visit http://www.objectmentor.com/resources/articles/dip.pdf

------------------------------------------- ARTICLE EXCERPT -------------------------------------------

Have you ever presented a software design, that you were especially proud of, for review by a peer? Did that peer say, in a whining derisive sneer, something like: “Why’d you do it that way?”. Certainly this has happened to me, and I have seen it happen to many other engineers too. Clearly the disagreeing engineers are not using the same criteria for defining what “bad design” is. The most common criterion that I have seen used is the TNTWIWHDI or “That’s not the way I would have done it” criterion.

But there is one set of criteria that I think all engineers will agree with. A piece of software that fulfills its requirements and yet exhibits any or all of the following three traits has a bad design.

It is hard to change because every change affects too many other parts of the sys-
tem. (Rigidity)
When you make a change, unexpected parts of the system break. (Fragility)
It is hard to reuse in another application because it cannot be disentangled from
the current application. (Immobility)

Moreover, it would be difficult to demonstrate that a piece of software that exhibits none of those traits, i.e. it is flexible, robust, and reusable, and that also fulfills all its requirements, has a bad design. Thus, we can use these three traits as a way to unambiguously decide if a design is “good” or “bad”

------------------------------------------- ARTICLE EXCERPT -------------------------------------------

Dependency Management

Remi Sabourin — Thu, 24 Jul 2008 07:09:32 GMT

The Challenge

Imagine a development shop that maintains 100 .Net solutions, each composed of multiple projects. Imagine roughly half of these being shared domain services or shared common libraries, the other half being end web applications servicing multiple large clients. Imagine any given project having the ability to reference classes/methods on any suitable re-usable service. Imagine trying to get a grip on it all - a great architectural paradise or a great architectural nightmare. Welcome to my world!

For a while I have been thinking about a way to manage dependencies. Dependencies hold us back, they make us scared to change things. But when we don't evolve, we don't improve, we spend time doing complete rewrites or adding layer over layer of code bloat and we don't refactor to achieve enough of my favorite thing..... (c'mon guys, you know it's coming).... RE-USE.

I am dreaming of a searchable database of dependencies, most it being automatically generated and updated, although some could be specified manually if indirect. Basically a master tool that would spider our build and production server(s) and document dependencies to any artifact - code, databases, folders, etc. If we want to remove/update/rename anything, we can be confident in our actions. It would point out orphans that we can clean up so as to keep our systems/code clean. It would plug into an automated build process and send dependency reports and alerts.

The Research

Voice inside my head says "Start with the code". I want to know for every method/property what method/property calls it in any project. Here's what has transpired so far:

First I figure a little app that calls the .Net framework could be in order, that spits the results to a database. I know System.Reflection has metadata on dlls built in. On further inspection, System.Reflection only exposes things in the manifest, which does include assembly references (a start), but does not provide an insight into method references. Next I look at the CodeDom namespace - if similar to XML DOM, I could create a solution to parse the source code and I could iterate the dom to find function calls. In fact, there is a CreateParser() method - too bad for me there's no implementation. Maybe in a future .Net version?

I took a look at NRefactory that is an API that comes with the SharpDevelop IDE but can't find much documentation on this to see if it could do the job.

Examined some out-of-the-box add-ins/tools, NDepends being a possibility. I figured I would have an automated build output an XML file from NDepends and then I could have code that parses it for the dependencies and puts them in a database.. NDepends looks like a good tool with some interesting features and a query language, but it does not output method dependencies (only type and assembly ones), although you can see them if you drill down using the application. There is also licensing per individual.

I thought dlls could maybe be reversed engineered; there is a tool that comes with .Net called ildasm.exe. Its output though is intermediate language code which I'm not familiar with and would be kindof brutal to parse. I found out Reflector has a "hidden" API if you reference the EXE. A nice MSDN article that has a promising name by James McCaffrey: http://msdn.microsoft.com/en-us/magazine/cc163641.aspx. May try this out, does not seem like the perfect high-level API but may not be too complicated knowing what exact statements to look for in intermediate language.

Then found out that FxCop also has a "hidden" API if you reference the EXE. Here's a nice doc Jason Kresowaty created to document this API: http://www.binarycoder.net/fxcop/pdf/fxcop.pdf. This looks like the abstract API I was looking for. This is my best bet right now.

The Execution

Concluding my research, my goal in the next few weeks is to:

- Use System.Reflection to log all the assembly references found in dlls

- Use FxCop or Reflector API to log all the method references found in dlls

- Log all the source code in the database (if this is impractical, use indexing service) so I can query it. What the hey, why not use SQL Server's scptxfr.exe utility and log all the stored procs/views/triggers too.

- Have a GUI for the development teams that displays dependencies between assemblies and allows you to drill down to methods. Have GUI that allows you to search for ad hoc strings in the source code..

I'll let you know how it turns out.

P.S. If you know of some wicked tool that can do this for me and allow me to RE-USE and conserve time, let me know!