Dynamic Concepts (in) Development

My New Blog

David V. Corbin — Mon, 18 Apr 2011 23:46:47 GMT

After much careful consideration, I have decided to host my primary blog with WordPress

Software Development in a Dynamic Environment

Working with the team at GeeksWithBlogs.net has been a wonderful experience, and I will periodically make posts here.

Please stop by the new blog, and see what is happening....

GiveCamp NYC 2011 - Project Development in under 48 hours

David V. Corbin — Tue, 18 Jan 2011 13:50:08 GMT

This past weekend (Jan 14-16) there were GiveCamp events accross the country. At the New York City event, nearly 100 developers, designers and database administrators came together to dontate their time to provide technology solutions for over a half dozen charities.

The experience was amazing, projects that would normally take weeks or months went from concept to delivery between 6:30pm Friday evening and 3:00pm Sunday afternoon. There were many comparisions to both Habitat for Humanity and Extreme Makeover Home Edition. Just like those events, the concept was simply, leverage the power of many volunteers to provide something (in this case, applied technology) that the people in need could not afford.

The majority of the projects were to create (or significantly rework) a web presence for the charity. Depending on the charities needs, different technologies were choosen (Joomla, WordPress, Orchard, were a few) and many of the websites included the ability to access online donations (typically via PayPal or Google Checkout). Typically the team that created the solution had only one or two members who had experience with the technology prior to the start of the weekend, making the successes even more amazing.

The project I was involved with, was a bit different from the rest [aren't most of my projects?]. Robin Hood is a charity that targets poverty in New York City by finding and funding the best and most effective programs and partnering with them with them to maximize results. They have an in-house staff of IT that develops and manages their web-site as well as an internal SharePoint site. In order to maximize the capabilities of the SharePoint site, they needed four functional items that were beyond their capabilities.

This meant we were going to be doing some "hard core development", with the result being a set of Visual Studio 2010 projects that the team at Robin Hood could then integrate with their overall site, and also continue to maintain the codebase to address the inevitable changes that would be necessary. Based on my experiences, I knew that if we adopted a purely "ad-hoc" process, that items would be missed and the it would be nearly impossible to coordinate the 9 developers (some of which had zero SharePoint knowledge, others which had SharePoint, but not Development, etc) in a way that could make the best use of their skills.

I began to think...Can you adopt a (Very) light weight ALM practices and use TFS 2010 to manage the projects development when the iteration/sprint cycle is 2-3 hours, to go from concept to a working set of deliverables uploaded to a "production" SharePoint site????

I spend Friday night setting up a development local area network (local wireless access to our "server", with a bridge over the facility's wireless network to access external sites), a development domain, a TFS server complete with a build machine, and a few Visual Studio 2010 / SharePoint 2010 development VM's. By Saturday morning it was ready to go, with the initial 4 requirements entered in as User Stories (Epics) and a number of workitems (tasks) ready for assignment.

For the next 30 hours, the team worked on the items, issues/bugs/tasks were created, items were commited to source control, and the incremental results uploaded to a temporary server (Thank you Becky Isserman for installing and configuring SharePoint!! in this environment).

With three hours left before the presentation of our work to the entire audience, we had "beta" versions of each of the components up and running on the "production server", and a full set of artifacts recorded in TFS that could be turned over the the charity (by detaching and delivering the TPC).

To the best of my knowledge, this is the shortest development cycle where TFS has been setup, configured, and used to create a set of deliverables. We took a lot of shortcuts in order to tailor "the process" so that it would work smoothly under this extreme time compression, but the only word I can think of to describe the result was Amazing!.

After a good nights sleep, I was reflecting back on the expreience. By Sunday, most of us were so sleep deprived that short term memory, retention and other cognititave thought processes were significantly impaired. If we had gone with an "ad-hoc" process, and attempted to just use the white board (which was filled and erased many times) along with manual notes, I am convinced that we would not have been able to achieve the results that we did.

Special thanks go out to team members (alhpabetically): Azret Botash, Becky Isserman, Qinfu Chen, Rajeshwar Kokkula, Ryan Finnesey, Tom Daly, and Tome Tanasovski. Jimmy Park from Robin Hood also deserves a big hand, for being able to deal with all of us!

Do (Software) Architects Architect?

David V. Corbin — Thu, 18 Nov 2010 15:04:25 GMT

The first question, is “Why is architect being used as a verb?”. Mirriam-Webster dictionary does not contain a definition of architect as a verb, nor do many other recognized dictionaries. On the other hand, dictionary.com does add an entry for the use of the word as a verb:

Architect
4. to plan, organize, or structure as an architect: The house is well architected.

This gives us three key words to start with: “plan”, “organize” and “structure”. Unfortunately these terms can apply to work done by many others involved in the project ranging from project managers to software engineers. So we must return to the key part of the definition “as an architect”

–noun
1. a person who engages in the profession of architecture.
2. a person professionally engaged in the design of certain large constructions other than buildings and the like: landscape architect; naval architect.
3. the deviser, maker, or creator of anything: the architects of the Constitution of the United States

Each these definitions provide some clues, however the last two do nothing to clarify the boundary between the architect’s role and that of the other team members. We are forced to return to the dictionary to see what the “profession of architecture” entails. Emphasis added to highlight aspects which will be discussed in this post.

architecture
–noun
1. the profession of designing buildings, open areas, communities, and other artificial constructions and environments, usually with some regard to aesthetic effect. Architecture often includes design or selection of furnishings and decorations, supervision of construction work, and the examination, restoration, or remodeling of existing buildings.
2. the character or style of building: the architecture of Paris; Romanesque architecture.
3. the action or process of building; construction.
4. the result or product of architectural work, as a building.
5. buildings collectively.
6. a fundamental underlying design of computer hardware, software, or both.
7. the structure of anything: the architecture of a novel.

None of these definitions (with the exception of #6) are specific to software architecture, but still can supply some valuable information as highlighted in the above text. These items can be divided into two distinct categories. The first (and most obvious) are those that relate to a specific undertaking. The second are the items which are more reflective in nature, typically the results of finding the common (and contrasting) properties among multiple specific instances. To simplify discussion of these two group, I propose the use of two terms: “Architectural Design” for the first category, and “Architectural Style” for the second.

Returning to the title question, we can rephrase it as two distinct questions: “Do software architects create architectural designs?” and “Do software architects create architectural styles?”. The answer to the first question is a resounding yes (although we must still determine show architectural designs are distinguished from more detailed design and construction artifacts and activities). The creation of architectural styles is a different matter.

The creation of a new “style” in any field is not something that is frequently attributable to a single person (or team). In the cases where attribution can be given, it is unusual for the originator to even have been aware that they were creating something that would be recognized and adopted as a style by their peers. Consider “brick&mortar” architecture; “Greek”, “Roman”, “Victorian”, “Tudor”, “Modern”, “Post-Modern” are all recognized and fairly well defined; however, all of them were identified “after the fact” by analyzing common features that were used by architects in creating specific “architectural designs” at a particular point in time and/or geographic location.

I believe that the same is true for software (and other technology related) architecture(s). A recognized architecture style is something that is typically distilled from the work of many, with only a few cases being the result of a specific individual or team.

So to all the software architects reading this, I wish you well in creating robust, effective architectural designs – and (if you are very skillful and lucky) may history possibly look back at some of your work and recognize your contribution to an architectural style.

Selectively Exposing Functionallity in .Net

David V. Corbin — Fri, 07 May 2010 02:44:51 GMT

Any developer should be aware of the principles of encapsulation, cross-tier isolation, and cross-functional separation of concerns. However, it seems the few take the time to consider the adage of "minimal yet complete"¹when developing the software.

Consider the exposure of "business objects" to the user interface. Some common situations occur:

Accessing a given element requires a compound set of calls that do not "make sense" to the User Interface.
More information than absolutely required is exposed to the user interface

It would be much cleaner if a custom interface was provided that exposed exactly (and only) the information that is required by the consumer.

Achieving this using conventional techniques would require the creation (and maintenance!) of custom classes to filter and transpose the information into the ideal format. Determining the ROI on this approach can be very difficult to ascertain, and as a result it is often ignored completely.

There is another approach, which is largely made practical by virtual of the Action and Func delegates. From a callers point of view, the following two samples can be used interchangeably:

    interface ISomeInterface
    {
        void SampleMethod1(string param);
        string SamepleMethod2(string param);
    }

    class ISomeInterface
    {
        public Action<string> SampleMethod1 {get; }
        public Func<string,string> SamepleMethod2 {get; }
    }

The capabilities this simple changes enable are significant (and remember it does not cange the syntax at the call site):

The delegates can be initialized to directly call the proper method of any target class.
The delegates can be dynamically updated based on the current state.
The "interface" can NOT be cast to the concrete class (which often exposes more functionallity).
This patterns
By limiting the interface to the exact functionallity required, the reduced surface area will typically result in lower development, testing and maintenance costs.

We are currently in the process of posting a project on CodePlex which illustrates this (and many other) techniques which have proven helpful in creating robust yet flexible solutions that are highly efficient² and maintainable. This post will be updated as soon as the project is published.

1) Credit: Scott Meyers, Effective C++, Addison-Wesley 1992

2) For those who read my previous post on performance it should be noted that the use of delegates is on the same order of magnitude (actually a tiny amount faster) as conventional interfaces.

Interfaces and Virtuals Everywhere????

David V. Corbin — Fri, 07 May 2010 01:46:19 GMT

First a disclaimer; this post is about micro-optimization of C# programs and does not apply to most common scenarios - but when it does, it is important to know.

Many developers are in the habit of declaring member virtual to allow for future expansion or using interface based designs¹. Few of these developers think about what the runtime performance impact of this decision is.

A simple test will show that this decision can have a serious impact. For our purposes, we used a simple loop to time the execution of 1 billion calls to both non-virtual and virtual implementations of a method that took no parameters and had a void return type:

Direct Call: 1.5uS
Virtual Call: 13.0uS

The overhead of the call increased by nearly an order of magnitude!

Once again, it is important to realize that if the method does anything of significance then this ratio drops quite quickly. If the method does just 1mS of work, then the differential only accounts for a 1% decrease in performance. Additionally the method in question must be called thousands of times in order to produce a meaqsurable impact at the application level.

Yet let us consider a situation such as the per-pixel processing of a graphics processing application. Here we may have a method which is called millions of times and even the slightest increase in overhead can have significant ramification. In this case using either explicit virtuals or interface based constructs is likely to be a mistake.

In conclusion, good design principles should always be the driving force behind descisions such as these; but remember that these decisions do not come for free.

1) When a concrete class member implements an interface it does not need to be explicitly marked as virtual (unless, of course, it is to be overriden in a derived concerete class). Nevertheless, when accessed via the interface it behaves exactly as if it had been marked as virtual.

Does Test Driven Development (TDD) improve Quality and Correctness? (Part 1)

David V. Corbin — Thu, 11 Mar 2010 14:16:43 GMT

Since the dawn of the computer age, various methodologies have been introduced to improve quality and reduce cost. In this posting, I will by sharing my experiences with Test Driven Development; both its benefits and limitations.

To start this topic, we need to agree on what TDD is. The first is to define each of the three words as used in this context.

Test - An item or action which measures something in some quantifiable form.
Driven - The primary motivation or focus of a series of activities (process)
Development - All phases of a software project/product from concept through delivery.

The above are very simple definitions that result in the following:

"TDD is a process where the primary focus is on measuring and quantifying
all aspects of the creation of a (software) product."

There are many places where TDD is used outside of software development, even though it is not known by this name. Consider the (conventional) education process that most of us grew up on. The focus was to get the best grades as measured by different tests. Many of these tests measured rote memorization and not understanding of the subject matter. The result of this that many people graduated with high scores but without "quality and correctness" in their ability to utilize the subject matter (of course, the flip side is true where certain people DID understand the material but were not very good at taking this type of test).

Returning to software development, let us look at some common scenarios. While these items are generally applicable regardless of platform, language and tools; the remainder of this post will utilize Microsoft Visual Studio and Team Foundation Server (TFS) for examples.

It should be realized that everyone does at least some aspect of TDD. At the most rudimentary level, getting a program to compile involves a "pass/fail" measurement (is the syntax valid) that drives their ability to proceed further (run the program).

Other developers may create "Unit Tests" in the belief that having a test for every method/property of a class and good code coverage is the goal of TDD. These items may be helpful and even important, but really only address a small aspect of the overall effort.

To see TDD in a bigger view, lets identify the various activities that are part of the Software Development LifeCycle. These are going to be presented in a Waterfall style for simplicity, but each item also occurs within Iterative methodologies such as Agile/Scrum. the key ones here are:

Requirements Gathering
Architecture
Design
Implementation
Quality Assurance

Can each of these items be subjected to a process which establishes metrics (quantified metrics) that reflect both the quality and correctness of each item? It should be clear that conventional Unit Tests do not apply to all of these items; at best they can verify that a local aspect (e.g. a Class/Method) of implementation matches the (test writers perspective of) the appropriate design document. So what can we do?

For each of area, the goal is to create tests that are quantifiable and durable. The ability to quantify the measurements (beyond a simple pass/fail) is critical to tracking progress(eventually measuring the level of success that has been achieved) and for providing clear information on what items need to be addressed (along with the appropriate time to address them - in varying levels of detail) . Durability is important so that the test can be reapplied (ideally in an automated fashion) over the entire cycle.

Returning for a moment back to our "education example", one must also be careful of how the tests are organized and how the measurements are taken. If a test is in a multiple choice format, there is a significant statistical probability that a correct answer might be the result of a random guess. Also, in many situations, having the student simply provide a final answer can obscure many important elements. For example, on a math test, having the student simply provide a numeric answer (rather than showing the methodology) may result in a complete mismatch between the process and the result. It is hard to determine which is worse: The student who makes a simple arithmetric error at one step of a long process (resulting in a wrong answer) or The student who (without providing the "workflow") uses a completely invalid approach, yet still comes up with the right number.

The "Wrong Process"/"Right Answer" is probably the single biggest problem in software development. Even very simple items can suffer from this. As an example consider the following code for a "straight line" calculation....Is it correct? (for Integral Points)

int Solve(int m, int b, int x) { return m * x + b; }

Most people would respond "Yes". But let's take the question one step further... Is it correct for all possible values of m,b,x??? (no fair if you cheated by being focused on the bolded text!) Without additional information regarding constrains on "the possible values of m,b,x" the answer must be NO, there is the risk of overflow/wraparound that will produce an incorrect result!

To properly answer this question (i.e. Test the Code), one MUST be able to backtrack from the implementation through the design, and architecture all the way back to the requirements. And the requirement itself must be tested against the stakeholder(s). It is only when the bounding conditions are defined that it is possible to determine if the code is "Correct" and has "Quality".

Yet, how many of us (myself included) have written such code without even thinking about it. In many canses we (think we) "know" what the bounds are, and that the code will be correct. As we all know, requirements change, "code reuse" causes implementations to be applied to different scenarios, etc. This leads directly to the types of system failures that plague so many projects.

This approach to TDD is much more holistic than ones which start by focusing on the details. The fundamental concepts still apply:

Each item should be tested.
The test should be defined/implemented before (or concurrent with) the definition/implementation of the actual item.

We also add concepts that expand the scope and alter the style by recognizing:

There are many things beside "lines of code" that benefit from testing (measuring/evaluating in a formal way)
Correctness and Quality can not be solely measured by "correct results"

In the future parts, we will examine in greater detail some of the techniques that can be applied to each of these areas....

Think you write good code????

David V. Corbin — Thu, 03 Dec 2009 15:18:20 GMT

I will formally announce that I have written some of the baddest [actually meaning worse, and not the venaculare where bad is really good] code in the universe.

How do I know this? Because I have carefully studied the code I have written over the past 37 years (yes, since Sept 1972) and analyzed how it has fared over time.

While there are only a few true "WTFs" [Worse Than Failure] , the majority of code in this category is code that I thought was excellent at the time of original implementation (and was even reviewed by peers). Yet, over time it was proven that an alternate approach (which was considered and disarded) would have proven better in the long run.

This is the "silent killer" of the software development world. Code that looks great, performs great, and yet still manages to incur higher costs over the entire lifecycle.

Do I have an answer for this? Unfortunately, "Not Really"! Yet I am 100% convinced that code in this category costs the industry significantly more than truely "bad" code.

The first step to addressing this issue is the implementation of a rigorous process where "issues" are tracked, with a focus on determining the original conditions where the decision to choose a certain implementation was made.

I cahallenge all software developers to take a hard look at this aspect of their careers. Not only "What worked well over time?", "WHY did it work out well?" but to also carefully look for the items where the code proved to be problematic over time.

Were decisions made..

To Early? (research "Big Requirements Up Front")
To Late? (are you abusing "agile" development processes?)
With Insufficient Information? (how do you research and analyze requirements?)
Ingoring Inevitable Changes? (do you make assumptions about invaraibility?)
Without sufficient Skils? (what is your approach to learning new techniques?)
In too much of a Rush? (how much impact analysis is taken into account when developing a schedule?)

There are no simple answers, but I have consistently found that when a developer or team carefully considers the above points, and has the support of management to make the necessary changes in process the cost of software development always decreases.

So spend a few minutes today and look back at the code you have developed. Ask yourself if when viewed from todqay's perspective you still think you write good code....

Sorry Johnny - There is NO Garbage Collection in .NET

David V. Corbin — Fri, 25 Sep 2009 00:46:22 GMT

[Originally Published Apr 2004 - Updated October 2009]

Everyone "understands" that Microsoft's .NET and the CLR is a "garbage collector" based environment; but is it really.

First we must establish what is meant be "garbage" in this context. When an object is created there is (typically) one reference by which it can be accessed (the return value of "new"). While the program executes, there may be other references established to the same item; and established references may terminate. When an object can no longer be referenced, it is deemed to be "Garbage". [note: This is a bit of a simplification but will satisify out needs]

Next we must look at the definition of "collection", Websters dictionary offers the following:

collection: the act or process of collecting.
colllect: to bring together into one body or place.

Now lets look at what happens when a "GC.Collect" occurs.... (For simplicity we will look at generation 0, and ignore the impact of "pinned" objects). The object graph is "walked" starting at the rooted references, and any reachable item that is in Generation 0 is marked. When the walk is complete, the live objects are moved to the Gen1 heap, and the Gen0 heap reset back to the beginning. The result is that the memory occupied by all of the previous Gen0 residents is now available.

This reveals the fundamental problem with calling this process "garbage collection". Absolutely NOTHING is done with the garbage. Specifically there are no operations which involve moving the garbage so it is "brought together in one place".

To see what a "real" garbage collection is, consider an anology. In ones house, there are likely to be multiple wastebaskets; one in the kitchen, one in the bathroom, and other scattered throughout the residence. On trash day (or earlier if the Wife has anything to say about the matter), one goes through the residence and collects all of the garbage from multiple locations, places it in one bag, and brings it outside to the rubbish container. The amount of work is dependant on the number of original locations of garbage, and the amount of garbage in each location. The amount of "precious" (non-garbage) item in the house has absolutely no bearing on the process or the effort it will involve.

But when we look at the .NET situation, the exact opposite is true. It is the number of LIVE objects that impacts the performance as these are what must be scanned and moved. It does not matter if there is a single small "garbage" object on the heap, or if there are tens of thousands (of varying sizes). Once the live (precious) objects have been moved out of harms way, it is a single, constant time operation to reset the heap to be ready to get new objects.

This shows that .NET implements a Live Object Preservation pattern, and NOT a grabage collection pattern.

While this entire post may seem like a "symantic quibble", it has serious ramifications when dealing with .NET architecture/design and implementation. In other environments there is NO overhead (aside from the actual memory) to keeping references to heap based object which will be needed (or even just possibly needed) later. In many cases, the cost of allocating [always higher in a conventional heap than in a CLR heap] and deleting (updating the freelist) far outwieghs the memory utilization issue, and so references are kept for an extended period of time.

When this approach is taken in a .NET application, these live objects represent a performance hit everytime (neglecting some optimizations) that the GC runs - simply because the GC deals with processing live objects. On the otherhand, allocating a (non-large) object in .NET is typically a simply pointer increment, and abandoning it (assuming no finalizer) is a 0 time issue.

Over the past few years, I have been involved with a number of projects where clients were complaining that ".NET was slow" and could not meet their perfomance demands. In the vast majority of cases, this was directly tracked to the implementation not having proper (for .NET) object lifetime management..

addendum: When one looks at environments such at C/C++, the conventional/standard implementation (pre C++0x) do not include "garbage collection". The heap is (typically, and simplified) implemented as a structure containing the "free blocks" of items that were previously deleted. This means that (a pointer to) memory that is not longer in use [i.e. garbage] IS actually MOVED. Each time there is a call to "delete" or "free(...)" there is a synchronous [i.e. it completes before delete/free() returns] collection of information about the garbage that occurs.

In .NET the large object heap [LOH] is used for items which exceed a threshold size [80,000 bytes]. This particular heap IS operated in a manner nearly identical to a C/C++, in that the "live" objects are NOT moved, and it is a set of references to the avilable memory (garbage) that is manipulated.

The CLR does not a Virtual Machine Make...

David V. Corbin — Fri, 25 Sep 2009 00:04:58 GMT

[Originally Written October 2004 - Updated September 2009]

Many people state that Microsoft .Net technology provides a "Virtual Machine" environment via the CLR. However, an examination of various definitions of Virtual Machine shows that this is not the best analogy.

For our first example definition, let us look no further than Microsoft's own site:

Virtual Machine: A software-implemented computer that emulates a complete hardware system in a self-contained, isolated software environment and runs its own operating system.

Clearly this does not apply so lets break down the parts of "a complete hardware system". The three major categories of devices that make up a system are: Memory (some type of storage), Processing, and Input/Output.

When a program written in any language uses the Microsoft Implementation of the CLR, it directly utilizes the actual memory presented by the underlying system, All processing is done using native instruction execution on the underlying processor, and all Input/Output is accomplished via the device (Drivers) provided by the underlying operating system.

So, while it is possible (and there are projects attempting to reach this goal) to implement the CLR as a Virtual Machine. It is clear that the current Microsoft implementation provides none of these features.

The feature that the CLR provides is that there is an intermediate stage where the source code has been reduced from the original form into a well defined set of intermediate instructions that are independant of any specific target environment. Additionally a rich library (the BCL) is provided for addressing many common constructs and providing additional abstractions over lower level functionallity.

But remember this intermediate code NEVER executes. It undergoes a second compilation phase to become pure native instructions.

This process actually has a long history. Long, long ago [1970's 1980's] it was quite common for high level language compilers to emit (either by default or as an option) assembly language SOURCE rather than object code. This output could then be copies to various target machines (with differences in capabilities) and run through the assembler (often with differing configurations or external linkages) to produce an executable that was specifically tailored to the targeted machine.

Although the mechanics are different this is completely analogous to what happens with a CLR based program.

The result is that while .Net (the CLR) does provide a level of abstraction from the actual executable code, it does not meet the criteria for a "Virtual Machine".

One mans view of the Microsoft MVP Program

David V. Corbin — Wed, 18 Mar 2009 02:22:18 GMT

As I indicated in a previous post, I was contacted by Newsday (a large regional newspaper from Long Island, New York) about what the MVP program meant for me as a small business owner. The original article is available at:

http://www.newsday.com/business/technology/one-of-microsoft-s-mvps-again-1.769775

update 26-Mar-2010: Newsday moved the article...link has been updated