How we’re going to use NuGet

This is a quick introduction to moving from our own internal assembly repository to using NuGet.  It’s terse for a reason, just a note to self and those I work with:

  1. What we had
    1. One source of builds external to our code repository
      1. Didn’t check in external builds
      2. Used DVCS to share single feed
    2. Rake task to update and copy latest builds local checkout (this location is excluded from VCS)
    3. Automatic updates if changed in central repository and version not incremented
      1. Caused problems when changes were breaking
      2. Sometimes people updated old versions, not realizing there were new versions
    4. Simple source of existing builds (easier to bring external libraries into a project)
  2. Why move to NuGet?
    1. So we don’t have to
      1. Maintain this tool
      2. Educate others to use it (easier to work with other teams)
      3. Maintain open source / 3rd party builds
    2. Easily add/remove/update external dependencies
      1. Keep up to date with open source / 3rd party builds (easier)
      2. Automatic versioning for every release (it reads the assembly version when building packages)
        1. Avoid automatic updates that break apps even for small changes
    3. Automatic dependency conflict resolution (if possible)
      1. We can update a base dependency with a non breaking change and not need to recompile all child dependencies of it.
    4. Multiple sources/feeds of packages
      1. Can segment internal feed versus customer specific feeds
      2. Decentralize feeds!
      3. Composite feeds, we can override what is in one feed with what is in another!
    5. Update checks if it’s already updated first, instead of copying every time, saves time on builds
    6. Distribute content as well as binaries
      1. Images
      2. Css
      3. Javascript
      4. etc
    7. Compressed builds (zip) will decrease the size of our internal feed(s)
    8. Easier to publish builds of our own packages to the open source community
    9. Simplified updating of all references of a dependency in one power shell command
  3. How
    1. Setup
      1. Links
        1. http://nuget.codeplex.com/
        2. http://nuget.org/
        3. http://docs.nuget.org/docs/reference/package-manager-console-powershell-reference
      2. Get 1.4 build of NuGet command line (fixes multiple feeds for CLI installer)
      3. Install NuGet Package Manager in Visual Studio (Extension Manager)
        1. http://visualstudiogallery.msdn.microsoft.com/27077b70-9dad-4c64-adcf-c7cf6bc9970c
      4. Install Package Explorer (optional) – view & edit packages
        1. http://nuget.codeplex.com/releases
      5. http://nuget.org/list/packages/nugetpackageupdater (optional) – update all, this is a feature added in 1.4 I believe
    2. Configuration
      1. Setup any local feeds via the Extension Manager in Visual Studio
    3. Rake tasks
      1. rake dep
        1. Updates dependencies if missing (so we don’t have to check them in)
        2. Run before builds
        3. Example:

          desc "Setup dependencies for nuget packages"
          task :dep do
              package_folder = File.expand_path('Packages')
              packages = FileList["**/packages.config"].map{|f| File.expand_path(f)}
              packages.each do |file|
                  sh %Q{nuget install #{file} /OutputDirectory #{package_folder}}
              end
          end

      2. rake nuget
        1. Used to build and deploy a package to a feed
        2. Use env variable to point to a dev feed source NuGetDevFeed
          1. All builds go here and can be copied to other official sources once verified
        3. Example:

          task :nuget => [:build] do 
              sh "nuget pack src\\BclExtensionMethods\\BclExtensionMethods.csproj /OutputDirectory " + ENV[‘NuGetDevFeed’]
          end

    4. Package Manager Console

      1. Use this to find, install, update and manage packages, works very rapidly to update references in a project as needed and to help modify config files for assembly bindings.

  4. Problems I encountered

    1. NuGet CLI is buggy with relative paths for package sources, use absolute paths

    2. Local feeds used to be able to segment different packages by folder, but the latest version doesn’t work with nested folders in the local (disk) feed.  So for now just dump the output right in the same folder as all other packages.

    3. Reactive team decided to pull Rx-Main newer version because they were building experimental and stable branches under the same package name.  This caused a bit of a headache.  To fix this I had to manually edit packages.config and remove the ref to Rx-Main and reinstall the older version.

      1. Also, had to nuke the local copy of the Rx-Main newer version from the packages folder as it would check there first.

      2. In the future it might be better to just copy the local version to my own NuGet local feed but in this case I decided I didn’t want the experimental code so I rolled back versions.

      3. Might be nice if NuGet had support to downgrade versions.

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati
Refactoring Part II - Tight rope walking / what can break and KISS

Like it or not, we humans make mistakes.  If we embrace the fact that we are going to make mistakes, we can direct our efforts to reduce mistakes in areas that are critical in exchange for potentially making more mistakes in areas that aren’t.  Gasp!  We need to get over the silly notion that our work can ever be 100% perfect and try to maximize in the areas that matter.

Does it really matter?

These are the things I’ve found that typically don’t matter as much in the grand scheme of development.  Start training yourself to identify areas that matter!

  • Infrequently used features
    • Especially if there’s an easy workaround during a failure
  • Administrative crud pages
    • Especially in smaller apps, the developer is usually the admin and can just hack at the DB in a failure
  • MVC Controllers
  • Logging
    • This just means debugging will be a bit harder and I’m sure we’ll fix it quick enough.
  • User management / authentication
    • Development typically involves logging in daily, so it’s likely we’ll catch the mistakes.
    • Please just use OpenId already or another common alternative to rolling your own.
    • If no one can login, deploying a fix won’t interrupt anyone!
  • Easily fixed/deployed features
    • Any feature that isn’t critical, that can easily be fixed and deployed.
  • CSS and images
    • How many of the things we do with CSS and images are just for aesthetic purposes and really don’t matter if the application still functions?
    • Do I really care if my bank sends me a statement and their logo shows up a red X?
  • Reports versus entry
    • If we allow a change (like a balance transfer) to occur that is invalid, it’s probably a bigger problem than if we accidently show an invalid balance total on a statement report.  This is highly subjective, but I’m more worried about data going into my systems than data coming out, except where data flowing out flows into other systems.
  • Features that are no longer used / should be removed
  • Areas where testing seems to slow down development
    • IMHO, testing typically slows down development in areas that don’t matter (stubbing controllers/services, duplicated/overlapping test cases, KISS code)  In areas that are important, we typically find complexity, and testing often helps avoid bugs in complexity faster than F5 debugging.

KISS

In the areas that don’t matter, we should strive for simplicity.  Readable and simplified code is less likely to contain mistakes.  Controllers can be thin instead of fat.  Reports can isolate complexity in a tested model or functional components of reuse (calculations). 

What does this mean?

So we know what doesn’t matter as much, what does that mean?  For me it means less testing:

  • Not writing automated tests (or very few) … integration or unit
  • Not double/triple checking my work
  • Sometimes compiling is good enough
  • Sometimes a quick run of a common scenario on a related page, but not all scenarios
  • Rarely, the occasional bat shit crazy refactoring with little regard to these areas.

 

Conclusions

Some of this may sound radical, if so, don’t adopt it.  I can refactor a lot faster if I know where I can run versus where I should crawl.  Always crawling (extra effort is expended upfront) is as reckless as always running (extra effort is expended after the fact), an optimum is usually found in balancing what can and what shouldn’t break.

Sadly, I’ve seen a lot of hugely beneficial refactoring passed up simply because it would be invasive to areas that ironically aren’t that important in the grand scheme of things.

Happy Coding!

-Wes

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati
Things I've noticed with DVCS

 

Things I encourage:


Frequent local commits

This way you don't have to be bothered by changes others are making to the central repository while working on a handful of related tasks.  It's a good idea to try to work on one task at a time and commit all changes at partitioned stopping points.  A local commit doesn't have to build, just FYI, so a stopping point doesn't mean a build point nor a point that you can push centrally.  There should be several of these in any given day.  2 hours is a good indicator that you might not be leveraging the power of frequent local commits.  Once you have verified a set of changes works, save them away, otherwise run the risk of introducing bugs into it when working on the next task. 

The notion of a task

By task I mean a related set of changes that can be completed in a few hours or less.  In the same token don’t make your tasks so small that critically related changes aren’t grouped together.  Use your intuition and the rest of these principles and I think you will find what is comfortable for you.

Partial commits

Sometimes one task explodes or unknowingly encompasses other tasks, at this point, try to get to a stopping point on part of the work you are doing and commit it so you can get that out of the way to focus on the remainder.  This will often entail committing part of the work and continuing on the rest.

Outstanding changes as a guide

If you don't commit often it might mean you are not leveraging your version control history to help guide your work.  It's a great way to see what has changed and might be causing problems.  The longer you wait, the more that has changed and the harder it is to test/debug what your changes are doing! This is a reason why I am so picky about my VCS tools on the client side and why I talk a lot about the quality of a diff tool and the ability to integrate that with a simple view of everything that has changed.  This is why I love using TortoiseHg and SmartGit: they show changed files, a diff (or two way diff with SmartGit) of the current selected file and a commit message all in one window that I keep maximized on one monitor at all times.

Throw away / stash commits

There is extreme value in being able to throw away a commit (or stash it) that is getting out of hand.  If you do not commit often you will have to isolate the work you want to commit from the work you want to throw away, which is wasted productivity and highly prone to errors.  I find myself doing this about once a week, especially when doing exploratory re-factoring.  It's much easier if I can just revert all outstanding changes.

Sync with the central repository daily

The rest of us depend on your changes.  Don't let them sit on your computer longer than they have to.  Waiting increases the chances of merge conflict which just decreases productivity.  It also prohibits us from doing deploys when people say they are done but have not merged centrally.  This should be done daily!  Find a way to partition the work you are doing so that you can sync at least once daily.


Things I discourage:

Lots of partial commits right at the end of a series of changes

If you notice lots of partial commits at the end of a set of changes, it's likely because you weren't frequently committing, nor were you watching for the size of the task expanding beyond a single commit.  Chances are this cost you productivity if you use your outstanding changes as a guide, since you would have an ever growing list of changes.


Committing single files

Committing single files means you waited too long and no longer understand all the changes involved.  It may mean there were overlapping changes in single files that cannot be isolated.  In either case, go back to the suggestions above to avoid this. 

Committing frequently does not mean committing frequently right at the end of a day's work.

It should be spaced out over the course of several tasks, not all at the end in a 5 minute window.

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati
Refactoring Part 1 : Intuitive Investments

Fear, it’s what turns maintaining applications into a nightmare.  Technology moves on, teams move on, someone is left to operate the application, what was green is now perceived brown.  Eventually the business will evolve and changes will need to be made.  The approach to those changes often dictates the long term viability of the application.  Fear of change, lack of passion and a lack of interest in understanding the domain often leads to a paranoia to do anything that doesn’t involve duct tape and bailing twine.  Don’t get me wrong, those have a place in the short term viability of a project but they don’t have a place in the long term.  Add to it “us versus them” in regards to the original team and those that maintain it, internal politics and other factors and you have a recipe for disaster.  This results in code that quickly becomes unmanageable.  Even the most clever of designs will eventually become sub optimal and debt will amount that exponentially makes changes difficult. 

This is where refactoring comes in, and it’s something I’m very passionate about.  Refactoring is about improving the process whereby we make change, it’s an exponential investment in the process of change. Without it we will incur exponential complexity that halts productivity. Investments, especially in the long term, require intuition and reflection. 

How can we tackle new development effectively via evolving the original design and paying off debt that has been incurred?

The longer we wait to ask and answer this question, the more it will cost us.  Small requests don’t warrant big changes, but realizing when changes now will pay off in the long term, and especially in the short term, is valuable.

I have done my fair share of maintaining applications and continuously refactoring as needed, but recently I’ve begun work on a project that hasn’t had much debt, if any, paid down in years.  This is the first in a series of blog posts to try to capture the process which is largely driven by intuition of smaller refactorings from other projects.

Signs that refactoring could help:

Testability

  • How can decreasing test time not pay dividends?
  • One of the first things I found was that a very important piece often takes 30+ minutes to test.  I can only imagine how much time this has cost historically, but more importantly the time it might cost in the coming weeks: I estimate at least 10-20 hours per person!  This is simply unacceptable for almost any situation.  As it turns out, about 6 hours of working with this part of the application and I was able to cut the time down to under 30 seconds!  In less than the lost time of one week, I was able to fix the problem for all future weeks!
  • If we can’t test fast then we can’t change fast, nor with confidence.
  • Code is used by end users and it’s also used by developers, consider your own needs in terms of the code base.  Adding logic to enable/disable features during testing can help decouple parts of an application and lead to massive improvements.  What exactly is so wrong about test code in real code?  Often, these become features for operators and sometimes end users. 
  • If you cannot run an integration test within a test runner in your IDE, it’s time to refactor.

Readability

  • Are variables named meaningfully via a ubiquitous language?
  • Is the code segmented functionally or behaviorally so as to minimize the complexity of any one area?
  • Are aspects properly segmented to avoid confusion (security, logging, transactions, translations, dependency management etc)
  • Is the code declarative (what) or imperative (how)?  What matters, not how.  LINQ is a great abstraction of the what, not how, of collection manipulation.  The Reactive framework is a great example of the what, not how, of managing streams of data.
  • Are constants abstracted and named, or are they just inline?
  • Do people constantly bitch about the code/design?
  • If the code is hard to understand, it will be hard to change with confidence.  It’s a large undertaking if the original designers didn’t pay much attention to readability and as such will never be done to “completion.”  Make sure not to go over board, instead use this as you change an application, not in lieu of changes (like with testability).

Complexity

  • Simplicity will never be achieved, it’s highly subjective.  That said, a lot of code can be significantly simplified, tidy it up as you go. 
  • Refactoring will often converge upon a simplification step after enough time, keep an eye out for this.

Understandability

  • In the process of changing code, one often gains a better understanding of it.  Refactoring code is a good way to learn how it works.  However, it’s usually best in combination with other reasons, in effect killing two birds with one stone.  Often this is done when readability is poor, in which case understandability is usually poor as well.  In the large undertaking we are making with this legacy application, we will be replacing it.  Therefore, understanding all of its features is important and this refactoring technique will come in very handy.

Unused code

  • How can deleting things not help?
  • This is a freebie in refactoring, it’s very easy to detect with modern tools, especially in statically typed languages.  We have VCS for a reason, if in doubt, delete it out (ok that was cheesy)!
  • If you don’t know where to start when refactoring, this is an excellent starting point!

Duplication

  • Do not pray and sacrifice to the anti-duplication gods, there are excellent examples where consolidated code is a horrible idea, usually with divergent domains.  That said, mediocre developers live by copy/paste.  Other times features converge and aren’t combined.  Tools for finding similar code are great in the example of copy/paste problems.  Knowledge of the domain helps identify convergent concepts that often lead to convergent solutions and will give intuition for where to look for conceptual repetition.

80/20 and the Boy Scouts

  • It’s often said that 80% of the time 20% of the application is used most.  These tend to be the parts that are changed.  There are also parts of the code where 80% of the time is spent changing 20% (probably for all the refactoring smells above).  I focus on these areas any time I make a change and follow the philosophy of the Boy Scout in cleaning up more than I messed up.  If I spend 2 hours changing an application, in the 20%, I’ll always spend at least 15 minutes cleaning it or nearby areas.
  • This gives a huge productivity edge on developers that don’t.
  • Ironically after a short period of time the 20% shrinks enough that we don’t have to spend 80% of our time there and can move on to other areas.

 

Refactoring is highly subjective, never attempt to refactor to completion!  Learn to be comfortable with leaving one part of the application in a better state than others.  It’s an evolution, not a revolution.  These are some simple areas to look into when making changes and can help get one started in the process.  I’ve often found that refactoring is a convergent process towards simplicity that sometimes spans a few hours but often can lead to massive simplifications over the timespan of weeks and months of regular development.

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati
What we are doing with HtmlTags Part 2 : Form Fields

Most simple forms we write, especially in LOB applications, are repetitive sections of inputs and/or displays.  Take a look at an exmaple from the MVC Music Store sample, the album editor template, notice anything repetitive?

<%@ Import Namespace="MvcMusicStore"%>

<%@ Control Language="C#" Inherits="System.Web.Mvc.ViewUserControl<MvcMusicStore.Models.Album>" %>

<script src="/Scripts/MicrosoftAjax.js" type="text/javascript"></script>
<script src="/Scripts/MicrosoftMvcAjax.js" type="text/javascript"></script>
<script src="/Scripts/MicrosoftMvcValidation.js" type="text/javascript"></script>

<p>
    <%: Html.LabelFor(model => model.Title)%>
    <%: Html.TextBoxFor(model => model.Title)%>
    <%: Html.ValidationMessageFor(model => model.Title)%>
</p>
<p>
    <%: Html.LabelFor(model => model.Price)%>
    <%: Html.TextBoxFor(model => model.Price)%>
    <%: Html.ValidationMessageFor(model => model.Price)%>
</p>
<p>
    <%: Html.LabelFor(model => model.AlbumArtUrl)%>
    <%: Html.TextBoxFor(model => model.AlbumArtUrl)%>
    <%: Html.ValidationMessageFor(model => model.AlbumArtUrl)%>
</p>
<p>
    <%: Html.LabelFor(model => model.Artist)%>
    <%: Html.DropDownList("ArtistId", new SelectList(ViewData["Artists"] as IEnumerable, "ArtistId", "Name", Model.ArtistId))%>
</p>
<p>
    <%: Html.LabelFor(model => model.Genre)%>
    <%: Html.DropDownList("GenreId", new SelectList(ViewData["Genres"] as IEnumerable, "GenreId", "Name", Model.GenreId))%>
</p>

It didn’t take me long to notice that the duplication was driving me absolutely crazy! Yet, the community is hailing these template helpers as the next best thing since web forms and for the life of me I have no idea why! Why stop at the abstraction of DisplayFor, InputFor and LabelFor?  Our views were suffering the same problem, even with a spark and HtmlTags spin:

<div class="fields">
	!{Html.LabelFor(m => m.Quantity).AddClass("label")}
	!{Html.InputFor(m => m.Quantity).AddClass("field")}
</div>

<div class="fields">
	!{Html.LabelFor(m => m.Price).AddClass("label")}
	!{Html.InputFor(m => m.Price).AddClass("field")}
</div>

Template Convention

Both frameworks (HtmlTags or MVC2 Templates) can be extended to support the abstraction we produced, but HtmlTags is geared to do this in a much cleaner and flexible fashion.  We created the idea of an edit template that includes the label and input (and validation if you so desire), all of which gets nested in some container tag.  We rolled all of this up into two conventions: EditTemplateFor and DisplayTemplateFor.  The difference being whether or not the field on the form is editable.  I’ll admit that we could probably roll this up further into a TemplateFor convention that has configurable builders much like DisplayFor/InputFor/LabelFor in HtmlTags, but right now these are static conventions in that they don’t have hot swappable builders.  That is an effort for another day :)

So here is what the views would look like now (sigh of relief):

<%: Html.EditTemplateFor(m => m.Title) %>
<%: Html.EditTemplateFor(m => m.Price) %>
<%: Html.EditTemplateFor(m => m.AlbumArtUrl)%>
<%: Html.EditTemplateFor(m => m.Artist)%>
<%: Html.EditTemplateFor(m => m.Genre)%>

!{Html.EditTemplateFor(m => m.Quantity)}
!{Html.EditTemplateFor(m => m.Price)}

How it works

The following is the EditeTemplateFor convention, the display version only varies by calling DisplayFor instead of InputFor. We simply build our label, adding a class that is statically configurable. Then we take a list of fields, in the event we have more than one falling under a single label, and for each we wrap it in a div with the class "field" and then we put the label and inputs into a div with the class "fields."
public static class TemplateHelpers
{
	public static string FieldDivClass = "field";
	public static string FieldsDivClass = "fields";
	public static string LabelClass = "label";

	public static HtmlTag EditTemplateFor<T>(this T model, params Expression<Func<T, object>>[] fieldSelectors) where T : class
	{
		Contract.Requires(fieldSelectors.Any());

		var label = model.LabelFor(fieldSelectors[0]).AddClass(LabelClass);

		var fields = fieldSelectors
			.Select(f => Tags.Div
							.AddClass(FieldDivClass)
							.Nest(model.InputFor(f)));

		return Tags.Div
			.AddClass(FieldsDivClass)
			.Nest(label)
			.Nest(fields.ToArray());			
	}

	public static HtmlTag EditTemplateFor<T>(this HtmlHelper<T> helper,
											 params Expression<Func<T, object>>[] fieldSelectors) where T : class
	{
		return EditTemplateFor(helper.ViewData.Model, fieldSelectors);
	}
Here is an alteration of the above to work with the MVC Music store ablum editor:
public static class TemplateHelpers
{
	public static string FieldDivClass = "field";
	public static string FieldsDivClass = "fields";
	public static string LabelClass = "label";
	public static string ValidationMessageClass = "validation-message";

	public static HtmlTag EditTemplateFor<T>(this HtmlHelper helper, Expression<Func<T, object>> field) where T : class		{
		var label = helper.LabelFor(field);
		var input = helper.InputFor(field);
		var validation = Tags.Span.AddClass(ValidationMessageClass)

		return Tags.Paragraph
			.Nest(label,field,validation);			
	}
}

Reproducing this with MVC2 Templates

	public static MvcHtmlString EditTemplateFor<T>(this HtmlHelper<T> helper, Expression<Func<T, object>> field) where T : class
	{
		var label = helper.LabelFor(field);
		var input = helper.EditorFor(field);
		var validation = helper.ValidationMessageFor(field);

		var template = String.Format("<p>{0}{1}{2}</p>", label, input, validation);
		return MvcHtmlString.Create(template);
	}
	
	public static MvcHtmlString EditTemplateFor2<T>(this HtmlHelper<T> helper, Expression<Func<T, object>> field) where T : class
	{
		var label = helper.LabelFor(field);
		var input = helper.EditorFor(field);

		var template = String.Format("<div class=\"fields\">{0}<div class=\"field\">{1}</div></div>", label, input);
		return MvcHtmlString.Create(template);
	}

The Verdict

This was my best attempt at producing the same behavior with ASP.Net MVC templates, and it's already run into some serious issues.

  1. I have to work with strings.
    1. Html is not a string, so why should I have to work with it as such.
    2. The MVC2 template return an MvcHtmlString unlike HtmlTags which returns HtmlTag
    3. I could improve this with an HtmlTextWriter but that’s not part of the template abstraction and is yet another burden to include.  HtmlTags supports a model OOB!
  2. I cannot modify my label to add the class “label”. Instead, I would have to wrap the label with another tag to get a class applied or I would have to alter my label template and then that class would be applied anytime I used a label in my entire application, not just in a form field.
  3. The HtmlTags version is much more readable, maintainable and representative of the structure of the html.
  4. I cannot plug this into the scaffolding in ASP.Net MVC with EditorForModel, this is the failure in their approach to scaffolding, since it’s built on an unchangeable model with no configuration and modification, it only works for the narrow out of the box case (smells of web forms days if you ask me).  In a subsequent blog post I’ll show how we can continue to build up, instead of top down, to produce flexible scaffolding on top of HtmlTags!
  5. I can literally “unit” test the HtmlTags version (if it added value to test it) without spinning up a view engine to render the result, and I don’t have to parse html!

-Wes

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati
What we are doing with HtmlTags Part 1 : Why HtmlTags?

Back in January, Jeremy Miller posted a nice article on HtmlTags: Shrink your Views with FubuMVC Html Conventions.  We were immediately in love with the idea and have spent several months adapting the conventions to work with our ASP.Net MVC applications.  I was having a conversation with Ryan recently, reflecting on how far we’ve come and how we had no vision of that when we first read that article.  I want to share some of that, so I will be working on a series of blog posts to show “What we are doing with HtmlTags.”  But first: why?

A lot of buzz has been generated around reusability and views, particularly html.  Html being a highly compositional language, but lacking any ability for reuse, has led to copy paste hell.  Several different philosophies are employed in View Engines (VE) to try to address this issue and recently MVC2 was released with it’s templated helpers.  The problem with reusing html directly is that lacks the benefits of separating the concern of what we need from the html (building it) and generating it, classically, the difference between “What” I want versus “How” I get it.  The power of the Linq abstraction shows the benefits of separating “What” versus “How.”  HtmlTags is another model based approach that separates the “What” and “How.”  It goes even further by allowing the “How” to be configured via conventions.  Here is a list of the benefits this separation has brought to our team:

  1. View engine agnostic.  HtmlTags is not coupled to any particular VE.  It can even be returned from a controller action and rendered directly to the response!
  2. Testability with out spinning up a costly integration environment, I can write assertions directly against the model! 
    1. Html is a language that lacks the ability to query it in any reasonable fashion.
    2. We integrate security as an aspect to our conventions that build our Html.  Now, we can write a simple test to ensure that the security shows/hides elements in the model by simply asserting the lack of their presence!  Try parsing html to do this after you spin up an integration test to access the view output!
  3. Translation: HtmlTags is a model that is easily traversed.  We leverage this to transform the html model into an export model that can then be translated to csv, pdf and other formats.  Now we get exports for free on any view result simply by writing the guts of the view in HtmlTags!
  4. The ability to translate from our data model to the html model directly, in c#, instead of using a view engine with a mish mash of concerns (html & code). 
    1. This avoids reshaping of our data to get it to easily work with our view engine.  Instead we can translate directly, on our view models, and have the full power of c# at our finger tips.
    2. This also helps with refactoring, where the story is still very poor with VE content.
    3. Simply put: avoids the html / code disparity
  5. Composable conventions, we build upon the HtmlTags DisplayFor, LabelFor and InputFor to build up higher level concepts in our application, like a form field (composed of fields for a label, input, validation etc).  This gives us a on stop spot to restructure the html for every form in our application, no more shotgun surgery on views! 
    1. We also leverage concepts around menus, buttons, forms etc.
  6. Reusable across projects: we don’t have to try to copy/paste a bunch of templates, we simply drop in a dll with default conventions and configure from there.
  7. Avoid fat fingering html tag names, attributes etc.  We don’t have to think as much about html anymore, amazing concept!  Templated html still has issues with fat fingering as <div> is all over whereas we have a nice Tags.Div static entry point to generate a div tag.  We do this for all tags we use.  This is the concept of reuse between templates that is rather lacking from MVC2 templates without bee sting hell.  This is a great example of leveraging a statically typed language to fail fast, we don’t have to render a few to find out if we fat fingered some html.  All html tags, attributes etc are kept as constants collections.
  8. Avoid hard coded css classes: we make our class names configurable and put them right with the builder so any project can override them.  Try doing that without a bunch of bee stings in a templated helper and a nightmare of classes coupled to the template to provide the constants in code.  This reminds me of code behind hell from web forms days.  Then try reusing that across projects.
    1. I’ve seen some really scary stuff in samples with templated helpers where people try to access attributes. property types, etc and conditionally render sections.  The only alternative is to put that code into helpers that are separated from the view and lead to a coupling mess when trying to reuse the templates across projects.

 

The next set of posts will cover examples of how we are using HtmlTags and how it’s paying dividends.

 

-Wes

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati
How I do VCS

After years of dabbling with different version control systems and techniques, I wanted to share some of what I like and dislike in a few blog posts.  To start this out, I want to talk about how I use VCS in a team environment.  These come in a series of tips or best practices that I try to follow. 

Note: This list is subject to change in the future.

  1. Always use some form of version control for all aspects of software development.
    1. Development is an evolution.  Looking back at where we were is an invaluable asset in that process.  This includes data schemas and documentation.
    2. Reverting / reapplying changes is absolutely critical for efficient development.
    3. The tools I use:
      1. Code: Hg (preferred), SVN
      2. Database: TSqlMigrations
      3. Documents: Sometimes in code repository, also SharePoint with versioning
  2. Always tag a commit (changeset) with comments
    1. This is a quick way to describe to someone else (or your future self) what the changeset entails.
    2. Be brief but courteous.
    3. One or two sentences about the task, not the actual changes.
    4. Use precommit hooks or setup the central repository to reject changes without comments.
  3. Link changesets to documentation
    1. If your project management system integrates with version control, or has a way to externally reference stories, tasks etc then leave a reference in the commit.  This helps locate more information about the commit and/or related changesets.
    2. It’s best to have a precommit hook or system that requires this information, otherwise it’s easy to forget.
  4. Ability to work offline is required, including commits and history
    1. Yes this requires a DVCS locally but doesn’t require the central repository to be a DVCS.  I prefer to use either Git or Hg but if it isn’t possible to migrate the central repository, it’s still possible for a developer to push / pull changes to that repository from a local Hg or Git repository.
  5. Never lock resources (files) in a central repository… Rude!
    1. We have merge tools for a reason, merging sucked a long time ago, it doesn’t anymore… stop locking files!
    2. This is unproductive, rude and annoying to other team members.
  6. Always review everything in your commit.

    1. Never ever commit a set of files without reviewing the changes in each.
    2. Never add a file without asking yourself, deep down inside, does this belong?
    3. If you leave to make changes during a review, start the review over when you come back.  Never assume you didn’t touch a file, double check.
      1. This is another reason why you want to avoid large, infrequent commits.
    4. Requirements for tools
      1. Quickly show pending changes for the entire repository.
      2. Default action for a resource with pending changes is a diff.
      3. Pluggable diff & merge tool
      4. Produce a unified diff or a diff of all changes.  This is helpful to bulk review changes instead of opening each file.
    5. The central repository is not your own personal dump yard.  Breaking this rule is a sure fire way to get the F bomb dropped in front of your name, multiple times.
    6. If you turn on Visual Studio’s commit on closing studio option, I will personally break your fingers.
      1. By the way, the person(s) in charge of this feature should be fired and never be allowed near programming, ever again.
  7. Commit (integrate) to the central repository / branch frequently
    1. I try to do this before leaving each day, especially without a DVCS.  One never knows when they might need to work from remote the following day.
  8. Never commit commented out code
    1. If it isn’t needed anymore, delete it!
    2. If you aren’t sure if it might be useful in the future, delete it!

      This is why we have history.

    3. If you don’t know why it’s commented out, figure it out and then either uncomment it or delete it.
  9. Don’t commit build artifacts, user preferences and temporary files.
    1. Build artifacts do not belong in VCS, everything in them is present in the code. (ie: bin\*, obj\*, *.dll, *.exe)
    2. User preferences are your settings, stop overriding my preferences files! (ie: *.suo and *.user files)
    3. Most tools allow you to ignore certain files and Hg/Git allow you to version this as an ignore file.  Set this up as a first step when creating a new repository!
  10. Be polite when merging unresolved conflicts.
    1. Count to 10, cuss, grab a stress ball and realize it’s not a big deal.  Actually, it’s an opportunity to let you know that someone else is working in the same area and you might want to communicate with them.
    2. Following the other rules, especially committing frequently, will reduce the likelihood of this.
    3. Suck it up, we all have to deal with this unintended consequence at times.  Just be careful and GET FAMILIAR with your merge tool.  It’s really not as scary as you think.  I personally prefer KDiff3 as its merging capabilities rock.
    4. Don’t blindly merge and then blindly commit your changes, this is rude and unprofessional.  Make sure you understand why the conflict occurred and which parts of the code you want to keep. 
    5. Apply scrutiny when you commit a manual merge: review the diff!
    6. Make sure you test the changes (build and run automated tests)
  11. Become intimate with your version control system and the tools you use with it.
    1. Avoid trial and error as much as is possible, sit down and test the tool out, read some tutorials etc.  Create test repositories and walk through common scenarios.
    2. Find the most efficient way to do your work.  These tools will be used repetitively, so inefficiencies will add up. Sometimes this involves a mix of tools, both GUI and CLI.
      1. I like a combination of both Tortoise Hg and hg cli to get the job efficiently.
  12. Always tag releases
    1. Create a way to find a given release, whether this be in comments or an explicit tag / branch.  This should be readily discoverable.
    2. Create release branches to patch bugs and then merge the changes back to other development branch(es).
  13. If using feature branches, strive for periodic integrations.
    1. Feature branches often cause forked code that becomes irreconcilable.  Strive to re-integrate somewhat frequently with the branch this code will ultimately be merged into.  This will avoid merge conflicts in the future.
    2. Feature branches are best when they are mutually exclusive of active development in other branches.
  14. Use and abuse local commits

    ,
    at least one per task in a story.
    1. This builds a trail of changes in your local repository that can be pushed to a central repository when the story is complete.
  15. Never commit a broken build or failing tests to the central repository.
    1. It’s ok for a local commit to break the build and/or tests.  In fact, I encourage this if it helps group the changes more logically.  This is one of the main reasons I got excited about DVCS, when I wanted more than one changeset for a set of pending changes but some files could be grouped into both changesets (like solution file / project file changes).
  16. If you have more than a dozen outstanding changed resources, there should probably be more than one commit involved.
    1. Exceptions when maintaining code bases that require shotgun surgery, in this case, it’s a design smell :)
  17. Don’t version sensitive information
    1. Especially usernames / passwords

 

There is one area I haven’t found a solution I like yet: versioning 3rd party libraries and/or code.  I really dislike keeping any assemblies in the repository, but seems to be a common practice for external libraries.  Please feel free to share your ideas about this below. 

 

-Wes

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati
Adopting DBVCS

Identify early adopters

Pick a small project with a small(ish) team.  This can be a legacy application or a green-field application. Strive to find a team of early adopters that will be eager to try something new. Get the team on board!

Research

Research the tool(s) that you want to use.  Some tools provide all of the features you would need while some only provide a slice of the pie.  DBVCS requires the ability to manage a set of change scripts that update a database from one version to the next.  Ideally a tool can track database versions and automatically apply updates.  The change script generation process can be manual, but having diff tools available to automatically generate it can really reduce the overhead to adoption.  Finally, an automated tool to generate a script file per database object is an added bonus as your version control system can quickly identify what was changed in a commit (add/del/modify), just like with code changes.

Don’t settle on just one tool, identify several.  Then work with the team to evaluate the tools.  Have the team do some tests of the following scenarios with each tool:

  1. Baseline an existing database: can the migration tool work with legacy databases?  Caution: most migration platforms do not support baselines or have poor support, especially the fad of fluent APIs.
  2. Add/drop tables
  3. Add/drop procedures/functions/views
  4. Alter tables (rename columns, add columns, remove columns)
  5. Massage data – migrations sometimes involve changing data types that cannot be implicitly casted and require you to decide how the data is explicitly cast to the new type.  This is a requirement for a migrations platform.  Think about a case where you might want to combine fields, or move a field from one table to another, you wouldn’t want to lose the data.
  6. Run the tool via the command line.  If you cannot automate the tool in Continuous Integration what is the point?
  7. Create a copy of a database on demand.
  8. Backup/restore databases locally.

Let the team give feedback and decide together, what tool they would like to try out.

My recommendation at this point would be to include TSqlMigrations and RoundHouse as SQL based migration platforms.  In general I would recommend staying away from the fluent platforms as they often lack baseline capabilities and add overhead to learn a new API when SQL is already a very well known DSL.  Code migrations often get messy with procedures/views/functions as these have to be created with SQL and aren’t cross platform anyways.  IMO stick to SQL based migrations.

Reconciling Production

If your project is a legacy application, you will need to reconcile the current state of production with your development databases.  Find changes in production and bring them down to development, even if they are old and need to be removed.  Once complete, produce a baseline of either dev or prod as they are now in sync.  Commit this to your VCS of choice.

Add whatever schema changes tracking mechanism your tool requires to your development database.  This often requires adding a table to track the schema version of that database.  Your tool should support doing this for you.  You can add this table to production when you do your next release.

Script out any changes currently in dev.  Remove production artifacts that you brought down during reconciliation.  Add change scripts for any outstanding changes in dev since the last production release.  Commit these to your repository.

 

Say No to Shared Dev DBs

Simply put, you wouldn’t dream of sharing a code checkout, why would you share a development database?  If you have a shared dev database, back it up, distribute the backups and take the shared version offline (including the dev db server once all projects are using DB VCS).  Doing DB VCS with a shared database is bound to cause problems as people won’t be able to easily script out their own changes from those that others are working on.

 

First prod release

Copy prod to your beta/testing environment.  Add the schema changes table (or mechanism) and do a test run of your changes.  If successful you can schedule this to be run on production.

 

Evaluation

After your first release, evaluate the pain points of the process.  Try to find tools or modifications to existing tools to help fix them.  Don’t leave stones unturned, iteratively evolve your tools and practices to make the process as seamless as possible.  This is why I suggest open source alternatives.  Nothing is set in stone, a good example was adding transactional support to TSqlMigrations.  We ran into situations where an update would break a database, so I added a feature to do transactional updates and rollback on errors! 

Another good example is generating change scripts.  We have been manually making these for months now.  I found an open source project called Open DB Diff and integrated this with TSqlMigrations.  These were things we just accepted at the time when we began adopting our tool set.  Once we became comfortable with the base functionality, it was time to start automating more of the process.  Just like anything else with development, never be afraid to try to find tools to make your job easier!

 

Enjoy

-Wes

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati
Database version control resources

In the process of creating my own DB VCS tool tsqlmigrations.codeplex.com I ran into several good resources to help guide me along the way in reviewing existing offerings and in concepts that would be needed in a good DB VCS.  This is my list of helpful links that others can use to understand some of the concepts and some of the tools in existence.  In the next few posts I will try to explain how I used these to create TSqlMigrations.

 

Blogs entries

Three rules for database work - K. Scott Allen

http://odetocode.com/blogs/scott/archive/2008/01/30/three-rules-for-database-work.aspx

Versioning databases - the baseline

http://odetocode.com/blogs/scott/archive/2008/01/31/versioning-databases-the-baseline.aspx

Versioning databases - change scripts

http://odetocode.com/blogs/scott/archive/2008/02/02/versioning-databases-change-scripts.aspx

Versioning databases - views, stored procedures and the like

http://odetocode.com/blogs/scott/archive/2008/02/02/versioning-databases-views-stored-procedures-and-the-like.aspx

Versioning databases - branching and merging

http://odetocode.com/blogs/scott/archive/2008/02/03/versioning-databases-branching-and-merging.aspx

Evolutionary Database Design - Martin Fowler

http://martinfowler.com/articles/evodb.html

Are database migration frameworks worth the effort? - Good challenges

http://www.ridgway.co.za/archive/2009/01/03/are-database-migration-frameworks-worth-the-effort.aspx

Continuous Integration (in general)

http://martinfowler.com/articles/continuousIntegration.html

http://martinfowler.com/articles/originalContinuousIntegration.html

Is Your Database Under Version Control?

http://www.codinghorror.com/blog/archives/000743.html

11 Tools for Database Versioning

http://secretgeek.net/dbcontrol.asp

How to do database source control and builds

http://mikehadlow.blogspot.com/2006/09/how-to-do-database-source-control-and.html

.Net Database Migration Tool Roundup

http://flux88.com/blog/net-database-migration-tool-roundup/

Books

Book

Description

Refactoring Databases: Evolutionary Database Design

Martin Fowler signature series on refactoring databases.

Book site: http://databaserefactoring.com/

Recipes for Continuous Database Integration: Evolutionary Database Development (Digital Short Cut)

A good question/answer layout of common problems and solutions with database version control.

http://www.informit.com/store/product.aspx?isbn=032150206X

  • Share This Post:
  • Share on Twitter
  • Share on Facebook
  • Share on Technorati