It always annoyed me that there is no way to output schema to a file from SQL 2005 Management Studio. It has the the basic functionality to script out simple objects like table, or database (just the creation of the database itself, nothing in it), but you have to do one object at a time. On top of that, there is no way to output the data into a script either. Sure, you can do a backup and restore your database, but this solution is just silly if you have to do it a few times each day. I just want something that can output DDL and DML to a file for all the objects in the database.
In the past, I wrote a CodeSmith template to do this. It took me a few days of messing around to get everything generated correctly. Basically it allows me to point at an existing database to script the schema and data into a file. The generated script goes into our MSI every build, so the client can create the complete schema during the installation. The template did the job, but it did not do it the most efficient way, so it was a little bit slow when there was a lot of data. It can take three to eight minutes on a large database. The output script was not in the most robust form because it assumes that you have a blank database (does not check if object already exists). I am sure if I had more free time, I could have done it better. (I enjoy my free time at home doing other stuff :-)
Database Publishing Wizard does exactly what I wanted and more. I wish this tool would have came sooner. I tested it out on one of our databases and it ran much faster then my CodeSmith template, and the output script is much more robust. It took just a few seconds to generate the whole database schema with a good amount of data. It also supports command line interface, so it is perfect for the build server. I think we can actually speed up our build by using this tool instead. Currently the tool supports all unencrypted objects in the database.
Schema generation is what I am mostly interested in, but it does more than that. You can actually output the schema to a remote database via web service. If you are interested, the full documentation can be found here.
There has been a lot of “buzz” around Ruby on Rails for a while now. I have been following it since it hit the street. I have to admit I have not done much with Rails except for reading and dreaming about it, but I hope to work with it someday. However, in the back of my mind, I always wondered if I could build a real world application using this framework. Now a day, everyone is building so called “Enterprise” (What is enterprise anyways?) applications. If I use Rails, can I say my application is enterprise ready?
I came across Jay Field’s post on Rails: Enterprise Ready. One of the points he made in his post is that RoR right out of the box might not solve your problem (I do not think any framework can), but it can be easily extended. Being easily extended is the key to any useful and sucessful framework.
Besides all the enterprise worthyness stuff, I also found Jay’s approach to unit testing interesting. On top of the MVC layer, he inserted a Presenter abstraction in between the view and the controller. The Presenter will be responsible for maintaining data for the view to consume. Normally the controller would be responisble for this which might make testing a little bit messy. Since the presenter maintains the data, it can be tested independently. Of course the draw back is having to create and maintain a separate class.
He also mentioned all his unit tests run in less than a second. Assuming he has more than a couple hundred tests, that number is amazingly fast! One of the reasons he is able to achieve those numbers is because none of his tests actually hit the database. This reminds me, our unit tests are running way too slow right now mostly because a lot of tests hit the database (especially when we run it on Oracle, it is killing me). I am going to take off my RoR hat and try to see if I can make our tests faster :-)
As described in the WiX (Windows Installer XML) tutorial, upgrades can be categorized into three following categories:
- Small Updates: Very small changes to existing files. Version number stays the same.
- Minor Ugrades: Small changes, but no new files are added or removed. Change the the version number (most likely, change the minor or the build).
- Major Ugrades: New files are added or removed. Definitely change the version number (major, minor or build).
In my opinion, the first two options are not as useful for a typical upgrade when new features are added. One of the biggest restrictions for the first two options is that you can not add or remove new files. This is especially true when the creation of MSI is automated. On top of that, you cannot setup upgrade conditions. For example, an upgrade from 1.0.0 to 1.0.1 if 1.0.0 is installed. This feature only works for major upgrades. Normally, the user can click on the MSI to install the software. If you do a small update or minor upgrades you cannot simply just click on the MSI. The only way to invoke the upgrade mode is to pass command line arguments to the MSI.
I prefer to always do the the major upgrade to have maximum flexibility and avoid all the problems described above. Of course there are down sides to this approach as well.
- All the files must be deployed every time in order to do an upgrade.
- By MSI definition, major versions can live side by side with an older version.
The first one is not a big deal if your upgrade contains significant change because chances are MSI will force you to do a major upgrade anyway. The second problem is a little bit more annoying because it might be undesireable to have many old versions lingering around. One way to solve this problem is to transparently uninstall the old version before the new version. This works fine except it probably will wipe out any user defined settings if they are stored in a file that was deployed by the original MSI during an uninstall.
Overall, I still prefer the major updates strategy over the other two.
Recently I got a chance to play around with the Data Protection Application Programming Interface (DPAPI). With .NET 2.0 installed, you can encrypt your Web.config just by using aspnet_regiis.exe on the command line.
aspnet_regiis.exe -pe "connectionStrings" -app “/YourWebSite” –prov "DataProtectionConfigurationProvider"
You can read the ConnectionStrings section back using ConfigurationManager as if the file is not encrypted. That is all you have to do, the encryption is transparent to your application.
This encryption works great for Web.config, however, it does not work if you want to encrypt the App.config for non web based applications. The aspnet_regiis is hardcoded to look for Web.config.
One can programatically encrypt just as easily:
Configuration configuration = ConfigurationManager.OpenExeConfiguration(appConfig);
ConfigurationSection section = this.configuration.GetSection("connectionStrings");
if (!section.SectionInformation.IsProtected) {
section.SectionInformation.ProtectSection("DataProtectionConfigurationProvider");
section.SectionInformation.ForceSave = true;
configuration.Save(ConfigurationSaveMode.Modified);
}
To decrypt just do the oposite:
if (section.SectionInformation.IsProtected) {
// …
section.SectionInformation.UnprotectSection();
// …
}
The same code above can work for Web.config and App.config. For Web.config I would use WebConfigurationManager.OpenWebConfiguration(webConfig). For our environment we have a web project and some winform projects, so it is easier for us to create a simple tool to maintain both configuration file types.
You can buy almost anything on Amazon, ranging from books to hairdryers. Now you can even buy your milk! Check it out for yourself. Read the comments and look at the images, it is funny as hell. It is probably a joke, but it is still funny.
A couple of weeks ago, I saw a documentary on HBO called Hacking Democracy. As software developersbg, we all know there are tons of crappy software out there. I could not believe how terrible the supposedly “high tech” and “top secret” voting machine software actually is. You do not have to be geeky to understand the documentary. Check it out, if you have not seen it.
Last week, we noticed our data grid (an Infragistics UltraGrid) was behaving very slow during data population and filtering. At first I assumed we were doing something wrong in our code to cause the slow down. After a few minutes of poking around I did not see anything obvious. Someone noticed the equals method was called frequently when the data is being populated in the grid. It seems like the equals method was being invoked when the BindingSource is being set. I decided to do a simple little experiment to figure out what was going on.
class Program {
public static int NumberOfItemsAdded = 1000;
static void Main(string[] args) {
BindingSource bindingSource = new BindingSource();
List<User> users = new List<User>();
for (int i = 0; i < NumberOfItemsAdded; i++) users.Add(new User());
bindingSource.DataSource = users;
Console.WriteLine("Equals was called: " + User.NumOfTimesEqualsCalled);
Console.ReadLine();
}
}
internal class User {
public static int NumOfTimesEqualsCalled = 0;
public override bool Equals(object obj) {
NumOfTimesEqualsCalled++;
return base.Equals(obj);
}
}
I started out with 1000 objects in my list then set it to the binding source. The equals method was not called once. This seemed weird to me because in our application the equals method is definitely being called. I decided to add a property to the User class and see what happens.
public string Prop1 {
get { return ""; }
}
By adding a property to the User class, the equals method is being invoked 4 times. I added a couple more properties and, sure enough, the equals method is being called 4 times per property. If you have 10 properties, the equals will be called 40 times. I decided to change the implementation of the equals method to see if that would change number of times being invoked.
public override bool Equals(object obj) {
NumOfTimesEqualsCalled++;
return true;
}
Nothing seemed to change when the equals is always returning true. How about the opposite?
public override bool Equals(object obj) {
NumOfTimesEqualsCalled++;
return false;
}
When equals is always returning false it became a little bit more interesting Now it calls equals on multiples of 18 times per property. It is not realistic for equals to always return false, so I decided to try a mixture of both, half true and half false.
public override bool Equals(object obj) {
NumOfTimesEqualsCalled++;
Random random = new Random();
int randomInt = random.Next();
return randomInt % 2 == 0;
}
This time it was not so straight forward with the number of time the equals method is being invoked. With one property it is called 13 times. With two properties, it is called 21 times. From this point on, it seems to be multiples of 4 times per property. What happens if equals return a value randomly? It depends on how many properties you have when you set it to be a binding source. If you are lucky, the equals method will be invoked 4 times per property. If you have 10 properties, it will be called 40 times. But in many cases it could be a lot worse. The equals could be invoked 18 times per property which is 180 times if you have 10 properties. All that work is done just to set the binding source’s data source. It is not even displaying in the grid.
In my example I created a list with 1000 objects, but that actually does not matter. The list could contain one object and the outcome will be the exactly same. In our application the slow down is more exaggerated because we are using reflection to perform the equals. The only reason we are using reflection is because we only use equals to perform dirty checking on the object after the user has performed an edit. This is not a performance issue because at most you are only doing reflection on object. However, if the binding source is calling it hundreds of times, this becomes an issue. Since we are only using equals for dirty checking, we can create a separate dirty checking equals method to get around the binding source calling equals. I guess I do not understand why setting the data source to a binding source would call equals at all. To make the matter worse, if you are manipulating the binding source, the equals method will be invoked again.
Recently I decided to check out the beta version of Agile Web Development with Rails book, which is targeted to be released this fall. It is very interesting that the authors also brought agility into book writing. It allows readers to provide feedback to new material during the development of the book. I am also glad to see that the migration part of the framework has become a big part of the book, even having a separate chapter dedicated to it. It uses migration instead of DDL in the entire demo application. Besides the migration, there are many significant updates to various parts of the book. But in this post, I will focus mostly on the migration tool.
The migration tool is targeting Rails applications, but it can also be useful outside the Rails world. There are some people out there already (includes us, here is my old post on it) experimenting using Rails migration as "Enterprise glue" to maintain database schemas. I hope the final release version of the book will have something on this topic.
I will outline the basics of rails migration below for those are not familiar with it.
What is Ruby on Rails Migration?
At the most simple level, it allows developers to change the database schema as the application requires it in a simple and easy manner. Instead of writing DDL scripts, you use Ruby as a DSL (domain specific language) to describe what the schema should look like when the changes are required.
Why use it?
The migration tool allows developers to upgrade or downgrade a schema version without loosing data (of course, if you drop stuff it will be lost). Sure, you can write DDL that does the same thing for you (sort of, but not really without some external tooling), but it will take a lot more effort especially when you are supporting more than one database. In theory, the migration script you write for one database should work on other databases as long as the operation is supported (currently, it supports all major databases including open source alternatives). I said in theory because not all databases behave exactly the same way for a given operation. There might be some minor differences, but that's for another post. If you need to extend or change the existing migration functionality, it is usually very easy to do so.
How does it work?
You basically write a Ruby class that inherits from ActiveRecord::Migration to describe what needs to be accomplished. There are only two methods you need to write, up and down. Up method will be called during an upgrade, and down method for downgrading.
class AddTable < ActiveRecord::Migration
def self.up
create_table :cars do |t|
t.column :model, :string
t.column :year, :int
t.column :make, :string
t.column :comments, :text
end
end
def self.down
drop_table :cars
end
end
In this trivial example, it creates a cars table with model, year, make, and comments columns. One interesting thing to note about model and make columns are both string type. This is not DDL we are working with. Migration uses Ruby classes to encapsulate the internal database types in order to abstract out the databases. Depending on what type of database you are using, string type might vary, but it tends to be pretty consistant across different databases. In the down method, it just does exactly the opposite, as if the script has never been run.
On the command line, if you run “rake db:migrate”, your database will be upgraded to the latest version. In this case, the cars table will be added. Each migration script will have a 3 digit number in the start of the file name which represents the schema version of the script. That same version number is also stored in the database you are using. That is how migration knows which script to run during an upgrade or downgrade. Using the example above, the file name would look something like the following:
001_create_cars.rb
Lets say you are currently on version four but you need to roll back to version one, you can run the following command:
rake db:migration VERSION=1
Migration supports most of the operations you would need to perform on the daily basis. But if you need to do something it does not support you can always execute DDL in your migration script.
One of the really cool things about using Ruby as a DSL is that you have the power of a real programming language to create your migration scripts. This comes in really handy when you need to create test data.
I only scratched the surface on the stuff I covered on Rails migration. I would encourage anyone that is interested to read the Rails book.
We finally moved into our highly anticipated new office space. It was designed to be friendly with plenty of white boards and space to move around. The whole back wall in our development area can be used as a white board, so we no longer have to erase our designs until they are no longer needed. In our old office, all the developers were packed into a small area, so there was no room for the product team. Now the product team is located behind the development team,which makes it easier to communicate with each other.
After a month of a busy schedule, I finally have time to go though my blog aggregator today. I noticed a few people in my feed mentioned this interesting article. If you have not read it yet, go read it. I think this guy is smoking something, but you can be the judge.
Retrospectives are a way for a software development teams to discuss and figure out ways to resolve issues when an increment of work is done. The strength of the retrospective comes from focusing on the team members and how to improve as a team. For the team I am working on, the increment of work just so happens to be one week. Our retrospective definitely helped us fine tune our team on the iteration basis. At times, I feel like our retrospective has become routine and rushed because the sooner we can finish, the sooner we can start coding since we are on a short iteration. I decided to check out Retrospective by Esther Derby and Diana Larsen to see if there is anything we can do to revitalize our retrospective. Overall, I found the book to be really helpful and easy to read and understand. The book encourages very structured and well planned out retrospectives, which I feel is our weak point. The book suggested five phases listed below:
1. Set the stage
2. Gather data
3. Generate insights
4. Decide what to do
5. Close the retrospective
Set the stage phase: Basically an ice breaker that the retrospective leader will facilitate in an environment that allows the team members to focus and participate on the retrospective. According to the book, this is one of the most import phases of the retrospective, and many teams usually skip it. As you can guess, we also skipped this phase with our past retrospectives as well. In order to have a successful retrospective, all team members need to participate early, otherwise people are less likely to say anything in the later phase.
Gather data phase: This s a great way to put everyone on the same page before getting to the meat of the retrospective even if you are on a short iteration. It is much easier to determine shifts and patterns by looking at hard data. On our team, most of the time we do not have hard data, so at times we rely on team members’ subjective feelings as to how things went. This can be problematic because it is hard to have a constructive discussion when people have different views on what went on.
Once the team starts to see some patterns and shifts, it is now time to move into generating insights by looking at the big picture and figuring out root causes. We tend to have pretty good discussions in this phase, but it usually went on longer than needed because we do not have hard data to help us.
Deciding what to do is the meat of the retrospective. This stage is pretty self explanatory, but the important point here is whatever the team decides to do, the team must commit to it.
Closing the retrospective is very helpful for the team to fine tune the next retrospective. Like anything else, it needs to be tuned every once in a while, so do the retrospectives.
The description I provided above is by no means comprehensive. If you feel your retrospective is not as effective any more, this book is for you. The book provides many activities for each phase of the retrospective, so you do not have to stick with the same old activity over and over again. We followed the format of the book, and ran two retrospectives with a great deal of positive feedback from the team members.
Logging is one of those things where once you have it, you never really pay much attention to it. Today I decided to investigate what is new in the logging world since the last time I looked at it. I heard of NLog some time ago, but never took the time to test drive it. In the past I have always used log4net, so I decided to compare NLog to it. Here are some of the things that I really like about NLog, and it might not be possible or “easy” to do with log4net:
Asynchronous Processing:
NLog provider wrapper and compound targets for asynchronous processing, loading balancing, buffering, and etc.
Easy Configuration:
The configuration file structure is simple and easier to write. If you do not remember all the options, no worries, you will receive Intellisense in Visual Studio.
Include Files:
You can split up the configuration to multiple files by using the <include />.
Contextual Information:
Like all other logging frameworks, NLog is able to add contextual information when each message is logged. You can use the contextual information to create a log file for each user, create one log file per day, one file per logger, or any combination. Stack trace can also be included when each message is logged.
Not Swallowing Exceptions:
Sometimes it is useful to know when the logging framework is throwing an exception to prevent logging from happening; NLog can be configured not to swallow any exceptions. It is useful to troubleshoot problems during deployment.
Defining Variables in Configuration File:
Common value can be stored in a variable. Great way to keep you configuration file DRY.
Note: This is a superficial look at some of the features of NLog, and I have not worked with NLog extensively.
For anyone doing Test Driven Development, mocks (stubs) are commonly used. Whether you hand roll your own mocks, or use a mock framework like Rhino Mocks, stubs are used to in order to isolate the code we want to test. Before we can isolate our code, proper separations of concerns are required; such as implementing the Model View Controller or Model View Presenter pattern for UI testing. After you practice TDD for a few months, and write countless tests, you start to see a pattern of how to make your presentation layer testable and everything becomes automatic and second nature.
One of the big benefits of TDD is using tests to guide your design. Have you ever stopped and wondered if your design has really improved by structuring your code in a manner that is testable? Are you creating more duplication or allowing too much accessibility? Let’s examine how the presentation layer can be tested using the MVP pattern. I will use a simple example where the customer information can enter entered. I purposely omitted the presenter interaction with the view in order to put more focus on the view.
public interface ICustomerDetailsView {
string CustomerFirstName { get; }
string CustomerLastName { get;}
List<Order> Orders { get; }
}
public class CustomDetailsView : UserControl, ICustomerDetailsView {
public string CustomerFirstName {
get { return this.firstNameTextBox.Text; }
}
public string CustomerLastName {
get { return this.lastNameTextBox.Text; }
}
public List<Order> Orders {
get {
List<Order> orders = new List<Order>();
foreach (Order order in this.ordersComboBox.Items) {
orders.Add(order);
}
return orders;
}
}
}
We start with an interface called ICustomerDetailsView to describe behaviors of the view. Now we can use the interface to create our mock view. In the MVP pattern, the presenter is an extension to the view, so all the interactions occuring in the view will be delegated to the presenter. With the delgation to the presenter, we would only have to explicitly test the presenter. The reason the view is not tested is because all the logic lives inside of the presenter, so we end up testing the API.
public class MockCustomDetailsView : ICustomerDetailsView {
// implement ICustomerDetailsView
// you can have setters here so values can be injected for testing
}
The presenter can now use the mock view during automated testing and will not know a difference from the real thing. All your tests pass and life is good. Before you checkin your code, you decide to smoke your application. To your surprise, your view does not behave correctly. You poke around for a few seconds and you remember your tests actually use the mock view, so you never wired up the real thing. In this simple scenario, when all your tests pass it does not mean your application will work correctly if you never wire it up. Sometimes it can feel a little awkward when you are writing the mock view to mimic the real view, so your tests can pass. On top of that, the interface was only added to make the mock possible in this scenario. I understand creating an interface is considered “best practice” to loosen up the coupling, but in this case, the interface was created to make stubbing possible. It is a little smelly to me because files were added only to make testing possible, and the real view was treated like a second class citizen. Maybe this is not so bad, and I should not lose sleep over it.
Maybe life would be a whole lot better if we could use the real view in this case for testing. That would eliminate our problem of forgetting to wire things up. It would also reduce the number of files we have to write, and would achieve the same outcome. Let’s see if we can create another implementation of CustomerDetailsView to accomplish this. We will call the new view ICustomerDetailsView2:
public class CustomDetailsView2 : UserControl {
public string CustomerFirstName {
get { return this.firstNameTextBox.Text; }
set { this.firstNameTextBox.Text = value; }
}
public string CustomerLastName {
get { return this.lastNameTextBox.Text; }
set { this.lastNameTextBox.Text = value; }
}
public List<Order> Orders {
get {
List<Order> orders = new List<Order>();
foreach (Order order in this.ordersComboBox.Items) {
orders.Add(order);
}
return orders;
}
set {
foreach (Order order in value) {
this.ordersComboBox.Items.Add(order);
}
}
}
}
We can use this view for both testing and production code, but the side effect is that we have to expose setters only for testing purposes. In our first implementation, we were able to put the setters only on the mock, so our real view is not polluted. The reason the setter is unnecessary for CustomerDetailsView2 is because we would never programmatically call the setter. The setter is used to simulate data input from the user.
Either way you look at it, there are some drawbacks to both techniques. This really depends on what you consider the lesser of the evils. I know there are ways to solve this problem by using a mock framework, but then again are you adding technology in order to make testing possible? Let’s say we live in a world where we do not need to do any testing. Whatever code we write will always work. Would you have implemented CustomerDetailsView the same way? The second example exposes setters when the real system will never use it. Is that breaking the accessibility of the view?
I understand the ability to mock things out is good because that means your system has enough abstractions. I am just not sure mocks should be used every where in order to make testing possible. I know there is a fine line here, like many good things in the Computer Science world, but I often struggle to choose the optimal way. I hope no one got the impression that I am bashing on TDD or mock objects. In fact, it is quiet the opposite.
Like many
teams, we always fine tune our process, so we can consistently be better.
Steve Eichert has a great
post that describes our process in detail. In addition to what Steve described, we also play a game that I feel has become part of our process. We call it “the leader game”. A person is chosen to be the “leader” for the iteration. The “leader” is in charge of tasks (not story tasks) that would normally fall through the cracks. It can be as simple as making sure time is tracked on each story task, or something more like leading a discussion on a particular part of the system in our software. Basically, the leader makes sure things get done in general.
The game seems pretty straight forward, right? There is one catch; no one knows who the leader is until end of the iteration. At the end of the iteration, everyone votes the person they think was the leader. The goal of the game is to be voted as the leader even though you are not the real “leader”. The leader game is a spin off
The Mole.