Architecture

The Joel Test for 2010

Bill Tudor — Mon, 04 Jan 2010 10:52:07 GMT

Back in June, I posted my own version of the Joel Test – updated for 2010. [See Here.]

Do you have a change management system?
Can everyone make a build in one step?
Do your daily builds include automated tests?
Is work item tracking integrated with source control?
Do you fix bugs and write new code?
Do you track progress and manage change?
Do you have a requirements management system?
Do programmers have quiet working conditions and teaming rooms?
Do you use the best tools money can buy?
Are your testers involved in requirements management?
Do new candidates review code during their interview?
Do you do hallway usability testing?

Static Reflection in .Net

Bill Tudor — Mon, 12 Oct 2009 17:08:21 GMT

Static Reflection in .Net

Where have I been?

I recently took a few minutes to look through the latest MSDN magazine and ran across a nice article by Jeremy Miller on “Functional Programming for Everyday .NET Development”. The section titled “Lambdas as Data” was my first introduction to static reflection in .Net. Where the heck have I been? I don’t write database persistence code, so I am not familiar with NHibernate (or fluent NHibernate) (which is where Jeremy’s sample comes from), but I am still surprised at how little I understand about everything Linq brought to the .Net party. A quick search around the internet reveals this topic is well-covered, and several years old.

Example Code

Suppose you have a need to dynamically discover properties on an object. Easy, right? After all, that’s what the System.Reflection namespaces is for! However..

Let’s look at a class named “SomeObject” with a property “SomeProperty”:

class SomeObject

    public string SomeProperty { get; set; }

You can use so-called “static reflection” like this:

   1: class Program

   2: {

   3:     static void Main(string[] args)

   4:     {

   5:         SomeObject o = new SomeObject()

   6:         {

   7:             SomeProperty = "Test"

   8:         };

   9:         staticReflection(o, e => e.SomeProperty);

  10:         Console.ReadKey();

  11:     }

  12:     static void staticReflection<T>(T o, Expression<Func<T,object>> expression)

  13:     {

  14:         MemberExpression me = expression.Body as MemberExpression;

  15:         if (me != null && me.Member.MemberType == System.Reflection.MemberTypes.Property)

  16:         {

  17:             Console.WriteLine("Property: '{0}' of type '{1}' has value '{2}'.",

  18:                 me.Member.Name,

  19:                 expression.Body.Type,

  20:                 expression.Compile().Invoke(o)

  21:                 );

  22:         }

  23:     }

  24: }

The output is:

Property: 'SomeProperty' of type 'System.String' has value 'Test'.

Very cool. Now I need to understand why the second argument has to be something like Expression<Func<T, object>> rather than simply Expression<T>. If the latter is used, the compiler rewards me with:

Error 1 Cannot convert lambda to an expression tree whose type argument 'StaticReflection.SomeObject' is not a delegate type

It’s the whole “return type” thing on the delegate that throws me as well – where does that come from? In my code sample I have declared the Func return type as type “object” (it is actually a string in my case). Why don’t I have to do something like “e => {return e.SomeProperty;}”, and how come a lambda expression with a statement body like that cannot be converted into an expression tree (compiler’s words, not mine)?

I still have a little more learning to do before really understanding what’s going on here. My suspicion is that Linq brought a heck of a lot more to the party than it might appear at first look.

Designing Software for Scientists

Bill Tudor — Tue, 06 Oct 2009 10:02:48 GMT

Designing Software for Scientists

Creating a software design for scientists can be tricky business. Unlike the businessman, who is better off looking at the pretty database report or marveling over the catchy name of the latest Forms over Data application, the scientist can actually contribute to the implementation – provided you use the right architecture. In year’s past (I’m referring to 20+ years ago), the typical Physicist / Chemist / etc. was nearly on par with the typical software engineer / computer scientists with respect to software design skills. Although the scientific community of that time preferred the ‘GOTO’, they nevertheless understood the value of using subroutines to nearly the same degree as a computer scientists.

Not so, today. Software design has come a long way since then.

Software Design has Matured

Software design has matured. We have terms like inheritance and encapsulation. We talk about singletons, abstract factories, and chain of responsibility. We use terms like polymorphism, covariance, and contravariance (as applied to software design). We compose systems with Inversion of Control containers. Mock objects are used for unit testing. Reflection, dynamic and static languages. The discipline of software engineering has matured to a point where the typical scientist cannot possibly hope to understand what makes a well-design system – no more than a software engineer can understand how to model protein folding of polypeptides.

So how do we design software so that other scientists and engineers can participate in the implementation? Here are some ideas.

1. Keep it Simple.

Don’t compose in an IoC container – too abstract. Do use layers, repositories, and multiple program units. Scientists understand process control, state transitions, and other engineering/scientific/mathematics concepts so use them in your software designs. [Object models? Not sure where I stand on this one. There might be a whole blog post in here somewhere.]

2. Work in Pairs

Match a subject matter expert with a software design expert.

3. Limit Participation

Limit the “coding” performed by the scientist to the functional code, not the design. The infrastructure, layout, program flow as well as all meta-design elements (project structure, build environment, coding standards, etc.) – essentially the software design, should be strictly the realm of the software architect. Leave the algorithms for the subject matter expert.

4. Move the problem domain.

Let the scientists program in their own “language” - provide a scripting component, meta-programming, or data-driven model such that the subject matter expert remains closer to their problem domain, even if this means additional software (such as interpreters, data transformation engines) must be developed. In a really large system, you might go so far as to develop a rules engine or other knowledge base approach.

5. Use Abstractions

The nature of software is to provide abstractions. This also happens to be common of all science and engineering. You can use your entire software design Bag ‘o Tricks provided you supply an appropriate level of abstraction over top. Explain the abstraction, and the typical scientist will have no trouble using it, even if they really don’t know what is going on. So in spite of the first idea listed here, you can fire up your favorite IoC container and use it – as long as you provide the right abstraction for your scientific colleagues.

The “Web Application” Must Die

Bill Tudor — Mon, 14 Sep 2009 09:55:56 GMT

Remember when a program was termed a “GUI Application” - a program with a Graphical User Interface? It had features like menus that “dropped-down” (or “popped up”), scroll bars to reveal more of the virtual surface area, a pointing device (still called a cursor), and lots and lots of Drag & Drop. Thankfully, that moniker has mostly died. Now it is time another term to fall by the wayside – the Web Application.

What is a Web Application?

Just what is a web application, anyway. An application delivered over the web? A Java applet, flash, or silverlight program running in a browser? How about HTML+Javascript with lots of Ajaxy asynchronous updating? What about plain old static HTML pages with hyperlinks to navigate between them?

Who are Web Developers?

Just who are these web developers? Are they PHP junkies, ASP.net gurus, Flex monkeys? Are the real web developers HTML+CSS+JavaScriptors? Rails developers? Web site designers? What about a C++ application that runs locally yet stores data in the cloud or communicates with other instances on the internet? Is the twitter client a web application?

The ubiquitous internet makes every application a web application. The term has lost its meaning, and the term must die.

Just like “GUI”. If an application requires a user interface, it will be a graphical one. If an application requires communication between computers, or presence on more than one computer, it will use the “Web”. In one way or another. Server side, client side, I don’t care. These days, every “application” is a “web application”.

The Joel Test – Updated for 2010

Bill Tudor — Tue, 16 Jun 2009 09:23:07 GMT

The Joel test is intended to gauge the quality of a software development organization. Ten years later, it is in need of an update. Below is a possible “New Joel Test”, updated for 2010.

1. Do you use source control?

1. Do you have a change management system?

There needs to be a configuration management plan, control of assets beyond “source code”, branching and merging strategy (even if it’s not to branch), release stabilization and deployment plan, security permissions and roles, quality gates for checking and/or merge, etc. Otherwise, “use source control” is just referring to a glorified backup system.

2. Can you make a build in one step?

2. Can everyone make a build in one step?

Individual developers need access to the automated build system. The official nightly build can “build everything”, but developers need to be able to build individual parts, pull together the required assets, and run from a representative staging area – maybe a virtual machine (or two), a network share, or just a folder on disk.

3. Do you make daily builds?

3. Do your daily build include automated tests?

It’s not enough to just build – we need an indication of build quality. “Break the build” should mean more than “does not compile”, it should also mean “does not pass the tests”.

4. Do you have a bug database?

4. Is work item tracking integrated with source control?

What artifacts were changed with a given bug fix? What bug or development task is associated with the most recent change-set? Which bugs are associated with a given merge?

Check-outs (or commits, depending on the source control system in use), must be associated with bug reports or development tasks. Each check-in should reflect changes for one task (unless they are coupled tasks). You should also be able to query, based on a bug report, which artifacts were changed (or ideally, currently in the process of being changed). In other words, change management must be integrated with bug tracking and project task tracking.

5. Do you fix bugs before writing new code?

5. Do you fix bugs and write new code?

A properly run project will have a branching / merging strategy (or tagging/labeling policy) that allows simultaneous bug fixes with new development. A quality organization will be able to correct bugs in previous releases, and ensure those corrections are in the current release. Without copying or duplicating the code base.

6. Do you have an up-to-date schedule?

6. Do you track progress and manage change?

Project tracking must include processes that allow for managing (planning / executing / tracking) change as well as tracking progress and periodic re-planning of the remaining work.

7. Do you have a spec?

7. Do you have a requirements management system?

The “spec” will change; that change needs to be managed. Too often I see organizations with an excellent process for development of the “spec” and a horrible process for managing change in the spec.

Note: In 2010, if you don’t have a “spec”, you automatically get a ZERO on the new Joel Test and there is no point in continuing on.

8. Do programmers have quiet working conditions?

8. Do programmers have quiet working conditions and teaming rooms?

The teaming rooms are more than a room with a white-board. There is a conference phone, computers (more than one), projector, and lots of white boards. Code reviews, code walk-through, mentoring, design, dispute resolution, meeting with customers – can all take place here.

9. Do you use the best tools money can buy?

No change needed. This one is timeless.

10. Do you have testers?

10. Are your testers involved in requirements management?

The test team needs to be involved in the requirements management system as well as the product validation process. Tests are executed against the requirements. The testers should have authority to approve/reject a deployment as well as approve or reject a requirement. Otherwise, they’re just testers.

11. Do new candidates write code during their interview?

11. Do new candidates review code during their interview?

“Writing code” is one thing – understanding code is another. Candidates should be able to answer questions like “is this code thread-safe?”, “is the code well-commented?”, or “Is the implementation appropriate” – and why. Printing fizz-buzz is one thing; being able to appropriately critique a page of code is another. As a bonus, you could always ask them to re-write it and get the best of both.

12. Do you do hallway usability testing?

Huh? I’ll have to go and read the original on this one, again. I will leave it alone here.

* * * * *

The Joel Test updated for the year 2010. And a few more months to get it right.

Top 5 Programmer Mistakes

Bill Tudor — Tue, 05 May 2009 03:17:39 GMT

This post is not another list of security mistakes in web application development – not my cup of tea. No, this is a list of the top five programmer errors. That is, errors made by software development professionals performing their craft.

No. 1 – Writing Too Much Code

“If I have seen further it is only by standing on the shoulders of giants” – Sir Isaac Newton.

I recently came across this post by Chris Williams asking for help in finding a bug that resulted in “serious error in the debugger”. There were actually several bugs in the code, some worse than others. One of the proposed solutions also produced a new bug, though less serious. The bottom line – a whole bunch of code could be replaced with a single library call. Now, it’s nice that today’s frameworks have lots of high level method calls preventing you from writing too much code. But what is really nice is that the less code you write, the less bugs you insert into your projects. Failure to take advantage of this gift comes in as the number one mistake programmer’s make.

Don’t write so much code!

No. 2 – Ignoring the Problem Domain

How many times have you seen an “ini” file reader that simply trades one storage mechanism (an INI file on disk) for another (a dictionary of key/value pairs). What’s the point? While this may be appropriate as a middle-tier step, there is a problem domain in there somewhere, right? A “To Do” application with categories can store its information in a ini file (or any other back-end store), but when I read it from storage, I want the result to be a collection of categories, each with collections of tasks hanging off of them. Not some dictionary for me to parse later (typically throughout the application several times over).

Move closer to the problem domain. Whatever your application is. If you are developing a generic solution/library, use a layered approach with problem-domain specific objects built on top of generic, low-level objects. Solving problems with data structures (consider the System.Data.DataSet object in .Net) shows up often enough to make this the number two mistake that programmers make.

Don’t ignore the problem domain.

No. 3 – Inappropriate Level of Rigor

Everyone likes one size fits all. Too bad one size never really fits all. Not enough design, too much design. Not enough specification, too much specification. Not enough test, too much test. Not enough extensibility, too much extensibility. Not enough documentation, too much documentation. Zealots in some of these areas will let you know that you can never have enough ___. You can always have too much X. More often, there’s too little: Design, specification. Sometimes, there is to much: extensibility. Sometimes both extremes are common: Test. [Nobody knows what to say about documentation – we like to say ‘never enough’ while refusing to read what little is there]. I don’t know what the correct level of rigor for your project is, but if you don’t think about it – and just do whatever you did before – you are making the number three mistake on my list.

Consider the level of rigor for each project you undertake.

No. 4 – Programming by Accident

“Works for me” – anonymous

When you don’t know why something works – find out. There is a world of difference between “working code” and “good code”. Never be satisfied with “getting lucky”. Your test case works, but you don’t know why. Not a good place to be. Understand what you are doing, understand why it works, understand when it works, and more importantly understand when it does not work. If you don’t know something – you have work to do. If you take a look at the post mentioned at the start of this document, the author states that in one case “Button1 read in File1 via a StreamReader and works perfectly”. No, it worked in that case by accident. If it works for you, but you don’t know why, then you are making the number four programmer mistake – programming by accident.

Don’t program by accident.

No. 5 – Tools are Not the Answer

Tools are great, but they are never the answer. Tools improve productivity. Tools used correctly improve quality, but they do not solve problems. Tools are good for problems that have already been solved, not ones that need programmers to solve them. Your tools will change, and there will always been another tool. The number five mistake programmers make – suggesting a new tool for an old problem.

Top 5 Network Programming Mistakes

Bill Tudor — Tue, 17 Feb 2009 09:11:00 GMT

Here are my top five. What are yours?

No. 1 Failure to clearly define a network protocol

In today’s world of everything over HTTP, designing low-level network protocols are mostly a thing of the past. Behind the firewall, however, failure to clearly define a network protocol before implementation is one of the largest sources of re-work.

Take the time to clearly define the network communication protocols – at whatever level is required for your application. Even if you are running over HTTP, define the application level protocols in such a way that implementation is easier, interoperability is easier, states and state transitions are well known, and always make maximum use of existing protocols in the design.

This error comes in No. 1 since it leads to an exponential increase in development and maintenance hours. And it cannot be easily re-factored out.

No. 2 Incorrectly Reading and Writing data over TCP

In most sockets libraries, writing N bytes to a TCP connection may result in less than N bytes transmitted. Similarly, asking for N bytes returns anywhere from 1 to N bytes (assuming no errors). You must continue to call read/write until the entire message defined by your protocol is sent or received.

For example, the Win32 API send() and recv() functions can send or receive less than the requested data amount. Most of the time, all of the data will be sent/received in one chunk - particularly if the message size is small. So poorly written code with this error “works” nearly all of the time – but not all the time.

This behavior is library dependant. For example, the Write() method on the NetworkStream class in .Net will block until all of your bytes are sent. On the other hand, the Send() method on the Socket class in the same library will either block until all bytes are sent, or, if placed in non-blocking mode, will behave like the send() function described earlier and it then becomes your job to get all the bytes out the door. The Read() method on any of these entities (NetworkStream, Socket) can always return less than the requested number of bytes.

I see code all the time where a service application sends a message block of, say, 100 bytes over TCP. The client reads 100 bytes at a time from the network but has no ability to react to receiving a partial message. In such a scenario, the application works most of the time – just not all of the time – which is why this issue clocks in at No. 2.

No. 3 Validating Protocols

I confess right now: I am no expert in network security. I live mostly behind the firewall and off the public internet. However, failure to validate network messages even when you fully control both sides of the client – server equation comes in at No. 3. Even if the protocol errors are not malicious, failure to validate data results in both crashes and hard to find bugs, if not security holes. For example, if your protocol contains a payload length as part of the data, verify that you actually read that length of data before attempting to access it.

No. 4 Buffer Overflows

When writing network programming code, you typically will define some fixed size buffers. While this is not always the case, performance and scalability issues tend to force you in that direction. Enough said – don’t overflow any fixed size buffers that you have defined. If we are talking network security, this error might be number one. For this discussion, it comes in at No. 4.

No. 5 Improper Error Handling

Network programming code must plan for errors – remote client goes down, network switch goes down, a router fails, a network cable becomes un-plugged. Nearly all the network programming code I come across dutifully checks for errors with each network API method call. This is a good thing. What the code does in the case of error, however, is where the problems come into play. Some examples: Gobble up all errors; Failure to design in any sort of logging/trace system; re-transmitting data in endless loops, etc., I see nice error checking, but poor error handling, often enough to make this item No. 5.