Geeks With Blogs
Bill Tudor Weblog

The Joel test is intended to gauge the quality of a software development organization. Ten years later, it is in need of an update. Below is a possible “New Joel Test”, updated for 2010.

1. Do you use source control?

1. Do you have a change management system?

There needs to be a configuration management plan, control of assets beyond “source code”, branching and merging strategy (even if it’s not to branch), release stabilization and deployment plan, security permissions and roles, quality gates for checking and/or merge, etc. Otherwise, “use source control” is just referring to a glorified backup system.

2. Can you make a build in one step?

2. Can everyone make a build in one step?

Individual developers need access to the automated build system. The official nightly build can “build everything”, but developers need to be able to build individual parts, pull together the required assets, and run from a representative staging area – maybe a virtual machine (or two), a network share, or just a folder on disk.

3. Do you make daily builds?

3. Do your daily build include automated tests?

It’s not enough to just build – we need an indication of build quality. “Break the build” should mean more than “does not compile”, it should also mean “does not pass the tests”.

4. Do you have a bug database?

4. Is work item tracking integrated with source control?

What artifacts were changed with a given bug fix? What bug or development task is associated with the most recent change-set? Which bugs are associated with a given merge?

Check-outs (or commits, depending on the source control system in use), must be associated with bug reports or development tasks. Each check-in should reflect changes for one task (unless they are coupled tasks). You should also be able to query, based on a bug report, which artifacts were changed (or ideally, currently in the process of being changed). In other words, change management must be integrated with bug tracking and project task tracking.

5. Do you fix bugs before writing new code?

5. Do you fix bugs and write new code?

A properly run project will have a branching / merging strategy (or tagging/labeling policy) that allows simultaneous bug fixes with new development. A quality organization will be able to correct bugs in previous releases, and ensure those corrections are in the current release. Without copying or duplicating the code base.

6. Do you have an up-to-date schedule?

6. Do you track progress and manage change?

Project tracking must include processes that allow for managing (planning / executing / tracking) change as well as tracking progress and periodic re-planning of the remaining work.

7. Do you have a spec?

7. Do you have a requirements management system?

The “spec” will change; that change needs to be managed. Too often I see organizations with an excellent process for development of the “spec” and a horrible process for managing change in the spec.

Note: In 2010, if you don’t have a “spec”, you automatically get a ZERO on the new Joel Test and there is no point in continuing on.

8. Do programmers have quiet working conditions?

8. Do programmers have quiet working conditions and teaming rooms?

The teaming rooms are more than a room with a white-board. There is a conference phone, computers (more than one), projector, and lots of white boards. Code reviews, code walk-through, mentoring, design, dispute resolution, meeting with customers – can all take place here.

9. Do you use the best tools money can buy?

No change needed. This one is timeless.

10. Do you have testers?

10. Are your testers involved in requirements management?

The test team needs to be involved in the requirements management system as well as the product validation process. Tests are executed against the requirements. The testers should have authority to approve/reject a deployment as well as approve or reject a requirement. Otherwise, they’re just testers.

11. Do new candidates write code during their interview?

11. Do new candidates review code during their interview?

“Writing code” is one thing – understanding code is another. Candidates should be able to answer questions like “is this code thread-safe?”, “is the code well-commented?”, or “Is the implementation appropriate” – and why. Printing fizz-buzz is one thing; being able to appropriately critique a page of code is another. As a bonus, you could always ask them to re-write it and get the best of both.

12. Do you do hallway usability testing?

Huh? I’ll have to go and read the original on this one, again. I will leave it alone here.

*  *  *  *  *

The Joel Test updated for the year 2010. And a few more months to get it right.

Posted on Tuesday, June 16, 2009 4:23 AM Architecture , Programming , configuration management | Back to top

Comments on this post: The Joel Test – Updated for 2010

# re: The Joel Test – Updated for 2010
Requesting Gravatar...
What are good requirements management systems?
Most of the ones I have seen are horrible.
Caliber-RM seems to be the nicest one - but few requirements systems seem to deal well with graphics intensive, prototype driven requirements.

Left by Christian Mogensen on Jun 17, 2009 5:17 AM

# re: The Joel Test – Updated for 2010
Requesting Gravatar...
I don't know... the beauty of the Joel test is it's reflection of simplicity vs effectiveness. The tasks it lays out aren't difficult to accomplish but provide fantastic returns. I think it's important to maintain a level of simplicity so-as not to get bogged down in the attitude of "we don't do that because it's too complicated/we're only a small team".

1. The test here is that there is some way of managing and tracking change, as opposed to all developers working on the same set of files. Stating that SCM is nothing more than a glorified backup system is perhaps doing it something of a disservice. While everything you mention is admirable, I get the feeling that a lot of people would balk at the long list of requirements they need to pass what should be a relatively simple case of being able to say "we can track what's changed, who's changed it and roll-back if needs be". There's nothing wrong with implementing everything in your list, and I'm sure it's something that teams should strive for, but the barrier to entry is significant.

2. I think this is possibly more a clarification of wording than anything, as the way you state it is how I read the original point.

3. Absolutely spot on, especially for .NET code written using Visual Studio, there should be no reason not to at least use the testing projects that come built-in to Visual Studio.

4. Again, while admiral, I'm not sure this should be essential to passing the Joel test. Having a method of tracking bugs is of vital importance, and adds significant value over not tracking them at all. Indicating whether a team tracks bugs gives a good indication of how aware they are of the quality of their software, adding source control integration just adds an extra level of granularity to that. I'm just not convinced that it's as much a reflection on the team as the original "should have bug tracking" is.

5. While this is true I think it changes the meaning of the test. Originally the test was an indicator of whether the team valued quality and recognised the importance, and ultimate saving in time, of fixing problems as early as possible. The updated test seems more centred around their mastery of their source control of choice. Everyone recognises that bugs need fixing, and branching is a valid method of controlling that change, but it doesn't check for the attitude that quality is achieved by fixing problems early and often.

6. To me this is just a re-wording of Joel's original statement. An up-to-date schedule reflects change...

7. I agree with this and think that perhaps it encompasses part of what you were trying to state in point 6, that you need a way of recording that a change from the original plan has occurred?

8. I like this and think it would be ideal; I'd love to work somewhere that did this (or passed this point at all).

10. I'm not entirely sure where you're going with this, I think it could do with further clarification... I agree on testers being able to reject a deployment but I'm just not clear about the requirements part.

11. I agree with this as well, although I might just tweak it to be "write and review"; nothing wrong with having them write FizzBuzz then if they pass that get them to review some actual code.

All in all, I quite like you updates, but think some of them could do with being toned down just a bit to keep the same measure of simplicity vs effectiveness. A test that's easy to remember, and captures the heart of what you're looking for in an ideal job. Perhaps things like having a change management system should be the *next* 12 steps that need to be taken once the first 12 have been accomplished?

Left by Dave on Jun 17, 2009 8:13 AM

# re: The Joel Test – Updated for 2010
Requesting Gravatar...
Christian – Thanks for the comment. I have not found a good requirements management tool. My updated Joel Test was intended to be more about process that tools on that issue.
Dave –
Nice comment. I would have to say that you have brought to light that my updated Joel Test loses some of the simplicity and “go/no-go” flavor of the original. That was not intended, but I have a few more months to work on it :-)

I would stand by my item #4, however, others have commented to me that some organizations do not track bugs. I must say that I do not know of any organization that does not have a bug tracking database, so I assumed the original Joel Test criteria was in need of an update in this area. I could be wrong. But I still cannot imagine an organization that does not have a bug database.

For item #1, note that I said “even if it’s not to branch” – there are cases where simple, down the trunk, checkout/checkin no labels, no tags, no branches is OK, but it is only OK if the team has decided to work this way and it is not simply the “default for all projects”. Only in the latter case is the SCM a substitute backup system.

For #10, I am trying to make the point that brining in testers at “test time” and asking them to “make sure this works” is much different than establishing a test team at the initial phase of the product, have them involved in requirements gathering, have them develop test plans/approach during development, and finally to execute tests. More often the testers execute tests against the product as opposed to against the requirements, and code goes out the door that does not do what it’s supposed to do yet still pass the tests. The side-effect here is that requirements are elevated in importance because they are now directly related to validation/tests.

You have made lots of good points and given me some things to think about. Thanks again.
Left by Bill on Jun 17, 2009 5:35 PM

# re: The Joel Test – Updated for 2010
Requesting Gravatar...
This is really a nice update.

Don't know why I foresee this to be the de facto standard for the coming decade.

Thank you
Left by Raghuraman on Jun 18, 2009 8:17 AM

# hallway usability test
Requesting Gravatar...
Thanks for the updated version of the Joel Test!

I think #12 is a bit redundant and too specific at the same time. If there is a testteam (#10) it should cover all kinds of test types in order to verify and validate the software, not only UI testing.

A "hallway usability test" is imho only applicable if the guy from the hallway is in the target audience.

Furthermore there are projects where usability is not a major quality characteristic.
Left by FuePi on Jul 28, 2009 7:59 AM

# re: The Joel Test – Updated for 2010
Requesting Gravatar...
I'd like to see #3 include continuous integration (or maybe another question?) Having quick feedback when you've broken the build is even more useful than finding out the next morning, especially for international companies where there is no "next morning."
Left by funtech on Oct 17, 2009 1:04 AM

# re: The Joel Test – Updated for 2010
Requesting Gravatar...
How about:
"Do you perform daily builds or use continuous integration"

A continuous integration process for most projects is easily acheivable, however, there are cases where it takes very long (hours) to perform a complete build (such as when there are thousands of artifacts creating hundreds of deployable assemblies and data files). I am trying not to make this list be prescriptive; but I like the idea of including the term "continuous integration" in the test.
Left by Bill on Oct 17, 2009 6:59 AM

# re: The Joel Test – Updated for 2010
Requesting Gravatar...
9. Do you use the best tools money can buy?
No change needed. This one is timeless.

I disagree - In the world of open source software and free (both gratis and libre) libraries such as boost are helping developers.

I would rather rephrase it to remove the "money can buy" part.
Left by LiraNuna on Feb 06, 2010 3:49 PM

# re: The Joel Test – Updated for 2010
Requesting Gravatar...
Agreed. Best tools available.
8. Do you use the best tools available for the job?
Left by bill on Feb 06, 2010 4:38 PM

# re: The Joel Test – Updated for 2010
Requesting Gravatar...
I would argue that it doesn't need updating.

The test is a tool to let you discard companies completely that make little effort and not waste your time proceeding to the interview stage. The specifics do not matter at that stage and the original test works fine.

As a second pass/post interview stage when you have narrowed down your options this is useful, but should not replace the original test.
Left by Dan Brown on Feb 17, 2010 7:08 AM

# re: The Joel Test – Updated for 2010
Requesting Gravatar...
Bill wrote:
"I could be wrong. But I still cannot imagine an organization that does not have a bug database."

Bill, believe me, you are wrong. I know this from bitter personal experience, not just once, but a number of times.

I agree that it's absolutely incredulous that an organisation developing software can exist and operate without this most basic of tools, yet they do. Albeit badly.
Left by WhatIsBug on May 30, 2010 3:47 AM

# re: The Joel Test – Updated for 2010
Requesting Gravatar...
I would agree with Dave that Joel's entire point was to fix bugs before writing new code, Joel was not referring at all to the complexity of branching/merging as the reason for fixing bugs first.

Left by Ed on Jun 13, 2010 7:56 PM

# re: The Joel Test – Updated for 2010
Requesting Gravatar...
Y'know, why mess with such a classic? Years later I don't think the Joel Test needs an update. Your modifications are addressing some of the finer sub-points to each major tenet of the JT -- but the whole point of the JT is to get a quick check of an organization's effectiveness. How many organizations have all of the JT points in even a basic form? Very few!
Left by Mark A on Sep 10, 2010 11:34 AM

# re: The Joel Test – Updated for 2010
Requesting Gravatar...
Nearly all your changes show that you didn't get it right what was meant from joel
Left by Manuel on Oct 16, 2010 11:00 AM

# re: The Joel Test – Updated for 2010
Requesting Gravatar...
Something I really liked about the Joel test is how easy it applied to projects of all sizes. Every software shop needs Source Control, we all need bug tracking, we all need to be able to quickly do a new build.

This revised list I feel doesn't apply as well to most situations. Smaller shops and teams can't as easily identify with some of these reworded bullets.

Ex: Do you have a change management system? What is a change management system? Now, source control on the other hand everyone gets.

Don't get me wrong though,Ii definitely agree with a lot of the things you mentioned. I would just rather keep the list as easy to digest as possible.

Good post
Left by Brian Wigginton on May 12, 2011 12:33 AM

Your comment:
 (will show your gravatar)

Copyright © Bill Tudor | Powered by: | Join free