I've had a little frustration lately with Team System as a result of not using the complete Team Suite version. When the licensing for Visual Studio changed last year to the role-based versions and/or Team Suite, we considered the options and decided that VSTS For Developers should be all we need. We do not do any web or load testing (just unit tests), and we do not have a need for the designers available in the Architect version.
Since then that has held true for the most part. Except for some important scenarios. Most of my problems arose when I tried to get a Team Build going. I managed to get the build itself running. But then I wanted to enhance it to run our tests as BVT's (Build Verification Tests). Based on what I'd read, I had believed this was possible. After all, creating unit tests is part of the capability of VSTS For Developers (you just have a limited amount of control over how you organize them compared to Team Test). Well it turns out that it's not so easy...in fact it doesn't appear to even be possible. I installed VSTS for Developers on my build machine, and updated my Team Build to run some tests. But I soon discovered that in order to have the Team Build run some tests you need to specify the test list to use (all I want to do is have it run ALL my tests...why should I have to create a test list for this?). Well creating test lists is a feature limited to Team Test and not available in VSTS For Developers. This was confirmed by Eric Jarvi in this forum post: http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=609672&SiteID=1
It turns out that their is no official support for creating a Team Build that includes BVT's if all you own is VSTS For Developers. I believe it may be possible by manually editing the .vsmdi file which is just XML and creating a test list in raw XML. Rather than mess with that, I decided to just fire up a VPC, install Team Suite Trial, grab a copy of my project from source control, add a test list, then check it all back in. Then I should just be able to use that test list for my Team Build right? Sure I'd have to go into my VPC and add in all my new tests every once in a while, and what would I do once the Trial expired, but at least I could get something running right now.
Well sadly not even that would work. Once I checked it in from the Team Suite edition and tried to run my team build, now it is complaining about all kinds of test related dependencies that aren't available on the build server (because it only has VSTS For Developers). I played with this for a while, trying to manually edit some of the XML files to remove the dependencies, but after an hour or two of trying I finally got frustrated and backed out my changes in source control.
I still don't have a Team Build running that will run my unit tests. Before doing anything drastic like abandoning MSTest and Team Build, I have another strategy I want to try. I think I might be able to modify the MSBuild script used so that it invokes MSTest directly. Failing that, I'm pretty certain I can use CruiseControl and MSTest.exe as a CruiseControl task. But in that case I think I might lose the code coverage data.
Anyways, I had somewhere I was meaning to go with this rant. Most of these difficulties could have been avoided if I was using Team Suite everywhere. I'm sure many others have experienced similar frustrations or confusion as to what they can and cannot do with the role-based versions of Visual Studio. My suggestion to Microsoft, is why not do away with the role based versions. Drop the price of Team Suite to somewhere between what the cost of a role-based version is and Team Suite is today (closer to the role cost than the suite cost). What you lose in margin you should make up for in volume. Sure the price to customers will go up a bit, but I think the added cost will be worth eliminating the frustration from trying to get the role versions to do something they weren't meant to do, and the confusion over what it is each one is meant to do and is not meant to do.
I can foresee this problem getting worse once VSTS For DB Pro's comes out. What if I buy VSTS For DB Pro for my DBA, and VSTS For Developers for my devs. I have unit tests that my developers write that hit against the database, these unit tests need to deploy the db schema as part of the test. Will they be able to compile and deploy the database from the dbproj within VSTS For Developers? It doesn't appear so (at least not until a new version of VSTS For Developers is released). I understand not allowing them to maintain the schema, you should need VSTS For DB Pro's for that, but the developers will need to be able to deploy that schema in order to run their unit tests. And to deploy the schema, they should probably also be able to build the dbproj to generate the latest version of the schema. Even if you ignore the database deployment with unit test issue. My understanding of MS's vision (and I share in this vision) is that every dev should be running a local instance of the database, and the master version is kept in source control and maintained by the DBA. Well when the DBA makes updates to the schema how is the developer supposed to deploy those changes to their local database? My understanding is that VSTS For Developers won't be able to recognize a dbproj at all, let alone compile and deploy it. Will it be the DBA's responsibility to deploy the database to every developers SQL instance? How will he know when it is safe to do so? Will the dev's have to explicitly request the DBA to push out the changes on a workstation by workstation basis when they are ready for a new version? Not only that (and I may be completely wrong here), but isn't SQL Server Developer Edition restricted to only support local connections. If so the DBA wouldn't be able to remotely deploy the database to the dev workstations even if he wanted to.
Don't misunderstand me, I am a HUGE fan of the new team system functionality. If I wasn't I wouldn't get so worked up over these issues. I'm just starting to feel some frustration with limitations that I don't agree with, and I only see it getting worse down the road if something isn't done to address/improve the issue. If anybody from the VSTS team is reading, please consider this feedback and think about ways to avoid these problems. As I said above, I'm all for eliminating the role-based versions and adjusting the pricing appropriately so everybody is either on VS Pro, or VSTS Team Suite. If that's not an option, then at a minimum consider what common scenario's teams will be trying support when they are using the role-based versions and make sure that enough functionality is included in each version to support those scenarios. Especially when VSTS For DB Pro's becomes available, I foresee some major hurdles for teams that are using VSTS For Developers for the devs and VSTS For DB Pro's for the DBA's. To solve the Team Build issues, another possible solution is to provide a special VS install for build servers, that doesn't support development activities, but can support Team Build's that include the features from all the different role versions (build dbproj's, run tests, etc).