While I am a huge fan of cloud and agile when they are done well, as an architect I am very cautious about some of the dangers which can result from the way good architecture can be side stepped by the "just get it done quick and simple mentality" which agile sometimes falls into and that cloud can make it easy to happen.
Anyone who knows me well will know that as an architect I am a big fan of a decision making framework where decisions are made by looking at the options in terms of delivery, design, operations and organization. In this space my dislike of the way agile architecture decisions are sometimes made is that in my experience most of the time:
- The only area considered is delivery. Design, operations and organization aspects aren't analyzed or considered
- The decision is rarely documented so we can't review the decision making process at a later date and understand the motives behind that decision.
So we are in a position where in the real world agile is allowing teams to get away with poor decision making but fortunately often on premise constraints can put the rains on some decisions. A telltale sign of this is often conflict between agile development teams and security or infrastructure teams. This is where cloud comes in.
In the cloud developers often have the ability to spin up new stuff without any real checks and balances about what they are doing. In most cases the only real obvious measure of the impact of this is the bottom line cost of your cloud investment, but also the billing models for cloud providers are still quite immature so the identification of any new resources isn't always obvious. There is also the pay as you go billing aspect which means that any new resource has its cost split over a period of time making it less obvious.
To summarize what I'm saying so far. Agile development teams often make delivery focused architecture decisions without enough consideration for other aspects. The cloud has typically removed some of the constraints on teams allowing the team to implement these decisions much more easily. The cloud billing models mean that you might not notice the impact of these decisions because rather than an upfront cost it would be spread over a longer period of time and also harder to identify.
That's the observation I have made and I see this being a general issue facing the IT industry over the medium term. Now let's look at an example to back up what I am saying.
We have a project on going at the moment were there was a requirement for a temporary storage requirement in the cloud. Early in the foundation sprints this design decision was looked at and after some discussion it was recommended to use the azure table storage.
Later in the project the scrum team implemented a solution with SQL Azure instead. The key to this implementation was that they had already prototyped this and it was easy to implement and get it finished by the end of the sprint. It's something when this was identified there was apparently no time to refactor this without having an impact on the delivery schedule. If you're technical you are probably assuming at this point that it would only be a relatively small amount of work to refactor this and I would agree with that. If it's not then again as an architect this is another yellow warning light that there are more problems.
As a non-technical person you're probably thinking "if it works so what"!
Well let's look at the implications of this decision. Probably the best way to do this is to consider the overall cost. Let's say that the cost to refactor to use table storage would be a very generous 3 days development and 3 days retesting. Even though I'd expect the solution to be behind the scenes and covered by normal regression testing but let's argue 3 days at £250 per day which is £750.
Now let's consider the cost implications of the approach implemented. Firstly let's ignore the cost of what's been implemented so far, what's done is done. Let's focus on what's left.
Licensing & Running Cost
To begin with there is licensing, this solution requires a SQL Azure database at present which costs £30 per month per environment for about 10GB and about £7 per month for 1GB. If we said that we think we will store around 10GB of data most of the time and there is production, 7 x test and 2 x development environments which is a ball park estimate for the environments plan we would be looking at something in the region of £300 per month. If we take the assumption that the solution will live for 5 years then we are talking approximately £18000 over the course of the life of the project. We also need to consider the size of the database because if it goes over 10GB then the cost would increase. There would probably be a maintenance/cleanup process to clean up un-needed data.
The table storage solution costs approximately 1p per GB per month. We can have as many storage accounts as we like and just pay per size for storage. If we assume that the data is 10GB ongoing then we will have a running cost for the same scenario of 10 environments would be approximately £1 per month.
Over the 5 year lifecycle then the approximate cost comparison would be:
- SQL Azure = £18000
Table Storage = £60
Knowledge Transfer Cost
The KT cost for the SQL Azure solution would be significantly higher than the table storage one. The SQL Azure solution would need review by a DBA and the organization currently doesn't really have experience with SQL Azure so there is likely to need to be some kind of training requirement. For table storage it's a much simpler concept and the KT for this could be done in a much quicker and lower cost way. The SQL Azure solution should really have a more sophisticated monitoring solution and needs closer review by a DBA.
For the specific scenario here there is the potential with table storage that we might never need to consider the archiving option. The cost is so cheap and the data retained if we didn't archive means that the cost of implementing the archive solution would probably outweigh the overall cost saved by doing it. With these levels of data performance wouldn't be affected either way.
In the SQL Azure approach there would need to be a more considered archiving or cleanup approach, the cost/size side of the database is much more significant for the SQL Azure solution at the wider scale.
Hopefully you can see that in this example a purely delivery focused approach to the architecture decision means that we will reach the end of the sprint the team will be able to do a nice show and tell but because they have not considered the bigger picture the longer term life of the project will be affected very significantly.
Using the cloud allows the team to be able to make this kind of poor decision without the checks and balances which a more mature governance model would provide and there would be a fair chance that the cost implication of this would go relatively unnoticed.