Geeks With Blogs
Matt Watson Software developer, product visionary, and master of #dadops
One of the biggest challenges for developers is getting access to production servers. In smaller dev teams of 5 or less people everyone usually has access. Then you hire developer #6, he messes something up in production... and now nobody has access. That is how it always starts. Just about every rule in life gets created this way. One person messes it up for the rest of us. Rules are then put in place to try and prevent it from happening again.



Breaking the rules is in our nature. In this example it is for good cause and a necessity to support our applications and troubleshoot problems as they arise. So how do developers typically break the rules? Some create their own method to collect log files off servers so they can see them. Expensive log management programs can collect log files, but log files alone are not enough. Centralizing where important errors are logged to is common. 

Some lucky developers are given production server access by the IT operations team out of necessity. Wait. That's not fair to all developers and knowingly breaks the company rule!  When customers complain or the system is down, the rules go out the window. Commonly lead developers get production access because they are ultimately responsible for supporting the application and may be the only person who knows how to fix it. 

The problem with only giving lead developers production access is it doesn't scale from a support standpoint. Those key employees become the go to people to help solve application problems, but they also become a bottleneck. They end up spending up to half of their time every day helping resolve application defects, performance problems, or whatever the fire of the day is. This actually the last thing you want your lead developers doing. They should be working on something more strategic like major enhancements to the product. Having production access can actually be a curse if you are the guy stuck hunting down log files all day. 

Application defects are good tasks for junior developers. They can usually handle figuring out simple application problems. But nothing is worse than being a junior developer who can't figure out those problems and the back log of them grows and grows. Some of them require production server access to verify a deployment was done correctly, verify config settings, view log files, or maybe just restart an application. Since the junior developers don't have access, they end up bugging the developers who do have access or they track down a system admin to help. It can take hours or days to see server information that would take seconds or minutes if they had access of their own. It is very frustrating to the developer trying to solve the problem, the system admin being forced to help, and most importantly your customers who are not happy about the situation. This process is terribly inefficient. 

Production database access is also important for solving application problems, but presents a lot of risk if developers are given access. They could see data they shouldn't.  They could write queries on accident to update data, delete data, or merely select every record from every table and bring your database to its knees. Since most of the application we create are data driven, it can be very difficult to track down application bugs without access to the production databases.

Besides it being against the rule, why don't all developers have access? Most of the time it comes down to security, change of control, lack of training, and other valid reasons. Developers have been known to tinker with different settings to try and solve a problem and in the process forget what they changed and made the problem worse. So it is a double edge sword. Don't give them access and fixing bugs is more difficult, or give them access and risk having more bugs or major outages being created!

At your company who caused this rule to be put in place? How do you get around it?

Matt Watson
Founder, CEO
Agile Support for Agile Developers

Posted on Saturday, September 22, 2012 10:01 PM | Back to top


Comments on this post: Production Access Denied! Who caused this rule anyways?

# re: Production Access Denied! Who caused this rule anyways?
Requesting Gravatar...
Our no production access rule actually comes from our interpretation of http://en.wikipedia.org/wiki/Sarbanes%E2%80%93Oxley_Act . Being a financial company we don't have a choice in the matter.
Left by Matt on Sep 23, 2012 1:44 PM

# re: Production Access Denied! Who caused this rule anyways?
Requesting Gravatar...
For me, and developers, it stems from the fact that they have read only accounts to slave databases, log collectors that aggregate to a searchable web UI, and processes to ensure that only committed code gets deployed to production.

They shouldn't be on a production, because we do give them limited access to staging (read only) and full access to sandboxes (which resemble production in that I use the same code I use to make the prod servers with it).

They shouldn't have production access, because they shouldn't need it.
Left by VJ on Oct 01, 2012 10:45 PM

# re: Production Access Denied! Who caused this rule anyways?
Requesting Gravatar...
To get access to remote servers I would recommend a tool like Stackify. You can use it to manage and monitor your servers very easily. Makes remote troubleshooting really easy. Check out http://www.stackify.com/remote-server-access-developers/
Left by Matt on Feb 20, 2013 10:10 PM

# re: Production Access Denied! Who caused this rule anyways?
Requesting Gravatar...
Whatever the challenges in their work, developers can somehow go through it successfully. - Marla Ahlgrimm
Left by Kyle Richards on Jan 27, 2017 5:15 AM

Your comment:
 (will show your gravatar)


Copyright © Matt Watson | Powered by: GeeksWithBlogs.net