Robert's Sysadmin Blog

So here is something silly I was running up against. In the end its super simple, but its not obvious, and not easy to google for.


I want to equip the new servers are are installing with a standard weekly reboot schedule.

I created a batch file that launched shutdown.exe with some fancy parameters, and set this up as a scheduled task for each server.
I created a special domain account called sa-scheduledreboot with normal user rights, and rights to access the share, and of course the famous "log on as a batch job" privilege, granted to each server via Group Policy.

But dispite this, rather textbook, rights scenario,  I was continuously getting "Could not Start"

However, if I ran the command using Runas, using the credentials of the sa-scheduledreboot account, it would work fine.

The Scheduled Task eventlog showed the following:

"Task Scheduler Service"
5.2.3790.3959 (srv03_sp2_rtm.070216-1710)
"Sheduled Reboot.job" (Reboot.cmd) 5/13/2008 5:43:54 PM ** ERROR **
    Unable to start task.
    The specific error is:
    0x80070005: Access is denied.
    Try using the Task page Browse button to locate the application.


I spent all several hours trying to find out where the "access denied" came from. Eventually, I stumbled apon this:

http://support.microsoft.com/kb/867466/en-us

as it turns out:

In Windows Server 2003, the Users group does not have Read and Execute permissions to the command processor (Cmd.exe). By default, the Cmd.exe program has the following permissions settings:
•    The Interactive implicit group and the Service implicit group have Read and Execute permissions.

Note On a member server, the TelnetClients group also has Read and Execute permissions. On a domain controller, the Batch implicit group also has Read and Execute permissions.
•    The Administrators group and the System implicit group have Full Control permissions.

One of those quirky things you just have to know.

The way I have solved this, is that I have created a special Domain Local security group called RG_command_processor_execute  (RG stands for Resource Group)

This group will allow me to control this specific privilege, and assign it to accounts, usually service accounts, that require the access to cmd.exe to run batch files.

I have added sa-scheduledreboot to this group.

I dont want to mess around on each individual server, so I have made it standard that -all- security settings, including changes to default ACL's, should happen via Group Policy.

For this we use the File System section of the Security Settings part of a Group Policy Object.
We can add files and folders here, and define how their ACL should look.

The tricky bit is that you have to remember that this Group Policy setting overrides and replaces the original ACL on the object.

Thats a bit annoying, cause it means I have to replicate its current ACL's, including any special permissions assigned to implicit security groups. 

The KB article shows two ways to do this.
The first is to add the account or group directly to cmd.exe. ACL
the second is to add the BATCH group to the cmd.exe ACL

The second option is interesting, because the BATCH built-in group implicitly includes all batch files that run on the system.

The way that would go would be:

sa-scheduledreboot -->member of--> RG_command_processor_execute -->member of--> %hostname%/BATCH -->applied to--> (ACL of) cmd.exe


This looked like a good option for a while, until I realized it was perhaps a bit broad. (all batch files, including those run by rogue processes? )

And since it only applies to batch files, if I ever needed to grant anything other than a batch file (say, a resident program or agent), that right, I would have to assign the group directly anyway.

So I decided to add the group directly to the resource, which also makes it easier to see what the ACL change is for, for anyone examining the GPO.

sa-scheduledreboot -->member of--> RG_command_processor_execute -->applied to--> (ACL of) cmd.exe

The scheduled reboot command works fine now. And I am confident I did not assign any more rights that I absolutely needed to to get it to work. (In contrast, the previous reboot account had domain admin rights).

The only thing I need to do now, is to remove many other rights from the sa-scheduledreboot service account.
Its currently a member of Domain Users, and that grants a load of rights this account certainly does not need. I will look more closely into that at a later time, as my solution will have to cover many service accounts, not just this one.

By giving out the exact rights needed in a very granular way for each service account I need, I can far more easily restrict ALL service accounts in other ways, all at once, making them useless to use for any other purpose than what they where intended for.

Documenting this is gong to be a challenge.

I need to document exactly what I am doing in the GPO that assigns the rights to these servers, and why each option was chosen the way it was.

I need to document the exact rights of the sa-scheduledreboot

And if I develop a blanket method to restrict ALL servuice accounts in other, general ways, I need to document that too!

I better get to it!

I pinged Loic le Meur, creater of Seesmic and owner of Twhirl on Twitter to draw his attention to my post on some of the issues i have with Seesmic.

He responded as follows:

“you have great points and we are working exactly in the spirit you expect”

I will take this to mean that we can expect groups/filter function in the Twhirl client or the Seesmic service in the near future. I cant wait :)

 

So I have been trying to find and add other System Adminitrators on both Twitter and Friendfeed.

I am a bit picky though. I looked for people that seemed to Tweet at least some of the time about their work, tweeted regularly, and in English. Also preffering Windows Sysadmins over Unix for now, but I might reconsidder that.

So far the results have been good, and with results I mean that I can get little conversations going about tech stuff.

What I would love to see happen at some point, is a discussion where multiple of these guys get involved. Its not grown to that level yet, and I am not sure if Twitter lends itself well for that, as the dicussion is public and all your follows get to “enjoy” it.

This brings me to current BIGGEST annoyance about Twitter and Friendfeed (and Seesmic, to an extent)  The total lack of any kind of groups feature.

Now it would be nice if Twitter supported groups, and made that stuff available via the API so clients like Twhirl can use it. But to be honest, Twhirl and Alertthingy could just as easily build in group support themselves.

That would have the added advantage of applying to any other service they choose to support. I already suggested this to Howard Baines of Alertthingy, and he found the idea “interesting” but its not high on the to-do list.

With groups, you could, at the very least, sort your “friends” into groups of your choosing, adding a powerfull filter to the lifestream that comes in.

Conversely, if Twitter itself supported this, perhaps it would be possbile to Tweet to just the members of a particular group. This would solve the above problem of irrelevant tweets being recieved by followers that might not be interested in the subjet.

It would make the experience overall more valuable and encourage more discussion.

Seesmic currntly suffers from the same problem. There they have the added issue of the focus of content flow still being mainly about the main public feed of all videos people post.

This is a leftover from whem the Seesmic community was very new and very small, but that is eroding now as the service gains users and the public feed becomes inpossible to follow.

However, many people there, especially of the old gard,  still feel the need to “discuss” any and all videos crossing the public stream. This might well include any video I post that is directed at Sysadmins.

Its has been my fear of spamming these people and getting low-quality feedback from them, that prevents me from using the service much currently.  

However, this is changing very fast with the brilliant move by them to produce blog plugins that allow video commenting. My blog, as well as big ones like Techcrunch now support these, even though they are not used much yet.

It was interesting to note that they deliberately are not including the comment videos in the Seesmic public feed. But they are including all the blog posts that people make, using the same plugins.

This is quickly going to make the main public feed unfollowable, much like Twitter, and I considder this a good thing.

Like Twitter, the faster the usage model of Seesmic changed to revolve around you and your own followers, and those who you follow, the faster the update will be.

The reason this is not happening already is because the user base is still too small, and the service is still closed alpha. I cant, for example, find even as much as 5 of the people I follow on Twitter and Friendfeed on there.

Once they open up to public beta, the influx should quickly re-arange the usage and then I will be using it a lot more.

Now to convince all the already aloof Sysadmins to start recording video of themselves…   lol .. thats a differnt problem alltogether ;)

 

Picture on Flickr
High-res version

This is version 2 of the rack plan, as it currently stands. To make sure we have enough room for everything that could (potentially) be moved in, I moved things about a bit from where we are now, means we need to re-rack 2 servers tomorrow, but other than that we are good.

I am really hoping we get to re-use the HP Storageworks tape library, I am not sure at the moment what type it is specifically.

I also dont know yet how much room the Advocent Switchviews have taken since they where installed this week, I think perhaps 1 extra U I need to clear above the KVM screen. I will also need some more shelves for some of these systems. 

I posted the following post on The Server Room forum at Ars Technica

—————————————————————

Hi all,

Maybe I just haven't looked hard enough, but I get the distinct impression that sysadmins, for some reason, are not very community-oriented.

If you look online for developer communities, you find a fantastic amount of forums, websites, and blogging communities.

When I wanted to start blogging about my sysadmin adventures, I looked for communities similar to developer sites like geekswithblogs.net, Channel9, asp.net. However, I couldn't find any kind of hub that revolved around systems administration in a similar way.

We just seem a bit under-represented online, imo.

I want to find places I can converse and chat with fellow sysadmins, besides, obviously forums like these (which rock, btw).

For example, I did a search on Twitter and found a whole bunch of people that described themselves as sysadmins. Now those are exactly the kind of people I would like to follow on Twitter.

Similarly with Flickr, there are a number of small groups that deal with sysadmin and datacenter stuff, and I regularly contribute to those groups, but there are not many people there.

Another thing I would really love to see, is a good IRC community in this space. Again, perhaps there already is one that people here know of, I just haven't found it yet.

What constantly suprises me, is how, whenever I find some kind of sysadmin community "hub", the way collaboration is encouraged is incredibly old-school. For example mailing lists, or usenet groups. Now I am not saying there is anything wrong with that, but I hardly find anything in the web2.0 scene at all. Have a look on Facebook, how many large sysadmin groups do you find there for example? Twitter? Friendfeed? I would really love to have a few more sysadmins on Friendfeed! We appear to be under-represented online, and very fragmented.

So lets get connected! I will list some of the resources/communities I have found, I hope you will add your own. What I am trying to collect is a list of active community hubs. I will update this post with the links people add.


Forums:
Ars Technica - The Server Room
Sysadmin Talk Forums


Social Networks:
Flickr - Systems Administrators Pool
Flickr - Rackmonkeys Pool


Clubs/Organisations/Conferences
LOPSA - the League of professional system administrators
SAGE - The Usenix Special Interest Group for Sysadmins
LISA - ( Large Installation System Administration ) Conference, organised anually by Usenix/SAGE

Podcasts/Vidcasts/Screencasts
Podcast - Casting from the Server Room

————————————————-

I am secretly hoping the post will get stickied or something. Regardless, it gave me the idea to build it out more. I can add these links to my sysadmin blog. But it also reminded me I really need a Wiki or something, that would be even better.

Damnit, time to start migrated jemimus.net over to some proper hosting. Come to think of it, I need a new Wordpress theme too!

 

So it turns out, we are gonna move all the servers to 1 location. Expect a new rack diagram soonish ;)

I hate politics.

Moving it all to 1 place is both a blessing and a curse. Naturally, its logistically easier for us. But it also means we are facing potential power and airco issues at the single location.

Moving to the 1 location was a political desicion, and not made with the best of technical considerations in mind.

On top of that, the big Citrix migration project, that we are depending on being finished by the time we move the bulk of our servers, is looking less and less likely to complete on time.

This means, in short we will be moving a lot more servers and equipment we are currently scoped for. I would very much like to prepare for the worst case, but politics are getting in the way again.

For example, I originally planned to move at least 1 IBM Bladecenter along with our NAS to the new location. These are needed to support the Citrix farm, in case the Prague project doesnt finish in time.

 The Bladecenters

However the project steering group told us this was not in scope, as the Prague project have indicated they would be finished, even though everyone knows they will never make it.

The reasoning is mostly to do with money. Moving the Blacecenter and the NAS, would place a power burden on the new TCR, that would require a more powerfull generator. These are fucking expensive.

So if the Prague move project -claims– they will finish end of May, then why spend thousands of euro’s on a new generator that wont be needed.

Well, because, dear project stearing group, the Prague migration project -wont– be finished on time, and guess what, your gonna have to buy it anyway, so lets by it now and give ourselves some breathing room.

I dont really care about the politics. All I know, is that when push comes to shove, its gonna be US that do all the hard work.

I dont understand this insistance to stand on principle, with all the risks associated with that. Why not play it safe, we are talking about all the Benelux operations of the company that are at stake if we dont mitigate some of these risks.

I cant understand their thinking at all, it seems insanely risky and dangerous to me.

To make things even more silly, there is now talk of at least one of the WMS’s (warehouse management systems) that the Prague project was supposedly gonna migrate we are going to have to move.

The only reason is that, apparently, they are starting to see they wont make it in time.

 The Alpha server running one of our many WMS’

The really funny thing, is that that particular WMS runs ons an Alpha box, with a similair power requirement to the Blade Center.. taking the serverroom over the powerlimit too! Whahah!

Anyway, back to the technology

The new HP servers and racks + options arrived, and the last week has been spent building it all up.

This is me at one of the 3 sexy HP TFT7210R 1U console options.

 

Cables are still messy, its al temporary till our network guys can put in the new core. Some of that will be happening tonight!

I labeled all the servers, and set up the ILO cards with the advanced licences, and gave them static IP adreses in a new management VLAN we created.

Currently we are on a temporary switch, and only room enough to hook up 4 servers at at time. Should be better after tonight when they bring the new core online.

I spent the last 2 days documenting stuff, deviding the licences, and trying to install the OS on these machines remotely, using ILO.

This didn’t go according to plan. At all.

I posted about this on the new Ars Technica forum “the Server room”

From my thread “ad-hoc Remote Windows installation stategies using ILO”

We have set up a number of servers in a remote location. They are all HP DL360 G5's with advanced ILO licenses.

Now this works great, but when I tried to remotely install, I run into problems.

I can mount an ISO of the SmartStart CD, and it boots as it should, its a little slow loading of course, but I can get through all the config steps.
But at the screen where the SmartStart says it is copying files to server, it often hangs, or at least, takes forever.
On the rare occasion it gets past this, it wont, for some reason, recognize the Windows OS installer ISO image.

Now both problems might be related to latency, or to a limitation in the way the ISO files are being mounted through the ILO.

Perhaps what I am doing is not supported (I know it isn't with IBM), but I have not found anything in the Smartstart or ILO documentation that says I cannot do Windows OS install remotely.

So, if I cant figure out why this isn't working, I am going to have to build some alternative way of remotely installing Windows. Probably via a distribution share, and network boot media like BartPE or something similar.

I don't have any commercial product available for OS imaging, unfortunately. And don't have the timescale to purchase these either.

Nor can I use a PXE boot option at this time, because of network limitations. And even then, i dont have time to set up RIS.

What are the kind of solutions you employ for remote OS provisioning?

Well I have gotten a number of responces so far, none of them very helpfull. By best bet is that latency us causing the issues, but I dont have any hardware at location just yet, that I can set up as a distribution point.

I will elaborate on the way I want to use ILO in a sperate post. I have yet to figure out the best way to deal with this.

In the meantime, tomorrow, I will be going to the location and starting the OS installs manually. That at least is a sure way of getting them installed.

Oh, finally, want to see why we need to move in the first place?

This is through the window, the office next to ours.

See the cable tray that was previously in the ceiling? The one marked with the red/white tape?

Thats all the copper and fiber of the current datacenter going through there. I am scared shitless they are gonna damage them while they strip the building.

 

Ars Techninca recently opened a new forum called “The Server Room”

The first thread I started there was about sysadmin and datacenter books, and I already have some real jewels that I will be ordering soon, I suspect.

From Cisco Press:

http://www.amazon.com/Center-Facility-Business-Networking-Technology/dp/1587051826 ( http://www.ciscopress.com/bookstore/product.asp?isbn=1587051826 )

and this one was recommended to me too:

 

http://www.amazon.com/o/ASIN/0201702711/tomontime-20 ( http://everythingsysadmin.com/ )

 

 

 

Ok, so this is the first post in a long while on this blog.

The reason I have started blogging again, is cause I finally have something interesting Techie to blog about – namely our datacenter move. I hope this will become a series of posts, over the next few months, relating to our move.

Our datacenter:

– 100+ Citrix servers
– Buch’O infrastructure server
– About 20 application servers
– About 8 web/FTP servers
– About 6 IIS/Web servers
– 2 IBM Blade Centers
– Bunch’O Unix servers
– About 6 SQL servers
– Lots of Cables
– Lots of Switches

Most of it is super old.
Some of it is nice and new and shiny.

Check out pics of the stuff here

          

 

So here is the story so far:

Now most of the servers, or I should say services are being moved to the big regional datacentre in Prague. This includes all Citrix servers. That project, lets call it “project purple” hopes to complete by end of May. We know they wont make it however.

Regardless, there will be plently left to to move anyway, not even counting what project purple leaves behind. And each of the servers and systems we need to move, present their own unique chalenges and problems.

In this series of posts, I shall dive into some of these challenges, and our adventure in dealing with them.

Its gonna be a bumpy road, we have a lot of work to do, and only 3 months to complete it in. Wish us luck

First, here is a nice diagram to get into the mood:

 

Above is the rack overview of our our move plan, about a week ago, when we where still hoping to locate the bulk of the servers to Amersfoort (Netherlands). This, however, is not going to happen, due to politics I will cover in a later post.

We are gonna be moving all to site Veghel, and, as I will elaborate on in a later post, that site hardly has the room or the facilities to host us.