Robert's Sysadmin Blog

Big Bang Servermove succesfull

Robert Kloosterhuis — Mon, 07 Jul 2008 20:51:09 GMT

This Saterday, we switched the subnet over, and moved all the remain criticle systems over to a new serverroom across the country.

I didnt get to make as many pics as I liked, and no video, this was mainly cause I was so bussy of course. :D

Half the pics below are curtesy of Arnold.

Back of the Digital Alpha box, refernece sho for recabling.

Packing up the servers in special locked crates. You could see these movers where the right stuff, big burly guys, but they handled the servers like feathers.

Justin felt at home at the new location.

x3800 waiting to be converted.

We layed out the servers in the hall in the order we where building them into the racks.

Our project operation center, round the corner of the serverroom. From here the managers of the project coordinated the downtime, and the business on-site testing.

Tom, our WAN guy, on the phone with Mohamed, our Unix guy, who was supporting us remotely.

Very good food and snacks where provided by Arnold and Jan. Thank you guys! You know the best way to an engineers heart is through his stomach!

Justin loves Legos, so this pictures seems to make sense.

Justin working on the rack conversion kit for the IBM systemx 3800

Tom and me discussing the Proxy server and internet line out. Old proxy, new Internet line, and some firewall rules where needed.

Arnold:“2 pizza please!” ; Mustafa is thinking: “I dont like Pizza”…

Paul being technical. We hold our collective breath.

Sliding in a server into its new home.

Bottom of racks SR3 and SR4

Top Left rack, SR4

A suprise box. This witebox FTP server turned out to be running 4 essential ustomer FTP/EDI flows. We had space for it thankfully, resting on the IBM x3800. Hopefully it will be gone in 2 weeks, but nothing is temporary. Made this picture of it for the Visio rack diagram.

One of the 2 redundant FTP servers failed on transport. Justin and Paul spend 2 hours putting a new one together out of spare parts of old ones. HP Netservers here, P2 machines, a decade old.

Behind locked door and locked racks, all servers humm quitely, content in their new home.

A day later, back in the first location:

Empty racks moved out of the server rooms

Mail Cluster going to get sent back to UK

Gertjan visibilly enjoying dismanteling the place. 6 years of mental burden of supporting tihs stuff being dealt with here ;)

The NAS, all the data for the Netherlands, with all volumes deleted and now unplugged, ready for storage.

Videos of “demolition”:

Mustafa and Gertjan have taken apart the entire second server room in 1 day. (click here for link if you cant see the embed above)

Oooh.. Nobs! Lovely nobs!! (click here for link if you cant see the embed above)

Meanwhile, back at the old location

Robert Kloosterhuis — Fri, 04 Jul 2008 19:15:41 GMT

A tour of the current state of the office, and one of the server rooms, where most racks are now empty or near-empty.

An Epic moment. Gert-Jan uninstalls Citrix on the last 2 servers in the farm, effectively ending 6 years of the 2000-user, 60-server Metaframe XP Citrix farm that served our Netherlands users. I would have had Marcel do this, but he has gone on holiday.

Various IBM servers and almost all the blades lie ready to be moved to the new location. We don't have a use for any of these currently, so they will go into storage.

Remember when we where young and the world (servers) where new?

Both blade centers are going into storage, we have no use for them.

Blade Centers in they Hayday

The racks are becoming quite bare now.

Before:

After:

Before:

After:

Before:

After:

Before:

After:

The pile for the garbage container crows and grows.

Marcel posing with all off the DL360 G3 severs (going into storage). he built the farm all those years ago, now he bears witness to its demise. Its a little sad for all of us that spent so years maintaining it all.

I managed to get my hands on one of my favorite servers, the IBM systemx 3650 with the 8 SAS disks in it. We reinstalled it as a new SQL2005 system, that will host, amungst other things, the HPSIM, IBM Director, Websense and Sharepoint databases. All for internal IT use.

Tags: server, servers, rack, racking, sysadmin, IBM, xseries, systemx, systemx+3650, bladecenter, blades, HS20

T-Minus 1 day to Big Bang serverroom move

Robert Kloosterhuis — Thu, 03 Jul 2008 19:12:27 GMT

Spent today going over some cabling details mostly. Its strange how stressed we all became over these little details, while we will have larger issues to worry about on the day.

(high res)

During the move, we will be moving the Voip servers, RF-Controllers (used by various sites to do handheld-scanning), an Authentication server, the Wyse-Terminal (thin client) management server, and 1 PDC.

We are also moving 2 very criticle FTP/EDI (Electronic Data Interchange) servers, that handle lots of customer-related FTP flows and internal EDI between warehouse management systems.

All the above systems (in the rack diagram in RED) rely on keeping their old IP adress for now, and thus we are moving the entire subnet over to the new site. This is why we call it the big bang. After that, the old site will no longer be routable to the rest of the network, and all that remains is stripping it and storing the hardware that is left.

Also the former Domain Controllers / DNS servers are an issue. Their IP adresses have long since been used in clients all over the place, the config of many of which we cant control centerally (for example through DHCP). Therefore I am moving 1 of the old PDC’s, and its keeping 2 of the 3 legacy IP adresses we need to keep alive.

We spent yesterday doing last cabling work and other such things in preperation for Saterday (see pics below). I still have some things to do, there are some management/reporting scripts still running on the old site I need to migrate tomorow, and I have yet to get round to re-installing HPSIM again. I might start on it after this post, actually.

Here are yesterdays pics:

Starting to look more like the planned diagram now. As you can see we, we replaced the Sun Storagetek with the HP Storageworks MSL9000 that we salvaged from one of the Warehouse Management Systems that was migrated to Prague last week. We dont know if we can use it though, its kinda overkill for the amount of data we now need to backup anyway.

Mustafa is set to become my new right-hand man after his project, he certainly has got the right attitude ;)

Oh thats Justin, he has been added to this project to take some of the load of me. He really knows his stuff and is a wirlwind of highly-opinionated energy ;)

Ninja-Consultant. Implements ESX-virtualisation solutions when you are not looking!

Ser is our resident LAN guy. Here we created a patching/switchport diagram for the racks (on the laptop screen), and he is tightening up the VLAN configs on the stacked core switches. Justin and me then quickly repatched everything before anyone noticed there was a service interruption ;) (dont worry, the really criticle stuff goes in during the Big Bang)

Tags: server, servers, rack, racking, sysadmin

Version 7.1 of the TCR rack diagram

Robert Kloosterhuis — Mon, 30 Jun 2008 21:57:22 GMT

This is more or less how its gonna be. All the bits are now in place, half the rack stuff has been put in place already, the last stuff goes in coming Vriday.

One of the biggest oversights we made was with power requirements. If you want redundant power for a lot of servers, you need to make sure you have a LOT of plugs, enough PDU’s, and plenty of PSU to back it all up. We didnt, partly due to time contraints, budget, but also short-sightedness, and my reliance on people that I, to put it bluntly, shouldnt have relied on to do the math.

Above pic also available this time is very high res, so you can see all the exact server types. These Visio stencils are very detailed, I love em!

Tags: rack, racks, server, servers, visio

Computer Room Cleanup and re-cabling Pics and Movies

Robert Kloosterhuis — Mon, 30 Jun 2008 21:37:20 GMT

So a few weeks ago, we spent our Saterday doing major recable work on the new Technical Computer room (TCR) in the new location, in preperation for our Big Bang services move coming up on the 5^th of July (it was delayed from the 22^nd))

So thise pics and videos are a few weeks old now, but who cares.

PICTURES (scroll down for movies)

Not enough room to neatly get all the fat KVM cables sorted, so later I actually extended the room for the cables to 2U, and used the cable guide rail only for ethernet cables and power.

2U above the KVM not just being used for those fat cables, but also for the Avocent Switchview box, and its adapter, that is attached to the HP KVM box. Advocent KVM/IP hardware rocks by the way. If you have a non-IP KVM swich, consider the Switchview IP 1020, it makes life so much easier and they are super simple to set up!

Too many cables, not enough room. Once we are done with the move, I am gonna seriously tie this shit down. Meanwhile I curse those HP 10K racks for not having ANY room at the sides for large cable bundles. Oh.. not to mention ANYTHING to attach a tierib to!

The lineup of servers. The missing one was at the old location being synced up with another FTP box.

The master at work ;)

Not easy to work with all the cables all over the place.

The patch panel guys where not done yet as you can see.

Hello!

Mustafa putting some of the servers into cold storage. You wouldn’t believe the kind of hardware we will end up putting into storage cause we have no immediate use for it!

This is what happens when you dont move your servers professionally. We dont have the budget for it, so that is our exuse. Servers where fine, though I would not recommend this.

Carting around servers.

So very little room to work with. Look at the space at the sides, notice the PDU, then imagine it being full of servers. See my point?

So here we decided to move some servers over to the middle rack. This is partly to do with the fact that I discovered we dont have nearly enough power connectors in these racks to fill them.

I havent been back there since, I really hope they cleared these bundles up now.

I hate loose cables like that.

HP-UX on a 9000 box. Very old, but still used. Hosts a warehouse management system, all terminal based.

The proud admin of the HP box.

Richard wasnt very usefull during our work, so he caught up on project admin ;)

MOVIES

A lot of engineers in a small space. This movie gives you an idea of the space :)

Me working on the cables. All the cables all over the floor was half the point of tackling this place, what a mess.

Me working on getting the KVM set up. The little box is the Avocent IP View 1020

The most important element of our productivity right there!

All the “temporary” plugs and cables finally being cleared out. I wonder how many admins out there wish they had the time or the downtime to do the same and really clean house.

Overview of our LAN and WAN racks.

Making slow but steady progress on the cables. 3 ours later it would look very diffferent!

Labling ALL of your cables on both ends means you can accuractly document what hooks up to what from A-Z. Doing the initial labeling though is a pain, so this is where our “certified labeler” came in ;)

The amount of old hardware our IT department has in stock is just crazy. They dont know what to do with it all! We should have a CRT yardsale or something. Anyone need a Lexmark multifunctional? The IBM xseries 336 being stored here are only a fraction of the hardware we will be storing.

Its 2 am and the end of our scheduled downtime. Clearing up the last stuff and wiping the servers off, and turning em all back on. Hooked up to the new UPS, we could finally have them ALL on without burning out the old grid. The best bit of all was we managed to clear almost all cables from the floor.

Datacenter Move post 4

Robert Kloosterhuis — Wed, 28 May 2008 18:28:30 GMT

Youtube videos are worth a thousands words, so I will let them do the talking.

Progress TCR Move May27 part 1 (the old location)

Progress TCR Move May27 part 2 (the new location)

What Mustafa thinks of IBM rackmount kits:

And some pics from the last 3 days:
(Click for larger versions)

IBM xseries 336, bout 3 years old now

ServeRAID 6M controller (in the PCI slot bay). Its IBM branded but its basically an Adaptec.
This comes out of one of the 2 xseries 336 servers that, together with the EXP400 shelf, served as a Windows 2003 cluster. The ServeRAID controllers are needed to provide failover control of the shared disk shelf.

Mainboard of an xseries 336, with the PCI card bay/thing removed. The blue bracket at the top is where usually an RSA-II management card would be sitting, but this one doesnt have one :(

ServeRAID 6M removed from PCI bay of the 336 server

The rack is slowly emptying. I remember when I was building it all up, 3 years ago! Check it out

Installing windows on an IBM xseries 336. Its been a while. I noticed the IBM Serveguide CD has a few more options now.

Picture I needed to have to illustrate where to connect everything.

Ready to move to new location

Not the most ideal way of moving servers, but its better than nothing. At least they are softer here than in the back of the car.

Richard, our project manager, trying to get more work in.

Temporary cabling ;)

Its slowly growing

They are not done with the rack interconnects, damnit. I cant finish my patching like this.

Our new firewall cluster

I love the blue glow of the console. Kinda wierd to have that out-of-place IBM in there, squashed between the HP.s. The cable rail for the server is different than HP aswell, so that will be fun to cable.

Moved some servers around, getting to our final config now.

WAN comms rack in the new location

LANcoms rack in the new location

Khalid on servicedesk duty

Gertjan is helping us decomission

Mustafa hard at work decomissioning servers

A lot of servers where decommissioned today, these are all basically being scrapped.

The Vegas SuperNAP

Robert Kloosterhuis — Tue, 27 May 2008 20:49:24 GMT

Datacenter Knowledge has 2 posts up about Switch Communications new datacenter in Las Vegas, which they are claiming is the highest-density datacenter in the world.

http://www.datacenterknowledge.com/archives/2008/May/27/the_vegas_supernap_a_data_center_revolution.html

http://www.datacenterknowledge.com/archives/2008/May/27/1500_watts_a_square_foot_a_look_at_tscif.html

Switch Communications says it is successfully cooling a section of its Las Vegas data center running at nearly 1,500 watts per square foot using air cooling. How are they accomplishing this?

The key to Switch's high-density cooling is a design known as Thermal Separate Compartment in Facility (TSCIF), according to company co-founder Rob Roy. The ingredients in this approach include high-capacity AC units placed outside the data center area, and a tightly integrated hot aisle containment system for the racks. Here's an overview:

* The cabinets are set on a slab, with no raised floor.

* Chilled air is delivered into the cold aisle near the ceiling rather than through the floor, and enters the cabinets through the front.

* Each cabinet fits into a slot in the TSCIF unit, which encapsulates the rear and sides of each cabinet, while the open front extends beyond the enclosure.

* The hot aisle containment system delivers waste heat back into the ceiling plenum, where it can be returned to the chiller.

Very cool video of the SuperNAP setup:

http://www.switchnap.com/pages/products/the-supernap-video.php

More pics of their T-Scif cooling system: http://www.switchnap.com/pages/tech-specs/thermal-scif.php

The statistics off their site:

407,000 square feet of space

250 MVA Switch owned substation

146 MVA of generator capacity

84 MVA of UPS supply

30,000 tons of system plus system cooling

4,500,000 CFM

30 cooling towers

100% heat containment using thermal-scif™

Designed for 1500 watts per sq. ft. density

7000+ cabinets

Armed 24/7/365 military trained
Switch employed security staff

Tags: Supernap, Switch+Communications, Datacenter

Desicionmaking on the new Proxy solution

Robert Kloosterhuis — Mon, 26 May 2008 17:34:29 GMT

For your enjoyment, here is a, slightly edited, email I just sent to the department head and various other decision makers. It goes over some of the options we need to consider to solve the current issues with our internet access.

Names and places have been changed to protect the guilty ;)

And please exuse the spelling. I was in a hurry and I really dont care about spelling as much as I do content.

----------------------------

Hi all,

We are currently faced with some decisions that need to me made in regard to the Internet Proxy solution for the Netherlands and Belgium.

This is the current situation in regard to the proxy servers in Lala City and Chipville.

Server Lala City: LA-Server-S99
Server Chipville: CHIPVILLE-Server-S99

Both servers are HP DL360 G2 servers, and are now approaching 8 years of age. They are very out of warranty, and no hardware support can be expected from HP anymore regarding these.
Both servers run Windows 2000 standard
The Proxy software on both servers is ISA server 2000, running on the SQL MSDE engine. This software is still supported by Microsoft, but has been superceded by 2 newer versions.
In addition, we currently run the Surfcontrol web-filtering software, as a plug-in for ISA.
This software allows us to tightly control web-behaviour, for example to allow certain users access to certain sites, and to block entire catagories of websites, or web-protocols.
We have built up a pretty extensive rule-set over the years on both machines, and both rulesets are largely identical.
The company "Surfcontrol" was aquired by Websense in 2007, and since that time the Surfcontrol software is no longer supported, no patches or service packs are being offered for download, and no licences are being extended or sold, forcing all former Surfcontrol customers, including us, to look for alternatives.
The software combination on these servers has causes us some issues in the past. Some elements of Surfcontrol have always been buggy, and as the hardware has aged, it has become unreliable.
Furthermore, the decision to use SQL MSDE has causes problems, because of its inherrent 2gb limit.

Lala City
The Lala City proxy server is due to be replaced with new hardware, located in SiteB. This action is outstanding as part of the TCR move project.
As part of this, a new server was purchased, together with W ISA server 2006, and SQL 2005
At the moment, no replacement for Surfcontrol has yet been purchased, although Dick Dickerson did get a cost estimate for the Websense software, based on a single server, 500 users, and 3 years of licensing. (included as attachement)
A decision on this has been on the back burner, due to the fact that we where also planning on moving the current ISA server to SiteB anyway, and using the Chipville ISA server as a backup.

Chipville
The old Proxy server in Chipville is in a similair state to the one in Lala City. Although one of its 2 disks (that run in a mirror) has failed since last week.
This causes a serious risk to internet service continuity. It also represents a risk to the TCR move project, as this server is now no longer a reliable fallback while we move the Lala City server.

We need to decide how to proceed going forward.

The time factor
We have only a limited time to come up with a solution. Currently the situation in Chipville is more pressing, because of the hardware failure of the server there.
The big-bang server move from TCR Lala City is sqeduled less than a month from now, and we need a stable and supportablesolution at the very least in Chipville before that time, and idealy a solution for Lala City aswell.

There are a number of options:

Option 1. Keep the current servers
The Lala City server can be moved to SiteB and continue to operate from there, serving Internet users (non-citrix) in the Netherlands.
However, the hardware and software is no longer supported, the software is in an unstable state due to past problems with Surfcontrol, and the ISA MSDE database.
Due to the advanced age of the hardware, it is only a matter of time before it fails. Moving it might actually break it too.
The Chipville server cannot operate as-is, on a failed hardware mirror. This absolutely needs to be replaced, more or less disqualifying this option.
Due to the above, I cannot recommend this option in any way.

Option 2. Outsource the Proxy service to European Datacenter / UK
This would involve redirecting all internet traffic from Netherlands and Belgium to an outside, centralised Proxy system for internet access.
This would simplify our support model somewhat, and remove the technical burden of supporting the solution ourselves.
The downside though, is that we no longer have direct control over what is allowed/disallowed over the Internet.
By default, as far as I have heard, no rules are in place for both the UK and European Datacenter proxy solutions, meaning that there are no limits on what people can do with the Internet connection, it would be a free-for-all, whereas right now, we have strict limits on usage.
This option should be considdered. But the question has to be asked why the web-filtering function was ever needed in the first place. If web-filtering and control remains a business requirement, that this options cannot be considdered.

Option 3. Hybrid In-country hosting / European Datacenter hosting
I have been made aware of a version of the European Datacenter hosting scenario, that includes re-directing in-country internet traffic to European Datacenter, but in combination with a local Proxy/web-filter server, running the Websense software. This would involve installing a local server with the "Websense" filtering software, and "chaining" it to the Websense Proxy server in European Datacenter. Many countries apparently already follow this model.
This has the advantage of retaining local control of a rulebase, allowing us to continue to restrict internet use where nessesary, but with the advantage of not needing local Internet line for basic Internet use anymore. MEGACORP(TM) also can retain an amount of corperate internet-use control, via the gateway in European Datacenter, as all internet fraffic eventually moves through there to get out. Currently MEGACORP(TM) does not pose any global restrictions on the Internet gateway in European Datacenter, as far as I have heard.
This option should be considdered, however it will take some time to study and set up properly. The support model may be complicated because of the fact you are dealing with possible web-filtering and proxying in 2 different locations, supported by 2 different organisations. It would however, also require that local websense software be purchased and supported. I have also been told by some, that the connection via European Datacenter is very slow and not that usefull for many operational tasks. This could hurt us, as we run a number of line-of-business web-based applications over the internet. (Hp Shipview, etc)
We would also benefit from the fact that the websense software can be centerally managed from 1 console, making it very easy keep the netherlands and Belgium ruleset identical, and simplifying reporting and failover.
I would recommend this option if we can be sure the performance is adequate for our business needs, and if the support model can be agreed apon quickly. The major downside of this solution currently is that it will take time to set up, and we dont have much time anymore.

Option 4. New installation In-Country
This involves basicly rebuilding the 2 Proxy servers on new hardware, and installing fresh, current and supported Proxy and web-filtering software.
In this scenario we would use our own local Intranet lines in SiteB and Chipville.
We would directly support the solution, and maintain direct control over the web-filter ruleset, this is the most simple support scenario.
Hardware for this is already available: The replacement of the Lala City server was already part of the TCR move project, as is the licence for ISA 2006 and SQL 2005.
Hardware for Chipville is also already available on site, in the form of a 3-year-old IBM server, however, this server may soon fall out of hardware support (needs to be checked).
Apart from the ISA and SQL license that would be needed for the Chipville server, we need new web-filtering software for both servers, again, IF the business still deems this a requirement.
If they dont, then this solution would provide unfiltered internet access to all (non-citrix) internet users in Netherlands and Belgium.
For the web-filtering requirement, I would at this time advise to als go with the Websesne software, as they are currently regarded as the market leader, and their software is well supported ans well known in the industry. (they are incorperating a lot of the Surftcontrol concepts as part of the aquesition )
We need to look at the current available hardware for this. Almost all the hardware we have is 3 years old or older, so it may be advisable to considder purchasing a new piece of hardware for this solution in Chipville.
This option should be considdered. It has the advantage of retaining central control and will be quick to set up, once the software has been purchased. The downside is that the Websense software is expensive, so we may want to considder looking at alternatives, even though it has becomes a defacto standard within MEGACORP(TM). Again, we have a time-constraint problem here.
We would also benefit from the fact that the websense software can be centerally managed from 1 console, making it very easy keep the netherlands and Belgium ruleset identical, and simplifying reporting and failover.
I would recommend this option first and foremost, and it is the prefered solution technically, considdering the circumstances.

Again, i wish to stress the timeconstraints we have, less than a month before big-bang, we want a new solution up and running within the next 3 weeks!

-------------------------