IBM

Datacenter Move post 4

Robert Kloosterhuis — Wed, 28 May 2008 18:28:30 GMT

Youtube videos are worth a thousands words, so I will let them do the talking.

Progress TCR Move May27 part 1 (the old location)

Progress TCR Move May27 part 2 (the new location)

What Mustafa thinks of IBM rackmount kits:

And some pics from the last 3 days:
(Click for larger versions)

IBM xseries 336, bout 3 years old now

ServeRAID 6M controller (in the PCI slot bay). Its IBM branded but its basically an Adaptec.
This comes out of one of the 2 xseries 336 servers that, together with the EXP400 shelf, served as a Windows 2003 cluster. The ServeRAID controllers are needed to provide failover control of the shared disk shelf.

Mainboard of an xseries 336, with the PCI card bay/thing removed. The blue bracket at the top is where usually an RSA-II management card would be sitting, but this one doesnt have one :(

ServeRAID 6M removed from PCI bay of the 336 server

The rack is slowly emptying. I remember when I was building it all up, 3 years ago! Check it out

Installing windows on an IBM xseries 336. Its been a while. I noticed the IBM Serveguide CD has a few more options now.

Picture I needed to have to illustrate where to connect everything.

Ready to move to new location

Not the most ideal way of moving servers, but its better than nothing. At least they are softer here than in the back of the car.

Richard, our project manager, trying to get more work in.

Temporary cabling ;)

Its slowly growing

They are not done with the rack interconnects, damnit. I cant finish my patching like this.

Our new firewall cluster

I love the blue glow of the console. Kinda wierd to have that out-of-place IBM in there, squashed between the HP.s. The cable rail for the server is different than HP aswell, so that will be fun to cable.

Moved some servers around, getting to our final config now.

WAN comms rack in the new location

LANcoms rack in the new location

Khalid on servicedesk duty

Gertjan is helping us decomission

Mustafa hard at work decomissioning servers

A lot of servers where decommissioned today, these are all basically being scrapped.

Forcing hardware power on state with IBM Director

Robert Kloosterhuis — Thu, 28 Dec 2006 12:22:00 GMT

Okie.. so I am totally bored at work, lets see what fun I can have with IBM Director and some spare servers lying around.

I thought it might be cool to try and make a rule that checked if a server had been turned off, and then brought it back up.

This would be usefull for us, as we cannot physically administer all the servers we are responsible for, and there are always overzealos ‘admins’ walking about in remote locations who have a tendency to sometimes turn off servers instread of just logging out.

Above you see 2 IBM xseries 306m servers on the bench. They both contain the PCI version of the IBM RSA-II remote management card.

This card gives you a nice web-based management interface for your server, even if its powered down. You can, for example, turn on the server from here, or monitor its vital statistics, and take control of the console.

You can also set up the RSA card to report to a central IBM Director management server. Here you see the server listed as a “physical platform”, and as you can see, it has been associated with the IBM Director Agent component running in windows.

We can do a lot of stuff with the object here, same as via the web interface, as much more, as you can see from the menu.

As you can see above, one of the things we can do from here is turn the power on and off remotely, as with the web interface.

We now have all the bits we need to build a solution.

We have an Alert source: The RSA-II card, and a management server to interpret the alerts, the IBM Director Server
We have an eventing engine that can bind the alert to some actions, and the IBM director server itself is capable of actually performing the actions too.

To start, we create a rule in the IBM Director Event Action Plan builder tool.

It consists of 2 components.. an event filter.. or ‘trigger’ as I like to call it, and then some actions to perform once the trigger is tripped.

In the IBM world, servers and software (such as agents) generate alerts using a bunch of different languages or protocols. You can read about some of them here.

The one we are looking for here is a so-called MPA Alert. Its sent out by hardware like the RSA card, or the xseries BMC (Baseboard Management Controller), or in our case by the RSA-II card.

We are gonna respond to the MPA.Component.Server.Power.Off event, here you see it occuring in the IBM director eventlog:

In our event action plan, we make a Threshold Filter that looks specifically for this event occuring.

We assign a 10 second timeout to the filter, and the Count field is set to 1, it only needs to occur once for the filter to trigger the actions.

Actions are pretty straitforward. We want the server in question to be turned back on when the event is triggered, and we want to recieve some kind of alert of this happening.

As you could see before, the IBM director server itself gives various power options for servers.

Luckily for us, when building events, the Event Action Plan Builder provides an interface to many of these controls for the Eventing engine to use

The last bit we need is a neato mail to be sent to us admins when this event is triggered.

So, now we can take our finished Event Action Plan, and apply it to some objects.

For this to work, you need to apply the Action Plan to the physical platform, not the Agent Object of the server.

Ok.. thats about it.

If all goes well, you will find you can no longer turn off the server, whahah. :D (unless you unplug the Ethernet of the RSA card of course). It will turn itself on every time within 10 seconds of being turned off. Talk about uptime!

And to boot, you get a nice mail in your inbox:

Tags: IBM, IBM+Director, RSA, RSA-II, Hardware+Alerts, MPA, SNMP, xseries, IBM+Director+5.20, Remote+Supervisor+Adapter

IBM Bladecenter power warning

Robert Kloosterhuis — Thu, 01 Jun 2006 11:48:00 GMT

This post discusses the Standard IBM Bladecenter Chassis, not the T -or H Chassis systems.

A common warning message you might recieve from an IBM Bladecenter is as follows:

"Demand exceeds a single power module. Throttling can occur in power domain 1."

The message is normal.

In a normal rudundant situation, all power needs are drawn from both PSU's, so you effectively have 2 x 2000 = 4000w for a single power domain during normal running. Its not a fail-over system, its drawns its power from both PSU's at the same time. So its kinda like 'load balencing'.

In a NON-redudant situation, so when you loose a single power domain, the bladecenter will only have 2000w left per power domain.

For 8 servers, its drawing more than 2000, as you saw, 2418 or therabouts.

With the power policy "Redundant with potential performance impact", the bladecenter will attempt to clock-down some of the blades servers until the powerconsumption is below 2000w, so it can keep running.

The clocked down servers will of course suffer the "performance impact".

In short: The Bladecenter you are using is not able to support all blades at full performance, IF you loose power redundancy.
Therefor, actually, these bladecenters can not be considdered "fully redundant" .
It may not suprise you that IBM, not its partners, do not like to easily admit to this limitation.

In the meantime, the warning remains, to remind you that this may happen when you loose redundancy.

There are 2 workarounds:
1. Redistribute your blade servers across both power domains evenly, so you dont use more than 2000 watt per power domain. If you place more servers this will be come unattainable though.

2. Make sure all 4 PSU's are connected to a redundant power source, so backup power (with generator). This more or less circumvents the entire problem.

My bladecenter pics on Wikimedia

Robert Kloosterhuis — Sun, 28 May 2006 13:27:00 GMT

I was approached by a contributer of the Wikimedia group, if they could use some of the pictures I had uploaded of Bladecenters, for the official Wikipedia article on them.

I of course said it was oke, and now 3 of my pictures have become part of the Wikimedia archive as offical resources under the GNU FDL licence.

http://commons.wikimedia.org/wiki/Image:IBM_HS20_blade_server.jpg

http://commons.wikimedia.org/wiki/Image:Pile_of_IBM_HS20s.jpg

http://commons.wikimedia.org/wiki/Image:IBM_bladecenter_%28front%29.jpg

I was very very impressed with the way I was approached and this was handled.
They now seem to use ‘official’ standard emails for request of usage of pictures and other media, and the email exchange was very well and professionally handled.

Here is a copy paste of the original mail I recieved.

Hi Jeminus,

my name is G.P. and I'm writing to ask you a permission to include some of your blade server images in Wikipedia, the free encyclopedia you may browse at:

http://en.wikipedia.org/

I think these three images would be a great addition to our blade server article:

http://www.flickr.com/photos/jemimus/66531212/

http://www.flickr.com/photos/jemimus/66530803/

http://www.flickr.com/photos/jemimus/74452762/

I have listed them in order of personal preference. As you might know, Wikipedia is a free encyclopedia that is collaboratively edited by volunteers from around the world.

Our goal is to create a comprehensive knowledge base that may be freely distributed and available at no charge.

Normally we ask permission for material to be used under the terms of the GNU Free Documentation License. This means that although you retain the copyright and authorship of your own work, you are granting permission for all others (not just Wikipedia) to use, copy, and share your materials freely -- and even potentially use them commercially -- so long as they do not try to claim the copyright themselves, nor prevent others from using or copying them freely.

You can read this license in full at:

http://en.wikipedia.org/wiki/Wikipedia:Text_of_the_GFDL

This license expressly protects creators from being considered responsible for modifications made by others, while ensuring that creators are credited for their work.

There is more information on our copyright policy at:

http://en.wikipedia.org/wiki/Wikipedia:Copyrights

We choose the GFDL because we consider it the best available tool for ensuring our encyclopedia can remain free for all to use, while providing credit to everyone who donates text and images. This may or may not be compatible with your goals in creating the materials available on your website. Please be assured that if permission is not granted, your materials will not be used at Wikipedia -- we have a very strict policy against copyright violations.

With your permission, we will credit you for your work in the image's permanent description page, noting that it is your work and is used with your permission, and we will provide a link back to your website.

We invite your collaboration in writing and editing articles on this subject and any others that might interest you. Please see the following article for more information.

http://en.wikipedia.org/wiki/Wikipedia:Welcome,_newcomers

Thank you for your time.

Kindly,

Finding Downloads on the IBM Site

Robert Kloosterhuis — Wed, 18 Jan 2006 18:14:00 GMT

Finding and Downloading anything from the IBM site is still a confounding process.
This post, attempts to illustrate some of the issues I encounter, and has also been my mental notepad in this process. This post is also a rant.

We purchased two IBM xSeries 366 servers, with the ServeRAID 8i SAS controller included.
We installed these servers using the latest ServerGuide CD, version 7.3.6, dates 11/25/2005
The OS is Windows Server 2003 SP1

However, once we installed the IBM director Agent, the servers started reporting the following error:

(screenshots from the ServeRAID manager)

Checking the driver tab in Windows cofirms the driver version used, this driver is included on the ServerGuide 7.3.6 CD.

However.. apon checking the ServerGuide CD 7.3.6 itself to confirm the driver version, on the CD,in the location sguide\w2003drv\$oem$\$1\drv\srsas , there is a readme file, but all it lists is that the version included is 8.15, which seems to be the version number for a ServeRAID CD release. .. the actual driver version is not mentioned at all in the file
Hoever.. checking the INF file itself, reveals that it is in fact, the correct driver, the one being used by our servers: DriverVer=09/02/2005, 5.0.2.8110

—————————-

So.. it made sense to go look for an updated version of the ServeRAID firmware, in order to get them all on the same level again, and working correctly with the 5.0.2.8110 driver.
However.. finding the correct download proves to be somewhat of a challenge.

Ideally I would want a bootable CD, or a Windows-based Firemware Updater, as any floppy images are pretty useless, as the 366 servers dont have a floppy drive. (ignoring for a moment that I could create a bootable CD with the floppy images, which is a lot of work and something I am not familiar with)

The Download Methods

There are 5 methods (or pages.. or ‘’ways’’) I have found, that can produce a list of downloads that should be relevant for what you are looking for.

Method 1:
The Multiple-file download page can be found here:

When I use it to look for downloads regarding this controller type, it uses the following page: http://www-307.ibm.com/pc/support/site.wss/product.do?brandind=6&familyind=50255&subcategoryind=60316&partnumberind=283070&operatingsystemind=49975&template=%2Fproductselection%2Flandingpages%2FdownloadsDriversLandingPage.vm&validate=true

These downloads dont seem to be related to the SAS controller..the ServeRAID CD's are not even listed.

Method 2:
If I use the same page as above, to look for downloads specific to our servers type (366), then I get the following page:
http://www-307.ibm.com/pc/support/site.wss/product.do?brandind=8&familyind=218719&machineind=219593&operatingsystemind=118054&template=%2Fproductselection%2Flandingpages%2FdownloadsDriversLandingPage.vm&validate=true

Method 3:
If can also search for relevant drivers using the following page:
http://www-03.ibm.com/servers/eserver/support/xseries/x366/downloadinghwonly.html
But under the 'Hardware Only' option, no downloads are displayed for the ServeRAID . I do get to see them when I choose the 'Windows' option:
http://www-307.ibm.com/pc/support/site.wss/product.do?template=/product.do?template=%2Fproductpage%2Flandingpages%2FproductPageLandingPage.vm&sitestyle=ibm&brandind=8&familyind=218719&machineind=0&modelind=0&partnumberind=0&subcategoryind=0&doctypeind=9&doccategoryind=0&operatingsystemind=118054&validate=true

Method 4:
Another option is to use the Software Download matrix for a server, the link to this matrix is sometimes listed in the support options pages for a server, in this case: http://www-307.ibm.com/pc/support/site.wss/document.do?lndocid=MIGR-59142

Method 5:
I stubled across this by accident, its the software download matrix for the ServeRAID 8i controller itself, though it is linked no where from any of the other pages: http://www-1.ibm.com/support/docview.wss?uid=psg1MIGR-61958&rs=100
However, as you can see.. only one of the enties actually works… the Windows drivers are not linked, and firmware is not even mentioned.

—————————-

The downloads you find

8.20 Support CD
http://www-307.ibm.com/pc/support/site.wss/document.do?sitestyle=ibm&lndocid=MIGR-61529

Method 2, 3 and 4 lists the link to the 8.20 ISO downloads, both for the Application CD, and the Support CD, which is the one we want, as it should update the firmware of the controller if we boot from it. The CD is dated

The 8.20 Support CD also includes Windows drivers for the controller.. these seem to be the latest ones found thusfar: DriverVer=11/17/2005, 5.0.2.8261 , they are newer than the current ones we are using.

The 8.20 support CD also includes the firmware, as I had hoped. But the included readme file does not list any version number starting with 5.0 .. it instead speaks of being version “”8.25” .. odd considdering this is the 8.20 CD.. and the only 8.25 CD that is available for download is the Japanse version of the Application CD.. I have not yet seen a 8.25 Support CD, in any language

I have no way of knowing if using this firmware version will solve the problem.

AAC-ServeRAID 8i Device Drivers and Firmware - Servers
Method 2 and 3 also links to what appears to be a seperate download of drivers and firmware: http://www-307.ibm.com/pc/support/site.wss/document.do?sitestyle=ibm&lndocid=MIGR-59553
Now.. hoever this download is called "AAC-ServeRAID 8i Device Drivers and Firmware - Servers", there are in fact, or so it seems, no device drivers included, merely the Firmware update for the controller. Also note that these are floppy-images, and the server in question, the xSeries 366, has no floppy drive!
Also, no where in the readme files are any version numbers mentioned.. only the rather generic "8.20" code, indicating that this firmware is probably also included on the 8.20 support CD.. but it wasn’’t .. remember?

IBM ServeRAID drivers for Microsoft Windows 2000 and Windows Server 2003 - IBM eServer xSeries 366
http://www-307.ibm.com/pc/support/site.wss/document.do?lndocid=MIGR-59076

Method 2, 3 and 4 all link to a page that contain a seperate file download for Windows 2003 drivers for the ServeRAID controller.
These seem to be the most recent ones that are seperatly listed on these pages. The download page and the readme file identify the version of the drivers to be 5.0.0.7771, dates March 23, 2005.
This version appears to be first version that supports the ServeRAID 8i SAS controller.
However.. we already know that both the 8.20 support CD, and the 7.3.6 ServeGuide CD contain newer versions 5.0.2.8261 and 5.0.2.8110 respectively.. why are those not seperately listed and available for download?

(Critical update) IBM ServeRAID Support diskette images for 8i for Microsoft Windows (32-bit) - Servers
http://www-307.ibm.com/pc/support/site.wss/document.do?lndocid=MIGR-61745

Method 2, 3 and 4 also list this download, dated 13 Dec 2005. Method 2 and 3 actually refer to this download as “ServeRAID 8i device drivers and firmware”
I can imagine firmware updates to be packaged as bootable floppy images, but Windows device drivers? Whats the point of that?
The download page lists the version as our good old 8.20, however the readme file of the download lists 8.25 as being the version.
The file name of the download is 40k8697.exe

Apon extrating the floppy image, it turns out indeed to be Windows drivers, version “”8.25” according to the readme, but actually DriverVer=11/17/2005, 5.0.2.8261, same as on the latest ServeGuide CD.

So why is this download sometimes refered to as ServeRAID 8i device drivers and firmware” ??

IBM ServeRAID Diskette Images for SCSI - Servers
http://www-307.ibm.com/pc/support/site.wss/document.do?sitestyle=ibm&lndocid=MIGR-60698

Methods 2 and 3, but not 4, list a rather new download.. the crypticly named IBM ServeRAID Diskette Images for SCSI - Servers , and dated 2006/01/11, so basicly just about a week old, as of this post.

The ‘’version’’ listed is 7.12.07. A strange version number, as it seems to refer to the ServeRAID 7.12 CD … that must mean it has updates for all ServeRAID versions exept the 8i… as that is the logic of the ServeRAID cd’s: only the 8:00 version and above contain support for the ServeRAID 8i SAS controller.. so its probably not even relevant for us.

Indeed.. as expected, no Firmware update floppy image is included for the 8i controller.. In fact, they all seem to be for the ServeRAID 7k controller… but you have to examine the individual files to find this out.

——————————-

In the end, I can find no newer version of the firmware, than the one included on the 8.20 Support CD.
That should mean that it will update the firmware to “5.0.2” or “8110”, or at least something that should stop the error we are seeing… right? I will let you know.

But this post serves first and foremost to illustrate the problems with the IBM website, and with the way IBM has (or rather has not) organised the support of their own products.

They do not seem to have any kind of consistancy in versioning, in naming, or in the format in which they present downloads and supporting software. Their website, and the way it presents downloads and information about them, seems very confused and chaotic, with many useless dead-end pages and a completely ineffective search engine. I find it very suprising that a company that has been in this business cannot seem to organise its own support in any decent way.

New IBM xSeries 346 servers

Robert Kloosterhuis — Fri, 30 Dec 2005 17:58:00 GMT

As part of a new global Active directory roleout, we recieved 4 xSeries 346 servers today, which will function as domain controllers.

Here are some pictures I made (also on my Flickr! work page)

IBM xSeries 346 server, with a 5 disks in the front. Mark is placing the controller.

The Blower-array of the IBM xSeries 346 server. Each fan can be taken out seperately, or you can take the entire rack of fans out in one go.

This little module that looks much like a memory stick, is in fact the Adaptec/IBM ServeRAID 7k SCSI controller.

New IBM xSeries 346 servers

Robert Kloosterhuis — Fri, 30 Dec 2005 17:58:00 GMT

As part of a new global Active directory roleout, we recieved 4 xSeries 346 servers today, which will function as domain controllers.

Here are some pictures I made (also on my Flickr! work page)

IBM xSeries 346 server, with a 5 disks in the front. Mark is placing the controller.

The Blower-array of the IBM xSeries 346 server. Each fan can be taken out seperately, or you can take the entire rack of fans out in one go.

This little module that looks much like a memory stick, is in fact the Adaptec/IBM ServeRAID 7k SCSI controller.

Building IBM Bladecenters - A foto journal

Robert Kloosterhuis — Thu, 24 Nov 2005 17:53:00 GMT

2 x xSeries 366 Servers, unpacked, and ready to recieve options

So much stuff to build, so much to unpack!!

Over 100.000 dollar worth of components on the table here. Extra CPU's, Harddisks, Option cards, even Acoustic Attenuation Modules .. all to be built into our Blades, Bladecenters, and xSeries 366's

21 HS20 Blade Servers, still in Box.

One of the 2 IBM 366 servers, they will be running Windows 2003 and SQL 2000.

The 2 Bladecenter Chassis waiting to be unpacked.

Way way waaaaaay to many boxes to unpack. Thank goed we got these 2 guys to do it :) And to actually put the stuff together!

Pile o' Blades. IBM HS20's to be exact, each has 2 2.8Ghz Xeon's and 2 x 36gb scsi 3.5" Disks, and 2gb of memory.

These two guys from BPSolutions are helping us putting it all together. Here they are working on equiping 2 IBM xSeries 366 servers that will function as database servers.

Placing 3 additional CPU's, 2.8Ghz Xeons

Figuring out how to move 280kg of IBM Enterprise Rack into the server room

An IBM HS20 Bladeserver. 14 of these go into a single chassis. Notice the slots of 2 x 3.5" SCSI harddrives. Click here for a closeup

View more pictures of cool server hardware here.

IBM BMC

Robert Kloosterhuis — Wed, 26 Oct 2005 12:13:00 GMT

For the sake of every IBM administrator’s sanity, and for the benifit of google:

The default password used by the BMC ( Baseboard management Controller ) for the eServer xSeries 336 3387 is as follows

username: USERID (all uppercase)
password: PASSW0RD (with a zero, and all uppercase)

IBM’s website seach is again useless to find this information, try this string:
http://www.ibm.com/search/?en=utf&v=14&lang=en&cc=us&lv=c&q=BMC+%22default+password%22

It seems to be unable to actually do a simple AND of the 2 words.

Anyhow. Google finally found:
http://www-307.ibm.com/pc/support/site.wss/document.do?lndocid=MIGR-42044

IBM is improving their xSeries resources online

Robert Kloosterhuis — Mon, 05 Sep 2005 12:39:00 GMT

So the guy from IBM who contacted me originally, also gave my name to a collegue, who is currently involved in streamlining the IBM Xseries resources, meaning, all the download and support stuff for the eServer Xseries family of products.

We spent about an hour last Thursday on the phone, going over various pages, and I had the opertunity to give my opinion on them, and I had some idea’’s of my own aswell. He will be getting back to me on the changed they are making.

I have said it before, I am quite impressed that IBM is going that extra mile to improve their customers experience when using the site.

Today I was already seeing some major changes to the site since last I visited, about 2 weeks ago. So it looks like they are putting a lot of effort into this initiative and pushing forward rapidly.

The X336 driver download page for example, seems to be much improved, although its possible it was always like this, and I simply never found this particular view of the support site.. that is a whole other problem with the IBM site generally ;)

Also the general Download and Drivers page is not the same one I saw before, it even has a dedicated Xseries link.. probably a testament to how hard this particular team is pushing their presense on the IBM site. But then there is this portal page aswell.. hmm.

Also.. they still seem to be giving you the same data via seperate pages: Page1 Page2 Page3, its a little confusing, but at least their intra-linking between all the sites seems to have improved.

They still seriously need to fix the clustering homepage though…

And some personal favorites I would like to see:

* Better online overviews of how certain IBM technologies work, like out-of-band management technologies (IPMI, Alerting, etc), Clustering (I still cant find any paper that explains to me what that IBM cluster solution software is, even though I have found out myself now)

* RSS feeds for just about everything

* The xSeries blog or blogs, by members of the xSeries engineering, dev, and support teams

* Links to external support pages and other IBM related resources, such as blogs

* A far better download manager

* Forums!!!!

* A proper sorting out of that whole UpdateXpress and UpdateXpress Live thing

All in all I am impressed at the progress! Well done xSeries web-team!