In this post I would like to share how's our life here at university developing the VUES (Virtual University and Expert System, pronouced as "views") which you see at www.aiub.edu. Well, you will have to login to see how big it is.
Let's start with this semester's (Spring 2004-2005) registration experience:
- We have a central server which hosts the MS Sql Server for VUES and Oracle for PLegacy Registration System and Accounts. That server for some reason crashed just before the registration day. The next day around 3000 students are going to come to register and the whole campus is going to be blocked by angry furstrated students, teachers, administration within an hour if the registration does not start in time.
- The server crashed around 9 PM at night when we were about to leave for home after whole days of laborous work for the preparation of registration system. We just deployed the latest code and database, doing some testing and the server crashed in front of our eyes.
- We somehow recovered SQL Server. It's always a breeze no matter what happens. But we could not recover the Oracle which we need for correct calcuation of accounts.
- After long struggle, Oracle was live but the Oracle Database service was not getting started.
- After long search, I found that it is a common problem. We need to login to SRVMGR and connect using "internal" user account and perform some command line to start the service.
- But we do not know the "internal" account password. It is with the "Head of IT" and he is in Phillipine on vacation. It's then 10 PM night in Bangladesh, around 1 AM in Phillipines, and there's no way we can call him and get it that night.
- We used a software called Chikkat to send urgent SMS to his mobile asking for the password. Hours passed by he did not respond.
- Out of frustration and fear, I started writing mock calculation code to at least do the assessment for the current semester. Who cares if around 8000 is lost per student anyway. We are talking about 50 millions! I also copied and pasted a table's data from our temporary Oracle server to source code to somehow get the assessment done. If we do not get the password next day, somehow we can run the registration by dummy calculation and buy another 24 hours to get the password.
- We went home around 11:30 PM, tensed, tired, worried and most important of all, frightened. Unfortunately, I had a delivery that night for a project I am doing for HP and I had to work whole night for that. I think I broke human endurance record that night. Too bad, no guiness record was recorded.
- Next day, we got the password! Hurray! We were back in business. But another problem strike, our public web server cannot connect to central database server, nither can any of our computers. If this happens, all students will have to come to university to do the registration. No more home service. There was a strike going on that day and we cannot even go to University to fix the problem.
- After long struggle, we found that there's been a widespread Winsock2 corruption and there's a handy utility on the net named "LSPFix" to fix this problem. We ran the program, it detected a DLL which was causing problem and everything went live online!
We have this type of experience every semester. During registration we go through so many variety of problems - Oracle crashing down, database corruption, network link breaks, server becomes too slow, hard drive gets out of disk space, bug in the code caught at last moment etc. Every semester, we try to prevent all possible cases we learn from previous semesters, but something new always happens. I wonder, if this is how it goes in such a small place, imagine how unimaginably challanging and life threatening it is to those administrators who run sites like Yahoo, Google, Microsoft etc.