I've been keeping a track on this forum entry at Brian Maddens Site for some time as I have seen similar issues elsewhere.
http://www.brianmadden.com/forum/tm.aspx?m=50628&mpage=2
It makes very interesting reading, and thanks to mkools for all the very useful info ;-)) It still makes an interesting insight on the device?
I'm just wondering out loud how long it might take before someone has Netscaler running in vmware?
+++++++++++ Quote ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
I managed to mount the CAG filesystem, they use lilo to boot the machine, check the lilo.conf, it has a line that says:
password="R35tr1tct3d*" (It's even spelled the wrong way with an extra 't')
Maybe that's the same as the bios password, they didn't even encrypt the filesystem so it might be worth to try
Anyway, there's another line that says:
append="devfs=mount"
You can easily add acpi=off to this section and acpi will be disabled, so you get:
append="devfs=mount acpi=off"
I enabled SSH daemon on port 22 and did a reset on the root password (which was succesfull) and rebooted the CAG.
After that I could SSH into the appliance and logon. So when the CAG eats your memory again you can logon through SSH and check (using 'top') the process that is eating up your memory, it might get you somewhere.
Here are some screenies:

Here's top which you can use to see which process is eating up your memory:

If you want to try some things let me know if I can be of any help.
I could write short instruction in order to get you in the box through SSH.
And how to get root access?
Ok here are the steps to get root xces to the CAG.
Please notice that I only tested this succesfull on a vmware image of version 4.2.0, but it should work for the appliance as well.
First step is to download a linux livecd, I used Ubuntu 6.06.1 LTS 32-bit intel version (check www.ubuntu.com).
Boot the appliance from that CD, when you see the ubuntu boot menu select the safe vga mode option, the livecd will now boot.
When it has booted, open a terminal (Accessoires > Terminal).
Enter: sudo passwd root
Type a new password for the Ubuntu rootuser.
Next, type: su , and enter the password just specified.
This step is optional and is done so that you don't have to sudo all the time.
Ok, you are now root, create a directory in the root: mkdir /cag
After that, mount the cag partition to that directory, in the vm I had IDE disks, so I had to do: mount /dev/hda1 /cag, but the appliance is using sata disks so you might have to do: mount /dev/sda1 /cag
After that, when you do a cd /cag and 'ls' you should see the appliance data.
Next you need to chroot into the CAG environment to make the changes, type:
chroot /cag
Your prompt will now change in something like bash-2.05$.
Next run 'setup', change the firewall settings from high to disabled.
Next, go to system services, deselect 'iptables', make sure sshd and xinetd are selected, and press quit to save changes.
Next, change the rootuser password.
type: passwd
Re-enter the password twice.
Next step is to run SSHD to generate the keys.
type: /etc/init.d/sshd start
I didn't get any output, but when you do ls /etc/ssh you should see the keyfiles.
type 'exit' to leave the chrooted environment, next type: umount /cag
Reboot the appliance, download putty (search google for putty) and logon to the appliance, port 22, username root with the password you've just changed and you should get access.
Next you can change some lilo settings, e.g. disable acpi, you have to edit /etc/lilo.conf (as described above, add it to the append section of the kernel). You have to use vi to edit the file, if you don't know vi you should read the manual, it can be a pain if you've never worked with it (man vi).
When you're done editing, save the file and run: /sbin/lilo to update lilo and reboot the system (type reboot).
I guess that's about it, all these steps are in my head so I might have forgotten something, let me know when you can't get in and I will see if i can help you out.
One important thing:
When you change the duplex settings of the CAG, another lilo.conf is copied over the original one, so you will lose your settings made at that time! If you want to disable ACPI with all duplex modes, all lilo configurations are in the folder: /config and you have to edit them all.
Good luck!
Thanks again to mkools