I spent a while trying to figure out this problem so I thought I'd post this tip to help others who may have the same issue. The problem I had was trying to connect a MacBook to a wireless router which uses 64-bit WEP key. The AirPort prompt on the Mac wasn't very helpful, giving only 3 options for WEP key (WEP Password, WEP 128 bit HEX and ASCII). I chose WEP Password and entered the 64-bit WEP HEX key and unbeknownst to me, Mac did not understand this key and cannot connect to the wireless router. Eventually I discovered the syntax to enter this key, thanks to the article below, to get the WEP Password prompt to understand the 64-bit HEX key you have to precede the key by 0x. This I thought is very counter-intuitive, considering Mac's reputation for the ease of use of their OS.

HOWTO: Configure wireless with 64-bit WEP on Mac OS X
http://domster83.wordpress.com/2007/03/09/how-to-configure-wireless-with-64-bit-wep-on-mac-os-x/