I spent a while trying to figure out this problem so I thought I'd post this tip to help others who may have the same issue. The problem I had was trying to connect a MacBook to a wireless router which uses 64-bit WEP key. The AirPort prompt on the Mac wasn't very helpful, giving only 3 options for WEP key (WEP Password, WEP 128 bit HEX and ASCII). I chose WEP Password and entered the 64-bit WEP HEX key and unbeknownst to me, Mac did not understand this key and cannot connect to the wireless router. Eventually I discovered the syntax to enter this key, thanks to the article below, to get the WEP Password prompt to understand the 64-bit HEX key you have to precede the key by 0x. This I thought is very counter-intuitive, considering Mac's reputation for the ease of use of their OS.
HOWTO: Configure wireless with 64-bit WEP on Mac OS X