We have a document library which makes use of the 'Link to a Document' content type which lets users add links to a document library. We noticed that users were using this feature to add links to documents hosted on sites outside of the SharePoint farm. This is all good as far as they are concerned because the content type does its job and end-user can access the external documents by clicking the link.
However, we noticed that none of these links were showing up in the search results. When we looked into the crawl log (/ssp/admin/_layouts/LogSummary.aspx) we found that all these links had an Error status type. The error text was:
This item could not be crawled because the crawler could not connect to the repository. (The item was deleted because it was either not found or the crawler was denied access to it.)
It took some digging, but the issue was because the index server was behind a firewall that was preventing traffic to external sites. Once we made exclusions to the firewall rules (by adding exceptions to the IP address and ports of the external site) and performed a full crawl the links were indexed and started showing up in the search results.