Search Crawl Deletion Policies SharePoint 2013

SharePoint 2013, like SharePoint 2010, uses four policies to control the removal of deleted content from the index.  When an error is encountered during a crawl, the indexer removes content based on the configured policies.

The table at the bottom of this post shows the differences between 2010 and 2013 default policies.

1.  Delete Policy for Access Denied or Not Found

There are two properties that govern the removal of items from the index; ErrorDeleteCountAllowed and ErrorDeleteIntervalAllowed.  The first property controls how many times this item should log an error, the second is the time since the first error is encountered, if both conditions are met, then the item is removed from the index.

The default values are ErrorDeleteCountAllowed: 10 and ErrorDeleteIntervalAllowed is 240 hours.  So by default an item would have to be not found ten times over a period of ten days before it is removed.

2.  Delete policy for all other errors

For all other errors, the two properties that control removal are ErrorCountAllowed and ErrorIntervalAllowed.  As with Access Denied and Not Found errors, the first property controls how many times this item should log an error, the second is the time since the first error is encountered, if both conditions are met, then the item is removed from the index.

The default values are ErrorCountAllowed is 100 and ErrorIntervalAllowed is 1440 hours (60 days)

3.  Re-crawl policy for SharePoint content

When running an incremental crawl, if the crawler encounters errors when crawling the content database, these two properties come into play; RecrawlErrorCount and RecrawlErrorInterval.  The first is the number of consecutive crawls errors can be encountered (not necessarily the same item) and the second is the interval from the first error.  If both conditions are met, then a re-crawl is performed on that content database.

The default values are RecrawlErrorCount is 10 and RecrawlErrorInterval is 360 hours (15 days)

4.  Delete unvisited policy

When running a full crawl of a content source, the delete unvisited phase comes into play, this policy removes items from the index that appear in the crawl history, but are not found in the current full crawl.  There are three options for the DeleteUnvisitedMethod:

• 0: all unvisited items are deleted
• 1 (default) unvisited items with the same host are retained but items discovered by following links to other hosts are removed.
• 2: none of the items are deleted.

How to view or change these values?

For some reason, Microsoft have not exposed these values in Central Administration, instead they must be changed using PowerShell.

To view and change these properties:

• Logon to the server with administrative privileges
• Open SharePoint 2013 Management Shell from the Start menu
• Type $sa = Get-SPEnterpriseSearchServiceApplication to gain access to the Search Service application

To view the settings for a property:
$sa.GetProperty("propertyname") e.g.: $sa.getproperty("ErrorDeleteCountAllowed")

To set the property:
$sa.SetProperty("propertyname", new value) eg: $sa.SetProperty(ErrorDeleteCountAllowed",1)

Comparison of Default Values between 2010 and 2013: