Aha! New blog...

I think I will start by posting the solution (or, rather, a solution), to a problem which has been troubling me for the best part of a day.

To replicate the problem, simply download and run the AJAX Control Toolkit sample website, and navigate to the Reorder List sample page using IE7. You can drag and drop the list items, and all works as expected.

Now go to the site's master page, and comment out the HTML DocType declaration at the top of the template markup, and run the ReorderList sample page again. Try scrolling down the page, and then reordering the list. If you are using the current version of the toolkit (3.0.20820) with the .NET Framework 3.5, things at this point go horribly wrong.

The javascript fails to account for the page's scroll offset when the browser is running in quirks mode, which produces the odd selection behaviour.

The actual cause of the problem seems to be, not in the AjaxControlToolkit code itself, but rather in the ASP.NET AJAX extension method: Sys.UI.DomElement.getLocation, which measures the scroll offset incorrectly when running in quirks mode vs. standards compliant mode.

The AjaxControlToolkit provides its own getLocation method, which is located within the Common.js file (AjaxControlToolkit/Common/Common.js). This method initially checks the browser type of the current browser; if the current browser is IE6 or earlier, it calculates the offset itself (to avoid precisely the type of odd behaviour you are currently experiencing!). However, whilst this method works for those running IE6, it fails to check for IE7 running in quirks mode...

In order to do so, I changed this (line 160): 

if (Sys.Browser.agent == Sys.Browser.InternetExplorer && Sys.Browser.version < 7) {

To this:
//EDIT: get the compability mode
var mode = document.compatMode;       
//EDIT: check the compatability mode
if (Sys.Browser.agent == Sys.Browser.InternetExplorer && 
(Sys.Browser.version < 7 || mode == 'BackCompat')) {

This by itself does not solve the problem, however. In quirks mode, you get the offset using: document.body.scrollTop. In standards compliant mode, you get the offset using: document.documentElement.scrollTop.

So, to account for this, I changed this (lines 212-3):

var ownerDocument = element.document.documentElement;
return new Sys.UI.Point(first.left - 2 - dLeft + ownerDocument.scrollLeft, 
first.top - 2 - dTop + ownerDocument.scrollTop);


To this:

//EDIT: get the scroltop/scrollleft values   
var scrollLeft = (mode == 'BackCompat') ? element.document.body.scrollLeft :
element.document.documentElement.scrollLeft;
var scrollTop = (mode == 'BackCompat') ? element.document.body.scrollTop : 
element.document.documentElement.scrollTop        
//EDIT: calculate the offsets using the correct scroll offset
var point = new Sys.UI.Point(first.left - 2 - dLeft + scrollLeft, 
first.top - 2 - dTop + scrollTop);        
return point;


And finally, just when I thought I had cracked it...I found things still weren't behaving as I had expected.

I then realised that the ReorderList's DropWatcherBehaviour.js script file (AjaxControlToolkit/ReorderList/DropWatcherBehaviour.js) was also attempting to perform its own correction for the scroll offset problem, albeit not terribly successfully.

I therefore commented out lines 138-9 of the DropWatcherBehaviour.js file, so it now looks like this:

if (oldOffset.x !== newOffset.x || oldOffset.y !== newOffset.y) {
  //var diff = AjaxControlToolkit.DragDropManager._getInstance().subtractPoints(
    oldOffset, newOffset);
  //var location = AjaxControlToolkit.DragDropManager._getInstance().subtractPoints(
    currentLocation, diff);
  $common.setLocation(dragVisualContainer, location);
}


And finally... success!

I should point out that, although this solution works for me, I haven't tested it for unintended side-effects. That said, my reorder list is now reordering regardless of scroll position, or browser compatability mode.

I hope this helps someone...