An interesting discussion began in the comments of Raymond Chen's blog entry the other day. His post was about canonical order of entries in an ACL, but the comments drifted toward some interesting behavior of NTFS under Windows. It was off topic there, but I think the discussion touched on some important points, so I'm continuing it here.
One of the things touched on was that file deletions are really a directory operation. NTFS supports hard links, which means a single physical file can be referenced by multiple directory entries. Normally you would think of a file deletion as removing the physical file, but in fact it is simply removing a link to it from a specific directory. The physical file is only deleted when all references to it are gone.
This makes the security semantics a bit interesting. Directories have the DELETE_CHILD permission, which grants the caller permission to remove any entry from it, regardless of permissions on the individual files those entries reference. However, a physical file also has a DELETE permission for itself, which grants you permission to remove any directory entry that references it -- even if you do not have DELETE_CHILD permission on that directory.
Norman Diamond raised the question of how deletions of open files behave in comparison with POSIX. Under POSIX, if an open file is deleted, the directory entry for it is immediately removed. If no directory entries are left, the file is effectively anonymous, but still physically exists until all processes have closed it.
Under NT, deletions are slightly different. When an open file is deleted, the directory entry used to reference it is marked as "delete pending", but it is not removed until all open handles to the file through it are closed. While the directory entry is in the "delete pending" state, no one can open the file using that path. Once all processes have closed the file, if no more directory entries for it exist, the file is physically removed.
Basically, the only major difference in behavior between POSIX and NT is that NT keeps the directory entry around for naming purposes instead of making the file anonymous.
Norman also posted about an interesting issue he encountered when playing with SFU. In short, he has two directories named "Pinball" and "pinball" that appear to go to the same place. He assumes this is due to hard links, but NT does not actually have hard links for directories. He also later implies that Win32 can be made case sensitive, which also isn't accurate. I can certainly understand why he would reach these conclusions, since information on this subject is rather hard to find and understand. So how do I explain the behavior he's seeing?
Internally NT treats everything as an object in a large directory tree, and NTFS volumes are accessed through branches on that tree. The Object Manager is normally case sensitive, but individual API calls can request case-insensitivity. The Win32 subsystem always requests case-insensitivity with just a few exceptions, such as CreateFile() with FILE_FLAG_POSIX_SEMANTICS. The POSIX subsystem installed by SFU does not.
The Native API interface to the Object Manager, which both the Win32 and SFU subsystems use, has an option to apply the case-insensitivity flag to all calls through it. When this option is turned on, no userland process is able to make case-sensitive calls. As of XP, this option internally defaults to on, which forces even SFU to be case-insensitive. However, it can be controlled via a registry option, which is what the SFU installer changes when you choose the case sensitivity option.
Presumably at some point shortly after Norman installed SFU, a POSIX application decided to create directory there using a different case than the one that was already present. Even under Win32, a directory list will show all entries in it, even if they only differ by case. However, whenever something attempts to access an entry with the case-insensitive flag in effect, the Object Manager will choose only one of those entries. It is simply not possible to access the other. Thus he gets the behavior he observed on the Win32 side -- it consistently accessed a particular one.
The behavior he gets from SFU's Korn shell could also be explained by case-insensitivity, where it is only accessing one directory even though he names the other. But this is SFU, which should be case sensitive, and obviously case sensitivity was in effect at the time one of these directories was created. What gives?
I'm betting Norman got bit by a little accident Microsoft had with a few Windows updates, such as the one mentioned in KB929110. In that case, a .NET Framework update mistakenly changed the registry key mentioned earlier to force case-insensitivity on all subsystems. Whoops.
key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\kernel
value name: ObCaseInsensitive
value type: DWORD
For case-sensitivity support under SFU, that registry value must be present and set to 0 (restart is required for changes to take effect). A while back I came across this key while trying to figure out why FILE_FLAG_POSIX_SEMANTICS was not working under Server 2003. Upon discovering the problem I promptly added deny and audit ACEs so I get notified if anything tries to change it. This stuff needs to be easier to find.