Geeks With Blogs
Caffeinated Coder A Grande, Triple Shot, Non-Fat Core Dump by Russell Ball

I just sent the following email to the BCL (Base Class Library) Team Blog email address in hopes of getting some insight into a weird problem we've been having.

I was hoping you could give me some general guidance on how Type.GetProperties() determines the sequence of the PropertyInfo objects it returns.

I just ran across this remark on MSDN about the order that Type.GetProperties() returns PropertyInfo objects

The GetProperties method does not return properties in a particular order, such as alphabetical or declaration order. Your code must not depend on the order in which properties are returned, because that order varies.

We have parsing code that uses the 1.1 framework which is dependent on the order in which properties are returned. It fails intermittently when run as part of a suite of MbUnit tests. After seeing this msdn remark, I changed the code to remove the sequence dependency and the MbUnit intermittent failure problem disappeared. This led me to believe that properties were sometimes being returned in an order other than what we expected (the order they were declared in the code files).

Normally we are not allowed to make production changes this time of the year unless the issue is critical, so we are trying to determine the likelihood of this causing production issues. The problem is that we get contradictory results based on how we test this. If I wrap the method in a hundred thousand loop iteration, then I can’t reproduce the error in MbUnit. The properties always return in the order in which they are declared in the class file. However, if I run the whole suite of tests several times in succession, then I receive random errors that seem to derive from the sequence being returned in a different order.

Is there some internal algorithm that might trigger a different sequential order to be generated based on some unique condition that running the whole MbUnit suite of tests as opposed to running them in a large loop would cause (i.e. a certain number of threads or memory/cpu pressure)?

Any insight you could offer in this problem would be greatly appreciated.

Thank you,

Russell Ball

Does anyone else have any ideas?

** Update **

We no longer get these failures in MbUnit if we revert it to the 1.1 framework by changing the supported frameworks node in the config file as described here. I'm guessing they changed the algorithm in the 2.0 framework that is used to determine the sequence.

That still doesn't explain why there is a difference between calling our code in a massive loop versus calling it as part of a suite of tests, but at least it puts our minds at ease with respect to the risk of intermittent failures occurring in our current production code.

Posted on Friday, December 21, 2007 10:09 AM Technical How-To's | Back to top


Comments on this post: Puzzling Issue Involving Reflection and MbUnit

# re: Puzzling Issue Involving Reflection and MbUnit
Requesting Gravatar...
I didn't drill very far with Reflector (plausible deniability and all that, you know), but a couple of levels in see this:


CerArrayList<RuntimePropertyInfo> propertyList = this.Cache.GetPropertyList(type, name);


To throw out a wild hypothesis, probably the cache is kept ordered to make lookups faster, but when you run the big momma, garbage collection is forcing the cache to get dumped, whereupon you see items returned in "random" order until the cache has reloaded.

It may not be that GetProperties() changed between 1.1 and 2.0, but rather something else completely unrelated changed which causes the garbage collector to run where it didn't before.

Pull out half your RAM and see if it happens more often.
Left by Dewayne Christensen on Dec 21, 2007 1:21 PM

# re: Puzzling Issue Involving Reflection and MbUnit
Requesting Gravatar...
@Dewayne
Good thought on the caching hypothesis. I just got a response from the BCL team and they also said that the loop probably wouldn't be a good test because of caching.

I just sent them a follow-up email asking if anything had changed between 1.1 and 2.0 that might have affected the sequence order. I'll publish the answer as soon as I get it.

Since I'm developing on a laptop that is already on the slow side, I fear that I might be restricted to notepad if I took any RAM out...:-)

BTW, Nice move spelunking that method with Reflector. That is why you'll always out-geek me...:-)
Left by Russell Ball on Dec 21, 2007 1:55 PM

# re: Puzzling Issue Involving Reflection and MbUnit
Requesting Gravatar...
I got a response within a few hours from Matt Greig on the BCL test team. Here's what he had to say:

My guess would be that your first method of testing (a hundred thousand loop iteration) is not actually testing the order the properties will be returned more than once. There is quite a lot of cacheing in the System.Reflection code looping is probably just getting the same cached list of properties each time - so really only testing the order once on the first time through the loop when the list is cached (didn't verify this but seems most likely given your results). I would trust the separate processes more.

From looking at this it seems this is a bug in your code (the order dependency), that you have already identified and have a fix for. As to how critical this issue is, or even if it is actually an issue - I can't tell without a lot more information and if the fix is warranted given the risk really is up to you and your management to make the call on.

I would also try to verify that it is indeed the order that is causing the failures (you didn't seem certain of this). Maybe something as simple as printing the expected and actual list of properties when the failure occurs to see if it is just the order could verify that.

Hope some of this helps,

Matt Greig
BCL test team
Left by Russell Ball on Dec 21, 2007 1:56 PM

# re: Puzzling Issue Involving Reflection and MbUnit
Requesting Gravatar...
Here's my follow-up question to them:

Thank you very much for the prompt reply and the insight into how caching works in loops.

I verified that the reflection sequence was actually the problem in our code by following your suggestion and adding debug code to verify how the sequence was being returned. I was able to do this because the properties all have custom attributes with a sequence property on them. (this was how I eventually fixed the dependency problem in our current code branch). The output of the intermittent test failures verifies that the properties were returned in a sequence that the code didn’t originally expect (see code excerpts at the bottom).

The one other interesting discovery we made is that we don’t have this issue if we force mbunit to use the 1.1 framework instead of the 2.0 framework(it’s a config option). Would changes in the 2.0 framework have affected how the order was returned in this method?

At the moment I am assuming that this is only a problem when our code is called from 2.0 code. Would you agree that this is a reasonable assumption to make? If so, I think I have enough evidence to wait until 1Q next year to push our fix out to production since this code isn’t currently being called by any 2.0 code in production.

Thanks again for your help,
Russ
Left by Russell Ball on Dec 21, 2007 1:57 PM

# re: Puzzling Issue Involving Reflection and MbUnit
Requesting Gravatar...
Here's the final response I got back from MS. I was impressed by the level of effort and timeliness that went into helping me. Thanks Matt and kudos MS!

Russell,
I wasn’t around when the changes from 1.1 to 2.0 were made so aren’t that familiar with exactly what changed. But with the documentation saying the order is not fixed it does mean that it is something that we would feel very comfortable changing between versions for any number of reasons – the most likely one seems to be performance.

If you have code that can fairly reliably reproduce the failure on 2.0 as your test seems to be able to and it isn’t able to reproduce it on 1.1, then it seems pretty safe that a change to 2.0 made this more of an issue. I hesitate to say that it is a problem only when calling from 2.0 code. It seems most likely that it was a change only present in 2.0 but it may be some other change just makes it more likely on 2.0. As an example, it could be that the Garbage Collector moving the object in memory may cause the properties to be returned in the different order and some changes to the Garbage Collector made it more likely to move the object in 2.0 vs 1.1. In that case the problem would still be present in 1.1 just less visible and thus harder to reproduce. Without knowing what causes the properties to come back in the specific order it is hard to make any firm claims about when it will happen and when it won’t. I guess what I’m saying is that your assumption that it is a 2.0 issue seems reasonable but I am not able to guarantee it in anyway.

Basically, I think you have it right. You have found the bug, and if it is an issue (depends on what happens when the list is not in the expected order) it is mostly (if not only) when running on 2.0. To me it sounds safe to wait, but if the results from hitting the bug are severe enough (i.e. leaking someone’s personal private data) it may justify more investigation to try and see if it can be reproduced under 1.1. At that point you really have two options that I could see. The first would be doing heavy debugging on the 2.0 version to try and determine what is causing the order to be different and then reproducing those conditions in 1.1 and verify it doesn’t affect the order. This approach would be the most complete but the debugging of the 2.0 version could be very intensive and may not justify the cost. The second approach would be more a blind brute force approach. This approach would be testing without knowing what is causing the properties to be returned in a different order in 2.0 and thus much harder to complete verify. Basically at that point you are needing to test Type.GetProperties() to see if an assumption about the order being constant is the case or not under 1.1, which would lead back to your question of what conditions that may cause this order to be different – and honestly this could be almost anything most likely though I would think would be what you originally mentioned threading or memory/cpu pressure. The trick would then be to do enough testing that you can justify an argument that it is constant. The advantage of this approach is that you only need to do as much testing as is warranted and it gradually increase confidence (rather than all or nothing like the previous approach), however the drawback is that it will never give you complete confidence that you have reproduced what caused the 2.0 order to change.

Hope this gives you what you need to know to move forward,
Matt
Left by Russell Ball on Dec 24, 2007 9:21 AM

Your comment:
 (will show your gravatar)


Copyright © Russell Ball | Powered by: GeeksWithBlogs.net