Geeks With Blogs
Greg Young Greg.ToString()

Some you have probably seen a post from last Tuesday entitled Floating Point Fun. If you have not read this I would recommend going back and reading it before continuing. In this post I discuss some of the interesting things that can happen when dealing with floating point math in C#, it is important to note that these items did not happen in version 1.x of the framework.

The root of these problems is that when in a register the floating point is treated with a different precision than when it is being held in memory. As such you can run into cases where you are comparing a Float32 or a Float64 against an 80 bit register based float. These equality comparisons (or conversions to other types such as an integer) can obviously fail due to the difference in precision.

After tracing through the generated assembly, I found a great reference on the subject at David Notario's Blog. David correctly points out that this is not a CLR/JIT issue, in fact changes like this were eluded to in the CLR spec (there is a quote from the ECMA spec on his blog) or here http://dotnet.di.unipi.it/EcmaSpec/PartitionI/cont11.html#_Toc527182172

 

There was some documentation on this breaking change in 2.0. Here is the listing from the breaking changes documentation

In the CLR model, we assert that arguments, locals, and return values (things which you can't tell their size) can be expanded at will. When you say float, it means anything greater than or equal to a float. So we can sometimes give you what you asked for 'or better'. When we do this, we can spill 'extra' data, almost like a 'you only asked for 15 precision points, but congratulations! We gave you 18!'. If someone expected the floating point precision to always remain the exactly the same, they could be affected. In order to faciliate performance improvements and better scenarios, the CLR may rewrite (as in this case) parts of the register. For example, things that used to truncate because of spilling, no longer do. We make these kinds of changes all the time. We believe this is an appropriate change, and it is even called out specifically in the CLI specification, as something which can, and will occur with different iterations:

What makes these changes particularly nasty is that you are forced to second guess how the JIT works in order to provide consistent results. In my previous post I used the example of

float f = 97.09f;
f = (f * 100f);
int tmp = (int)f;
Console.WriteLine(tmp);

 

This code will work in either debug or release mode when a debugger is attached, having the debugger attached will disable the JIT optimizations that cause the problem. It does as I describe in the previous post fail when run without the debugger. If we wanted it to work all of the time we would need to write it in the form.

float f = 97.09f;
f = (float)(f * 100f);
int tmp = (int)f;
Console.WriteLine(tmp);

 

The explicit cast to a float forces it to be narrowed back to a float32, without the narrowing it will actually be in a register as an 80 bit float. As such we end with a predictable behavior of always producing the correct result of 9709.

The problem I have with this behavior is that it is a leaky abstraction. In order to have our code work properly (and to be efficient) we need to know exactly how the compiler and the JIT intends to optimize our code. This introduces a logical problem though as by its very definition we do not know how the JIT will optimize our code. The JIT very well could place this into a register at some times and not at others or the JIT run on a different platform could offer a different behavior than the JIT we tested with.

This becomes especially nasty when dealing constants, consider the following code.

float f = 97.09f;
f = (f * 100f);
bool test = f == 97.09f * 100f;
Console.WriteLine(test);

 

What is the value of test? The abstraction leaks for both the compiler and the JIT. To start with, are the floats actually being calculated at runtime or is the compiler smart enough to realize that they are constants? In this particular case the C# compiler generates instructions for the first floating point operation but recognizes that the second is a constant value and as such pre-computes the value. These types of scenarios are exactly the type of thing that compilers look for when optimizing. 

If the compiler did not recognize the constant expression this might work as both of the calculations would have been done with their result being saved in a register, at that point we would actually have to look at how the JIT handled this case. Both of these items may change based upon environment.

The CLR has basically left the choice to the language as to how it wants to handle these cases. Visual C++ has handled this by providing compiler switches. The link is also interesting as it deals with how the switches apply as well to optimizations that occur within the compiler that can cause further issues. C# does not have many such optimizations at this point but it is only a matter of time before they get introduced.

I would therefore propose that C# should be given switches as well (similar to those available for C++) which could allow for the automatic narrowing of floating point values.

It is often brought up that C# does it the way it does it for performance reasons; it is obviously faster to leave values in registers when possible as opposed to narrowing them. The only way consistent way of doing it is through the use of the narrowing. From all of the studies I have seen, C# is primarily used for business applications where consistency (and reduction of programmer thought) is the primary goal and quite often run-time speed is sacrificed in order to better meet these goals (think abstractions). If an argument can be made for C++ to have an option of a precise switch, I would imagine a better argument can in fact be made for C#.

Based upon this I would also propose that the default behavior of the compiler should be to support consistent operations (/fp:precise in C++).

This switch would not eliminate people from writing code that was dependent upon how the compiler/JIT treated things; it would however force the programmer to make a conscious decision by setting the switch that they were assuming the risks associated with the performance gains. VC++ by default runs with /fp:precise so I would not think it a large jump to make the C# compiler consistent.

 

As a note for the people I am sure will say, “don't do this .. use a precision range or round instead“. These are simply examples ... I am fine with using these solutions (in fact I normally use range checks). The problem is that code like this crops up regularly and it creates a very subtle problem (that did not exist in 1.x). That and there are times when you actually want (validly) to do an equivalence test on two floating point numbers that should have a consistent value (i.e. results of the same calculation). If these operations are to be disallowed, that is fine as well .. but lets completely disallow them and have the compiler generate an warning/error in the circumstance.

This issue is known by very few, if you agree with the concepts here I ask you to either leave a comment below or to link to the post on your blog. Hopefully getting this knowledge more into the mainstream will both reduce the number of bugs caused by this subtlety and bring more focus on it by those with the power to change it.

Posted on Monday, June 5, 2006 6:41 AM Under the covers | Back to top


Comments on this post: C# FP Math Leaky Abstraction

# re: C# FP Math Leaky Abstraction
Requesting Gravatar...
I agree. If it is good enough for C++, it is good enough for C#.

And if Microsoft insists that we should use ranged checks to compare values, they should at least make it easier to do so in code. I'd like to be able to do this:

float x = 90.10f;
float y = 100f * 0.9010f;
bool c = (x.IsInRange(y, .0001))

rather than:

float x = 90.10f;
float y = 100f * 0.9010f;
bool c = (Math.Abs(x - y) < .0001);

The first is more readable.
Left by Haacked on Jun 05, 2006 9:30 AM

# re: C# FP Math Leaky Abstraction
Requesting Gravatar...
A couple thoughts... firstly, you should NEVER expect precise results when doing floating-point. Equals is an evil operator (you need to do "near enough" comparisons).

That said, I'm totally against your suggestion of command line switches because it's not self documenting code. If you WANT narrowing symantics, I should be able to tell that simply by looking at your code. So, if you want narrowing at every step, then put in casts to (float) or (double) as needed. The C# compiler, CLR, and JIT will honor those calls.

As an aside, I'm not clear as to WHY you want to enforce narrowing behavior along the way... can you explain (without using == vs. "near enough" on the final answer)?
Left by Marc Brooks on Jun 05, 2006 11:55 AM

# re: C# FP Math Leaky Abstraction
Requesting Gravatar...
There's actually a coding standard for C/C++ (MISRA Rule 50, if I remember correctly) "Floating point values shall not be tested for exact equality or inequality".

Because of the inaccuracy of floating-point storage it's just not safe to perform this test; so adding something to the compiler to "fix" it is incorrect. You should be clearly testing for range of values, not for [in]equality.
Left by Peter Ritchie on Jun 05, 2006 12:35 PM

# re: C# FP Math Leaky Abstraction
Requesting Gravatar...
I agree that epsilons should be used .. I stated this in my post but somehow that was missed ...

This does not only apply to == .. it applies to <> as well.

If you prefer as an example ...

float f = 97.09f * 100f;
bool test = f > 97.09f * 99.999999f;
Console.WriteLine(test);



My point is that this is a breaking change from 1.x as an optimization (btw your code for doing with this with an epsilon would be on the order of 10 times slower compared to the narrowing). Also keep in mind I am bringing this up not for myself but for the thousands of people who will sit there and scratch their heads for a week at this code who did not have such issues in 1.x. What will really get them is the fact it only occurs in "production" mode. No amount of debugging will ever find this for them.

Just to be clear, I agreee 100% that equalities should not be done (I even put this in the post). I also think that if telling the compiler to do narrowing for you does not happen that it should fall onto the compiler to tell you that you are breaking the rule to help prevent it.

Keep in mind that these suggestions are focused at people who will not understand this discussion (in fact they would probably never read it), these people represent the majority of C#/VB.NET developers. Having the compiler check for tests such as these (and to force narrowing) is a trivial task and would make their code work without them thinking about it. If you don't believe the assumption on the majority of developers I invite you to nearly any question/answer site.

Surely more complex compiler or JIT optimizations will be coming soon .. at which point flags will be needed anyways.

Keep in mind this is a business vs academic discussion .. I know the academic answer.
Left by Greg Young on Jun 05, 2006 1:08 PM

# re: C# FP Math Leaky Abstraction
Requesting Gravatar...
Peter:

I think he makes that quite clear doesn't he?
"As a note for the people I am sure will say, “don't do this .. use a precision range or round instead“. These are simply examples ... I am fine with using these solutions (in fact I normally use range checks). The problem is that code like this crops up regularly and it creates a very subtle problem (that did not exist in 1.x). That and there are times when you actually want (validly) to do an equivalence test on two floating point numbers that should have a consistent value (i.e. results of the same calculation). If these operations are to be disallowed, that is fine as well .. but lets completely disallow them and have the compiler generate an warning/error in the circumstance."
Left by Bill on Jun 05, 2006 1:38 PM

# re: C# FP Math Leaky Abstraction
Requesting Gravatar...
Obviously if it affects == it will also affect < and > as well.

This is basic FP stuff.
Left by Matthew on Jun 05, 2006 4:29 PM

# re: C# FP Math Leaky Abstraction
Requesting Gravatar...
I was trying to be a wise ass to the "can you do it without a ==" question. As it is yet again missing the point.


Perhaps a bit more clarification ... As my original post says .. I use ranged equality checks.

In a perfect world everyone would always use ranged comparisons but unfortunately we do not live in a perfect world and there is a ton of code out there that already relies (albeit misguidedly) on operations from 1.x floating points and I believe there will be people making the same mistakes as long as fp code is being written. To assume that it will be done right all the time in a developer environment such as C# is naive.

A great parallel can be drawn to the 1.x winforms controls and their handling of threading. Sure some people knew that you shouldn't touch controls from another thread but may people did not know (or just did it anyway) as it was at times a simple mistake that would not usually break easily.

2.0 comes out and checks are added to the controls, next thing you know it becomes a FAQ on every question / answer site on the internet. I wonder how many subtle bugs in applications were fixed by this addition? MS also created the backgroundworker for those who needed background processing but probably could not handle the complexity of threading (yes there are _many_ such people out there).

Coming back to the FP narrowing .. right now there are countless systems silently having this problem. While taking the high and mighty academic route saying do it right all the time sounds great it does next to nothing to help the problem. I suggested putting a default behavior to handle narrowing automatically which some people seem to not like. I have yet to hear any intelligent discussion about the other option in my post of adding a warning/error message to the compiler if it detects these situations (which should be fairly easy to detect)

We have to face the fact that there are huge developer populations out there using the tool that are not well trained. Many struggle with things that others would consider basic. I have spoken with someone that said arrays are useless .. Frans gave a great example of wanting to use partial classes for business logic so "it's reusable".

This is a major problem .. providing that you work alone or with a tam of good people you will probably not have too many (if any issues). But if you try to work with an average team this is the type of problem that slowly and silently corrupts your data

I imagine that anyone managing a project is at this very moment wonderring if any of these types of issues wanderred into their system.
Left by Greg Young on Jun 05, 2006 5:47 PM

# re: C# FP Math Leaky Abstraction
Requesting Gravatar...
It is remarkable, very much the helpful information
When you invest in stocks, you have the potential of making more money than you would with other types of investments, such as fixed rate bonds and certificates of deposit, because stocks participate directly in the growth of the economy and over the long run have historically outperformed any other form of investment.et's see some benefits like first of all they don't require high investments and there are many products which are having good demand in today's market. Some of which i know is gold, silver, lead, zinc, copper etc. For details:
http://commodity-tips-mcx.blogspot.com/2010/12/crude-aluminium-copper.html
Left by Mcx copper Price on Dec 08, 2010 5:08 AM

Your comment:
 (will show your gravatar)


Copyright © Greg Young | Powered by: GeeksWithBlogs.net